Machine vision systems and methods for determining attributes of a container in a digital image and detecting presence of capped tubes in a diagnostic instrument

Abstract

Accessing a subject digital image of the subject container including TV pre¬ determined element positions, accessing one or more template digital images including a first region of interest (ROI) having a first attribute and one or more second ROIs having a common second attribute, calculating for a first element position in the subject digital image a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image, determining whether one or more first subject attributes associated with the first element position include the first attribute based on the first similarity score, calculating for the first element position one or more second similarity scores for the one or more second ROI having a common second attribute, and determining whether the first subject attributes associated with the first element position include the common second attribute based on second similarity scores.

Description

[0001] MACHINE VISION SYSTEMS AND METHODS FOR DETERMINING ATTRIBUTES OF A CONTAINER IN A DIGITAL IMAGE AND DETECTING PRESENCE OF CAPPED TUBES IN A DIAGNOSTIC INSTRUMENT

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] The application claims priority to U.S. Provisional Application No. 63 / 661,266, filed June 18, 2024, the contents of which is incorporated herein by reference in its entirety.

[0004] BACKGROUND

[0005] This disclosure relates to machine vision techniques, and to methods for determining attributes associated with an element position of a container in a digital image.

[0006] Operation of a diagnostic instrument can involve use of various containers.

[0007] For example, tubes, reaction vessels, and other consumable items can be used in process procedures within the diagnostic instrument. During operation of certain diagnostic instruments, it can be beneficial to monitor the various containers and consumables used in the system. For example, it can be beneficial to monitor whether containers, such as tubes, within the system are covered or capped. Errors can occur if a capped tube is inadvertently introduced to certain processes or areas. Therefore, detecting if a tube has a cap on can be important for operation of certain diagnostic instruments. If an inadvertent capped tube is detected, the instrument may, for example, be able to trigger a process to uncap the tube.

[0008] Single beam optical sensors can be used to monitor containers within a diagnostic instrument, such as by detecting caps of the test tubes. For certain single beam optical sensor solutions, the optical beam scans across each item to be detected, which can be slow and time consuming. For example, a work deck of a diagnostic instrument may include 500 slots for receiving tubes, resulting in a long time to run a full scan. Installing a sensor, such as a laser beam, for each slot may be an alternative but significantly more expensive.

[0009] Certain machine vision algorithms, such as provided by vendors Cognex and Jadak, have been used for detecting caps of test tubes. However, these machine vision algorithms can be processor intensive, and incompatible with the processors and embedded devices used in certain diagnostic instruments. Additionally, these machine vision algorithms can be too slow to meet the demands of certain diagnostic instrument process requirements. Further these machine vision algorithms can require specific hardware positioning, which can be incompatible with determining attributes of containers in more than one area of the diagnostic instrument using reduced hardware, such as for example a single camera.

[0010] Thus, there is a need for improved machine vision techniques, which can be used, for example, to support diagnostic instrument operation.

[0011] SUMMARY

[0012] Certain purpose and advantages of the disclosed subject matter will be set forth in and apparent from the description that follows, as well as will be learned by practice of the disclosed subject matter. For purpose of illustration and not limitation, the various embodiments described herein relate to machine vision systems and methods for determining attributes associated with an element position of a container in a digital image. Additional advantages of the disclosed subject matter will be realized and attained by the systems and methods pointed out in the written description and claims hereof, as well as from the appended drawings.

[0013] To achieve these and other advantages, the disclosed subject matter provides machine vision methods. Machine vision methods in accordance with the disclosed subject matter include accessing a subject digital image of a subject container comprising N pre- determined element positions. Disclosed methods further include accessing one or more template digital images including a first region of interest (RO I) having a first attribute and one or more second ROIs having a common second attribute. Such methods further include calculating, for a first element position among the N pre-determined element positions in the subject digital image, a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image. The methods further include determining whether one or more first subject attributes associated with the first element position include the first attribute based on the first similarity score. In response to a determination that the one or more first subject attributes associated with the first element position do not include the first attribute, one or more second similarity scores for the one or more second ROI having a common second attribute can be calculated for the first element position. Disclosed methods further include determining whether the first subject attributes associated with the first element position include the common second attribute based on the second similarity scores.

[0014] Disclosed methods include calculating similarity scores by comparing digital content in ROI and corresponding digital content in the subject digital image. In certain embodiments, a pre-determined search region corresponding to an ROI may be defined for a first element position among the N pre-determined element positions in the subject digital image. The pre-determined search region may have one or more sub-regions of a size of the ROI. When calculating a similarity score of the first element position for the ROI, the machine vision system may calculate a normalized 2-dimensional (2D) correlation coefficient between each sub-region among the one or more sub-regions and the ROI. The machine vision system may determine a highest normalized 2D correlation coefficient among the calculated normalized 2D correlation coefficients as the similarity score of the first element position for the ROI. In certain embodiments, the first ROI can contain an empty element position, and the first attribute indicates the element position being empty.

[0015] In certain embodiments the one or more template digital images include k second ROIs. Each of the k second ROIs can contain an element of a respective element type having the common second attribute, with k being a number of possible element types. In certain embodiments, methods may further include determining which of the k element types is present at the first element position based on the second similarity scores.

[0016] In certain embodiments a determination that the one or more first subject attributes associated with the first element position do not include the common second attribute can be made when the second similarity scores indicate that none of the k element types having the common second attribute are present at the first element position.

[0017] In certain embodiments, the subject container can be configured to receive tubes at each of the N pre-determined positions, the tubes being either capped or uncapped, and the common second attribute can include an uncapped tube.

[0018] In certain embodiments, methods can include determining an exposure mode. For example, methods can include determining optimal exposure time and sensor gain for the determined exposure mode during a camera calibration period. For example, the subject digital image of the subject container can be taken using a digital camera with the determined optimal exposure time and sensor gain. The subject container may be located at a pre-determined relative location from the digital camera.

[0019] In certain embodiments, the subject digital image may comprise a singlecolor channel. The single-color channel producing highest contrast may be selected among red, green, blue (RGB) channels. The machine vision system may perform a gamma correction to the subject digital image to improve brightness of the subject digital image. In certain embodiments, the subject digital image may be with perspective distortion. For example, the subject digital image may include perspective distortion because the subject digital image is taken by the digital camera with a tilted view angle. In certain embodiments, a perspective transform can be performed on the subject digital image, and the perspective transform can correct the perspective distortion of the subject digital image.

[0020] In certain embodiments, a perspective transform matrix used for the perspective transform may be calculated based on current coordinates of the plurality of source anchor points in the subject digital image. For example, the perspective transform matrix can be selected to linearly relocate the plurality of source anchor points to respective pre-determined destination anchor points. In certain embodiments, the machine vision system may perform a perspective transform of the subject digital image to define a transformed subject digital image.

[0021] In certain embodiments, a perspective transform matrix used for the perspective transform may be pre-calculated based on registered coordinates of a plurality of source anchor points and applied to the subject digital image.

[0022] In certain embodiments, a camera calibration of the digital camera may be performed during the camera calibration period. Camera calibration of the digital camera can include taking a calibration digital image of a calibration target container at a predetermined relative location from the digital camera. Camera calibration can further include determining coordinates of a plurality of source anchor points of the calibration target container within the calibration digital image and registering the coordinates of the plurality of source anchor points. Camera calibration can further include creating a perspective transform matrix that linearly relocates the plurality of source anchor points to respective pre-determined destination anchor points. In certain embodiments, a perspective transform of the subject digital image can be performed to define a transformed subject digital image. The perspective transform of the subject digital image can be performed before calculating the first similarity score.

[0023] As embodied herein, to determine coordinates of a plurality of source anchor points of a container, the machine vision system may generate a brightness-improved calibration digital image by performing a gamma correction to a digital image containing the container. The machine vision system may generate a binary digital image by performing Otsu Thresholding algorithm on the brightness-improved digital image. Each pixel in the binary digital image may belong to a foreground or background. The machine vision system may detect a bounding box of the calibration target container based on projected sums on the binary calibration digital image. The machine vision system may determine the coordinates of the plurality of source anchor points by performing a template matching for each comer of the bounding box with a respective template image. A search region for each comer may be determined based on the bounding box.

[0024] In certain embodiments, the camera calibration may be performed for each of one or more container types. The coordinates of the plurality of source anchor points may be registered per container type.

[0025] As embodied herein, when the subject digital image is perspective transformed, the one or more template digital images can include one or more perspective transformed template container digital images of a template container comprising the N predetermined element positions. Each of the N pre-determined element positions on a perspective transformed template container digital image may be associated with a ROI and have a template attribute uniquely associated with one of the one or more template digital images. The first ROI and the one or more second ROIs may be associated with the first element position of the N pre-determined element positions in the one or more perspective transformed template container digital image.

[0026] In certain embodiments the one or more perspective transformed template container digital images include ROIs having the first attribute and the common second attribute associated with each of the N pre-determined element positions.

[0027] In certain embodiments, methods can include calculating, for each of the N pre-determined element positions in the transformed subject digital image, a first attribute similarity score for at least one of the one or more perspective transformed template container digital images by comparing digital content in an associated ROI of the one or more perspective transformed template container digital image having the first attribute and corresponding digital content in the transformed subject digital image. Methods can further include determining, for each of the N pre-determined element positions in the transformed subject digital image of the subject container, whether one or more first subject attributes associated with the respective element position includes the first attribute based on the first similarity score. Methods can further include calculating, for each of a subset of the N predetermined element positions for which a determination was made that the position does not include the first attribute, one or more second similarity scores for the associated ROI having the common second attribute; and determining whether the first subject attributes associated with each of the subset of the N pre-determined element positions include the common second attribute based on the second similarity scores.

[0028] In certain embodiments, the one or more perspective transformed template container digital images may include a first perspective transformed template container digital image having each of the N pre-determined element positions empty. The one or more perspective transformed template container digital images may include a second perspective transformed template container digital image having each of the N pre- determined element positions loaded with an element of a respective element type having the common second attribute.

[0029] The disclosed subject matter further provides machine vision systems for a subject container. Machine vision systems in accordance with the disclosed subject matter include a camera having a field of view and at least a portion of a diagnostic instrument positioned within the field of view and configured to receive the subject container, the subject container comprising N pre-determined element positions. Systems further include a data processing unit operatively coupled to the diagnostic instrument, and a memory storing instructions which, when executed by the processing unit, cause the system to perform any of the machine vision methods and procedures described herein.

[0030] In certain embodiments, the subject container can be a rack. The rack can be configured to receive a tube at each of the N pre-determined positions. The tubes can be either capped or uncapped, and the common second attribute can include an uncapped tube.

[0031] In certain embodiments, the first ROI can contain an empty element position, and the first attribute can indicate the first element position being empty.

[0032] In certain embodiments, the one or more template digital images can include k second ROIs, each containing a tube of a respective tube type having the common second attribute, with k being a number of possible tube types.

[0033] In certain embodiments the system can make a determination that the one or more first subject attributes associated with the first element position do not include the common second attribute when the second similarity scores indicate that none of the k tube types having the common second attribute are present at the first element position. The determination that the one or more first subject attributes associated with the first element position do not include the common second attribute can include a determination that a tube received in the first element position is capped. In certain embodiments the diagnostic instrument can configured to receive the subject container at a first area and a second subject container including O predetermined element positions at a second area. The machine vision system may capture, using the camera, a second subject digital image including an image of the second subject container, and determine one or more second container subject attributes for each of the O pre-determined element positions in the image of the second container.

[0034] It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the disclosed subject matter claimed.

[0035] The accompanying drawings, which are incorporated in and constitute part of this specification, are included to illustrate and provide a further understanding of the systems and methods of the disclosed subject matter. Together with the description, the drawings serve to explain the principles of the disclosed subject matter.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG. 1 illustrates an example auto exposure algorithm for determining improved or optimal exposure time.

[0038] FIG. 2 illustrates exemplary digital images having a single-color channel.

[0039] FIG. 3 illustrates a comparison between an original image and a gammacorrected image.

[0040] FIG. 4 illustrates a comparison between the input gamma-corrected image and the binary image after Otsu thresholding.

[0041] FIG. 5 illustrates an example unit-to-unit variation in camera FOV.

[0042] FIG. 6 illustrates an example camera calibration algorithm using an empty subject container as a calibration target. FIG. 7 illustrates an image of an exemplary vessel holder used as a calibration target.

[0043] FIG. 8 illustrates example procedures for ellipse-fitting.

[0044] FIG. 9 illustrates a first example of a perspective transform.

[0045] FIG. 10 illustrates a second example of a perspective transform of an image of a subject container having N pre-determined element positions.

[0046] FIG. 11 illustrates example projected sums on a binary image of a container.

[0047] FIG. 12 illustrates an example of template matching.

[0048] FIG. 13 illustrates example computations for the template matching.

[0049] FIG. 14 illustrates an example method for identifying source anchor points in a subject image of a subject container using template matching.

[0050] FIG. 15 illustrates an example bounding box and source anchor points identified for the subject container.

[0051] FIG. 16 illustrates an exemplary calibration target.

[0052] FIG. 17 illustrates an exemplary container with different amounts of a consumable item therein.

[0053] FIG. 18 illustrates an example subject image of a subject container having N pre-determined positions configured to receive consumables for a laboratory diagnostic instrument.

[0054] FIG. 19 illustrates an exemplary real-time source anchor point detection algorithm for identifying source anchor points of a subject container.

[0055] FIG. 20 illustrates an exemplary template digital image for an empty tube rack.

[0056] FIG. 21 illustrates exemplary template digital image for a rack of uncapped long tubes. FIG. 22 illustrates exemplary template digital image for a rack of uncapped long-slim tubes.

[0057] FIG. 23 illustrates exemplary template digital image for a rack of uncapped short tubes.

[0058] FIG. 24 illustrates exemplary comparison of an element position with an ROI containing a portion of an empty tube holder.

[0059] FIG. 25 illustrates exemplary comparison of an element position with an ROI containing a portion of an uncapped long tube.

[0060] FIG. 26 illustrates exemplary comparison of an element position with an ROI containing a portion of an uncapped long-slim tube.

[0061] FIG. 27 illustrates exemplary comparison of an element position with an ROI containing a portion of an uncapped short tube.

[0062] FIG. 28 illustrates an example algorithm for performing dynamic template matchings to detect capped tubes.

[0063] FIG. 29 illustrates exemplary dynamic template matching with an empty tube holder template image.

[0064] FIG. 30 illustrates exemplary dynamic template matching with template image fully-loaded with uncapped long tubes.

[0065] FIG. 31 illustrates exemplary dynamic template matching with template image fully-loaded with uncapped long-slim tubes.

[0066] FIG. 32 illustrates exemplary dynamic template matching with template image fully-loaded with uncapped short tubes.

[0067] FIG. 33 illustrates first example results of determining subject attributes based on dynamic template matchings with a plurality of digital template images. FIG. 34 illustrates second example results of determining subject attributes based on dynamic template matchings with a plurality of digital template images.

[0068] FIG. 35 illustrates third example results of determining subject attributes based on dynamic template matchings with a plurality of digital template images.

[0069] FIG. 36 illustrates fourth example results of determining subject attributes based on dynamic template matchings with a plurality of digital template images.

[0070] FIG. 37 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including an empty template container.

[0071] FIG. 38 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped long tubes.

[0072] FIG. 39 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped long-slim tubes.

[0073] FIG. 40 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped short tubes.

[0074] FIG. 41 illustrates an exemplary machine vision system.

[0075] FIG. 42 illustrates a portion of another exemplary diagnostic instrument and machine vision system.

[0076] FIG. 43 illustrates an example calibration target that is a rack with a white board in the front, containing four corner marks.

[0077] FIG. 44 illustrates an example perspective transform of an empty rack. FIG. 45 illustrates example color images, having red, green, blue (RGB) channels, taken from a simulation testbed.

[0078] FIG. 46 illustrates an example Gamma Correction on Tray of Lysis Tube and Transfer Tip.

[0079] FIG. 47 illustrates an example Gamma Correction on Tray of AV and AV Cap.

[0080] FIG. 48 illustrates an example Otsu thresholding.

[0081] FIG. 49 illustrates example projected sums on a binary image.

[0082] FIG. 50 illustrates another example projected sums on a binary image.

[0083] FIG. 51 illustrates four template images and four search regions for template matching algorithm.

[0084] FIG. 52 illustrates first example detected bounding boxes and detected source anchor points.

[0085] FIG. 53 illustrates second example detected bounding boxes and detected source anchor points.

[0086] FIG. 54 illustrates a third example detected bounding box and detected source anchor points.

[0087] FIG. 55 illustrates an example perspective correction on the tray of Lysis Tube and Transfer Tip.

[0088] FIG. 56 illustrates an example perspective correction on the tub of Wash Vessel.

[0089] FIG. 57 illustrates an example flowchart of image quality monitoring functions.

[0090] FIG. 58 illustrates an example architecture of the machine vision software.

[0091] FIG. 59 illustrates an example sequence diagram. DETAILED DESCRIPTION

[0092] Reference will now be made in detail to the various exemplary embodiments of the disclosed subject matter, which are illustrated in the accompanying drawings. The structure and corresponding method of operation of the disclosed subject matter will be described in conjunction with the detailed description of the system. The accompanying drawings, where like reference numerals refer to identical or functionally similar elements throughout the separate views, serve to further illustrate various embodiments and to explain various principles and advantages all in accordance with the disclosed subject matter.

[0093] The disclosed subject matter provides various machine vision methods and systems for determining attributes of an element position in a container. Determining attributes of an element position in a container in accordance with the disclosed subject matter generally includes accessing a subject digital image of the subject container including N pre-determined element positions. A perspective transform of the subject digital image can be performed by relocating a plurality of source anchor points corresponding to the subject digital image of the subject container to respective pre-determined destination anchor points to define a transformed subject digital image. One or more template digital images can be accessed, where each template digital image including an image of a template container including the N pre-determined element positions, each of the N pre-determined element positions in the template digital image being associated with a region of interest (ROI) and having a template attribute uniquely associated with one of the one or more template digital images. A similarity score can be calculated of each of the N predetermined element positions in the transformed subject digital image of the subject container for at least one of the one or more template digital images by comparing digital content in an associated ROI of the at least one template digital image and corresponding digital content in the transformed subject digital image. One or more subject attributes can be determined for each of the TV pre-determined element positions in the transformed subject digital image of the subject container based on the calculated similarity scores.

[0094] Throughout this disclosure, a container can include a tray, a tub, a rack, or any suitable container comprising one or more element positions.

[0095] 1 Setup and Calibration

[0096] Methods in accordance with the disclosed subject matter include accessing a subject digital image and accessing one or more template digital images. Subject and template digital images can be accessed using any suitable technique. For example, and not limitation, digital images can be accessed from a memory associated with a digital camera. Various techniques can be used to capture images for use with the disclosed subject matter. For example, camera and / or subject container position can be calibrated to ensure the subject container is located within a pre-determined region in the captured images. Additionally, or alternatively, camera settings can be selected according to lighting conditions and / or the subject container properties to, for example, provide suitable levels of contrast between the subject container and the image background. Exemplary setup and calibration techniques for subject and template digital images are described below.

[0097] 1.1 Exposure Mode

[0098] Machine vision methods can include determining an exposure mode. For example, and not limitation, methods can include determining optimal exposure time and sensor gain for the determined exposure mode during a camera calibration period. As an example, and not by way of limitation, a manual exposure mode can be implemented. Manual exposure modes can be implemented, for example, if images are captured in an environment with constant illumination. In a manual exposure mode, the sample images of the same test target with various exposure times can be collected and compared to determine an optimal exposure time and sensor gain during the calibration time. An improved or optimal exposure time and sensor gain can be different for different applications, containers and environments. The optimal exposure time and sensor gain can be stored and selected according to the application, container, and environment. A machine vision camera can store a lot of camera jobs and a vision control software can load a camera job per target type per application in real time.

[0099] Additionally, or alternatively, an auto exposure mode can be implemented. Auto exposure modes can be implemented, for example, if images are captured in an environment with varying illumination. Auto exposure modes can be implemented in a variety of ways. For example, the camera used to capture the subject and / or template images may include an auto exposure algorithm. Commercially available camera sensors can have an auto exposure algorithm embedded in the sensor chip, for example. This option can require the camera vendor to enable the auto exposure algorithm via the camera firmware. Additionally, or alternatively, camera processor chip can be shipped with an auto exposure algorithm. For example, commercially available camera models can have an auto exposure algorithm implemented on the camera processor chip. The camera vendor can make the auto exposure algorithm available for external software to trigger the algorithm in real time.

[0100] Additionally, or alternatively, machine vision software can have an auto exposure algorithm: FIG. 1 illustrates an example auto exposure algorithm for determining optimal exposure time. The sensor gain is assumed to be fixed in the algorithm illustrated in FIG. 1. Although this disclosure describes determining exposure time and sensor gain in certain manners, this disclosure contemplates determining exposure time and sensor gain in any suitable manner. As embodied herein, machine vision methods and systems can use a digital camera to take a subject digital image of a subject container located at a pre-determined relative location from the digital camera with the determined optimal exposure time and sensor gain. As described further herein, the subj ect container can include N pre-determined element positions, and methods can include accessing the subject digital image of the subject container comprising the N pre-determined element positions.

[0101] 1.2 Color Channel Selection

[0102] Subject digital images and template digital images can be captured in full color, black and white, or any other suitable mode. Additionally or alternatively, and as embodied herein, subject digital images and template digital images can include a singlecolor channel. For example, a color channel producing highest contrast can be selected among red, green, blue (RGB) channels. The color channel can be selected, for example, to provide contrast between a container and elements in the container, as described further herein.

[0103] FIG. 2 illustrates exemplary digital images having a single-color channel. The exemplary images are of exemplary fully-loaded subject containers, each having A predetermined element positions and elements positioned in each of the respective N predetermined element positions. The images presented in FIG. 2 are template digital images for an exemplary machine vision system for a diagnostic instrument, as described further herein.

[0104] As noted above, it can be beneficial to have contrast in the subject and template digital images between the container and the elements in the container. Contrast between the container and elements to be received in the container can, for example, improve detection accuracy. For example, and not limitation, if elements to be received in the container are bright or translucent, a darker container can be preferred. Vice versa, if elements to be received in a container are dark, a bright container can be preferred.

[0105] Additionally, when images include a single-color channel, the color can be selected based on the properties of the container and / or elements to be received in the container. For example, yellow objects absorb blue light. Thus, yellow objects look like dark objects in blue channel of a color image. Yellow objects do not absorb green light. Thus, yellow objects look like bright objects in green channel of a color image.

[0106] In FIG. 2, exemplary images 201, 202, and 204 include containers having bright or translucent elements received therein, and the blue-channel image can introduce a high contrast between the containers and elements received therein. Exemplary images 203 and 205 include containers having dark elements received therein, and the green-channel image can introduce a high contrast between the containers and elements received therein. Exemplary images 201-205 depict containers of consumables for use in an exemplary diagnostic instrument, as described further herein.

[0107] Additionally or alternatively, a color channel can be selected to provide contrast between attributes of elements to be received in a container. For example and not limitation, exemplary image 206 depicts tube elements received in the pre-determined element positions of the container, with each of the tubes having a cap. As embodied herein, the tubes are translucent and include orange caps, which absorb blue light. Because the tubes re translucent, the blue-channel image can introduce a high contrast between the tube and the cap.

[0108] 1.3 Image Resolution

[0109] Subject and template digital images can have any suitable resolution. For example and not limitation, image resolution can be selected to balance processing time and accuracy. For example, images having higher resolution can provide higher detection accuracy, and images of lower resolution can be processed faster. As described further herein, in some embodiments, different subject and template image resolution can be selected depending on application.

[0110] 1.4 Gamma Correction

[0111] Gamma Correction can be used, for example, if operators will review or label subject digital images or template digital images. FIG. 3 illustrates a comparison between an original image 301 and a gamma-corrected image 302. The gamma-corrected image 302 is brighter and shows more details. Gamma Correction is known in the art. An exemplary Gamma Correction method is provided at

[0112] 1.5 Otsu Thresholding

[0113] Otsu Thresholding can be used to perform automatic image thresholding. In the simplest form, Otsu Thresholding returns a single intensity threshold that separate pixels into two classes: foreground and background. FIG. 4 illustrates a comparison between the input gamma-corrected image 401 and the binary image 402 after Otsu thresholding. On the binary image 402, black pixels are the background pixels, and white pixels are the foreground pixels. Otsu Thresholding is known in the art. An exemplary Otsu Thresholding method is provided at https: / / en.wikipedia.org / wiki / Otsu%27s_method.

[0114] 1.6 Calibration

[0115] As embodied herein, machine vision methods can include performing a calibration of a digital camera during a camera calibration period. Calibration can be used, for example, to align the relative positions of a camera and subject container so that the subject container is positioned in a desired region of a subject digital image. As embodied herein, calibration can be performed with the camera used to capture subject digital images. Calibration of the digital camera can include taking a calibration digital image of a calibration target located at a first relative location from the digital camera. Calibration of the digital camera can further include determining current coordinates of one or more calibration markers in the calibration digital image. If the current coordinates of the one or more calibration markers are within a pre-determined region of the calibration digital image, the relative positioning of the camera and the subject container is acceptable. Additionally or alternatively, at least one of the current coordinates of the one or more calibration markers in the calibration digital image can be outside of a corresponding pre-determined region of the calibration digital image. In response to the determination that the at least one of the current coordinates of the one or more calibration markers in the calibration digital image is outside of the corresponding pre-determined region of the calibration digital image, the calibration target or the digital camera can be adjusted to provide a second relative location between the target and the digital camera. The at least one of the current coordinates of the one or more calibration markers can be calculated to be within the corresponding predetermined region of the calibration digital image when the calibration target is at the second relative location.

[0116] For purpose of example, and as embodied herein, the calibration target can be a calibration target container, and the one or more calibration markers can be one or more corners of the calibration target container. The coordinates of one or more corners of a container can also be used as source anchor points for perspective transform, as described further herein. Current coordinates of one or more comers of the calibration target container can be determined, for example, using bounding box detection and template matching. Bounding box detection and template matching are described further herein with respect to perspective transform.

[0117] As embodied herein, a calibration digital image of a calibration target container can be brightness-improved by performing a gamma correction to the calibration digital image. A binary calibration digital image can be obtained by performing Otsu Thresholding on the brightness-improved calibration digital image. Each pixel in the binary calibration digital image can belong to a foreground or background. The machine vision system can detect a bounding box of the calibration target container based on projected sums on the binary calibration digital image. A search region for each comer can be determined based on the bounding box. The machine vision system can also determine current coordinates of one or more corners of the calibration target container by performing a template matching for one or more comers of the bounding box with a respective template image using the search window determined based on the bounding box. Bounding box detection and template matching are described further herein.

[0118] Causing the calibration target to be at a second relative location from the digital camera can include causing the calibration target container to be moved to the second relative location from the digital camera. Additionally or alternatively, causing the calibration target to be at a second relative location from the digital camera can include changing the position or angle of the digital camera relative to the calibration target. As an example and not by way of limitation, a camera calibration can be required to register the optimal XY coordinates for picture taking because the unit-to-unit variation in camera FOV can be noticeable.

[0119] FIG. 5 illustrates an example unit-to-unit variation in camera FOV. In image 501, the whole tray is included in the FOV of a first camera, whereas in image 502 the right edge of the tray is clipped by the FOV of a second camera.

[0120] The calibration target can be moved using any suitable method. As described further herein, methods in accordance with the disclosed subject matter can be implemented in diagnostic instruments, and the subject container or calibration target can be positioned in a field of view of the camera using a robot. As embodied herein, the robot can be used to adjust the position of the subject container or calibration target relative to the camera. For example and not limitation, the robot can adjust the tray position to left / right / forward / backward. The Z position of the subject container or calibration target can be fixed. For example, the Z position can be fixed to maintain a desired working distance between the camera and the subject container or calibration target. For example and not limitation and as embodied herein, the working distance can be between 50 mm and 500 mm. Additionally or alternatively, the working distance can be between 150 mm and 400 mm. Additionally or alternatively, the working distance can be between 200 mm and 350 mm. Additionally or alternatively, and as embodied herein, the working distance can be about 290 mm.

[0121] For purpose of example and not limitation a robot can position a calibration target at a default XYZ position. An image of the calibration target can be captured to detect the current coordinates of one or more calibration markers. For example and not limitation, the one or more calibration markers can include XY coordinates of an upper left hand comer of the calibration target in the calibration image. A pre-determined region of the acceptable XY coordinates on the image is defined. If the detected XY coordinates are outside predetermined region, an iteration process can be performed between the robot and the vision system. After an image is processed, the vision system can instruct robot control software to move the robot left / right / forward / backward by x mm. Then, the robot control software will move the robot and the calibration target to left / right / forward / backward by x mm. The machine vision system can take another picture to detect the XY coordinates at the upper left corner of the calibration target. If the XY coordinates are within the pre-determined region, the vision system can inform the robot control software to register the XY coordinates of the robot position during the picture taking for all subject containers to be imaged. Bounding box detection and template matching can be used to detect XY coordinates at the upper left comer of a calibration target, as described further below. Although this disclosure describes performing a calibration in a certain manner, this disclosure contemplates performing a calibration in any suitable manner.

[0122] For example and not limitation, when machine vision methods and systems include subject containers of different types and sizes, calibration can be performed using a calibration target or calibration target container of the largest size to be imaged. After calibration is performed and the largest container is within the field of view of the camera, smaller containers to be imaged are also captured within the field of view.

[0123] In certain embodiments, the subject container can be circular or elliptical. In such embodiments, a separate calibration target can be used to perform calibration. For example, and not limitation, a separate calibration target can be positioned in the subject container. Additionally, or alternatively, the subject container itself can be used as a calibration target, and the one or more calibration markers can include an empty container. For example, a rim edge of the empty container can be fitted to an ellipse. The center pixel coordinates of the fitted ellipse can be used to calibrate the camera position in x and y directions. The center of the fitted ellipse should be detected near a preset pixel coordinate in x and y directions. If not, the camera position may have drifted. The major / minor axis half lengths of the fitted ellipse can be used to calibrate the camera position in z direction. If the major / minor axis half lengths are not close to preset lengths, then the camera position has drifted in z direction. The orientation of the fitted ellipse can be used to calibrate the camera view angle. If the measured ellipse orientation is not close to the preset orientation, then the camera view angle has drifted.

[0124] For example, a center of the empty container can be identified by performing a least squares ellipse-fitting method on a cropped region of the calibration digital image containing the empty container. The machine vision system can also determine a major-axis half length, a minor-axis half length, and an orientation of the fitted ellipse. Causing the calibration target to be at a second relative location from the digital camera can includes adjusting a position and a view angle of the digital camera based on the calculated coordinates of the center of the empty element holder, the major-axis half length, the minoraxis half length, and the orientation of the fitted ellipse.

[0125] In such a case, to determine current coordinates of the empty element holder, the machine vision system can generate a brightness-improved calibration digital image by performing a gamma correction to the calibration digital image. The machine vision system can detect the empty element holder in the brightness-improved calibration digital image by performing a template matching with a template image of an empty element holder.

[0126] FIG. 6 illustrates an example camera calibration algorithm using an empty subject container as a calibration target. The input image can include a calibration digital image of the empty subject container located at a first relative location from the digital camera. As embodied herein, gamma correction, can be applied to the input image. Gamma correction can be used to improve image brightness as described above. As embodied herein, a template matching process can be performed to determine whether the calibration target is present in the scene. As embodied herein, the calibration target can be a subject container and the subject container can be a vessel holder for receiving one or more vessels in a diagnostic instrument.

[0127] FIG. 7 illustrates an image of an exemplary vessel holder 701 used as a calibration target. As embodied herein, template image 703 of an empty vessel holder can be used in a template matching process to determine whether the vessel holder 701 is present at the scene. In the example illustrated in FIG. 7, the size of the template image 703 is 107 pixels by 85 pixels. If the calibration target is not detected in the scene, a determination that the camera position has drifted greatly from the original field of view (FOV) can be made and an error flag for a calibration failure can be triggered.

[0128] If the calibration target (vessel holder 701 in this exemplary embodiment) is determined to be present in the scene, template matching can be used to identify a region of interest (ROI) containing the vessel holder within the input image, as represented with bounding box 702. As embodied herein, the ROI can be used in the ellipse fitting process, which can be applied to the sub-image of the original input image as identified by the ROI. The ellipse-fitting process can output the following attributes of the fitted ellipse: (i) the center coordinates that can be used for inferring a camera position drift in x and y directions, (ii) the major / minor half axis length that can be used for inferring a camera position drift in z direction, and (iii) the rotation angle that can be used for inferring a drift in camera view angle. If drift exceeds a threshold, then an error flag for a calibration failure can be triggered. The ellipse-fitting process is described in further detail below.

[0129] 1.6.1 Ellipse-Fitting of Vessel Holder

[0130] FIG. 8 illustrates example procedures for ellipse-fitting. Ellipse-fitting can be performed on a cropped subset of an original input image including a region of interest (ROI) 801. As described above, the ROI 801 can be identified using template matching. As embodied herein, the ROI 801 can, contain the vessel holder from the original image. As embodied herein, the vessel holder can be used as the calibration target. As embodied herein, the contrast within the ROI can be enhanced by applying a traditional computer vision algorithm called “Global Histogram Equalization” to the ROI image to produce a contrast-enhanced image 802.

[0131] As further embodied herein, a binary edge image 803 can be generated by applying a traditional computer vision algorithm called “Canny Edge Detection” to the contrast-enhanced image. The binary edge image 803 includes a plurality of edge segments from the ROI image of the vessel holder as depicted in white in FIG. 8. As further embodied herein, a traditional computer vision algorithm called “Connected-component Labeling” can be applied to the binary edge image. In connected-component labeling, each edge segment of the binary edge image can be isolated, and statistics of each edge can be generated. Two longest edges, one near the top center x coordinate, and the other one near the lower center x coordinate, can be extracted based on the edge statistics as shown in image 804. As embodied herein, some edge pixels in a known region of strong reflection in the ROI image can be erased. As further embodied herein, an at least squares ellipse-fitting method can be applied to the remaining edge pixels in the two longest edges. The fitted ellipse is shown in image 805.

[0132] As further embodied herein, the calculated X coordinate of the center of the ellipse can be used to adjust the camera position in X direction. The calculated Y coordinate of the center of the ellipse can be used to adjust the camera position in Y direction. The calculated major axis half-length and minor axis half-length can be used to adjust the camera position in Z direction. The calculated rotation angle can be used to adjust camera view angle.

[0133] For purpose of example and as embodied herein, the machine vision methods can be implemented in a laboratory diagnostic instrument, and the camera can be located on a robot within the laboratory diagnostic instrument. Based on the results of ellipse fitting, such as the exemplary ellipse fitting described above, the robot control software can move the robot to adjust camera position in X, Y, or Z direction and to adjust camera view angle based on the calculated ellipse properties.

[0134] Although this disclosure describes performing a camera calibration using an empty vessel holder as a calibration target, this disclosure contemplates performing a camera calibration using any suitable manner. 2 Perspective Transform

[0135] Certain embodiments can include performing a perspective transform of the subject digital image to define a transformed subject digital image. For example, the subject digital image can have perspective distortion because the subject digital image is taken by a digital camera with a tilted view angle. A perspective transform can correct the perspective distortion of the subject digital image. Performing the perspective transform of the subject digital image can includes relocating a plurality of source anchor points corresponding to the subject digital image of the subject container to respective pre-determined destination anchor points. In certain embodiments, a perspective transform matrix used for the perspective transform may be calculated based on current coordinates of the plurality of source anchor points in the subject digital image. For example, a perspective transform matrix can be selected to linearly relocate the plurality of source anchor points to the respective pre-determined destination anchor points.

[0136] As embodied herein, subject digital images can be captured using a camera that is offset from the subject container. For example, and as described further herein, an axis can be defined through a center of a subject container and perpendicular to a plan defined by a mouth of the subject container, and the camera used to capture images of the subject container can be offset from the axis. A subject digital image captured using a camera offset from the subject container can have perspective distortion. Performing a perspective transform can correct the perspective distortion in the subject digital image.

[0137] FIG. 9 illustrates a first example of a perspective transform. The original image 901 with source anchor points 906 is shown on the left. The transformed (also called “warped”) image 902 with pre-determined destination anchor points 907 is shown on the right. FIG. 10 illustrates a second example of a perspective transform of an image of a subject container 1003 having N pre-determined element positions 1004. As embodied herein, the subject container is tray, and consumables 1005 for use in a laboratory diagnostic instrument are received in each of the N pre-determined element positions 1004. The original image 1001 is taken from a camera offset from the subject container 1003. Four source anchor points 1006 are shown with circles for purpose of illustration. In the transformed (warped) image 1002 the source anchor points 1006 have been relocated to destination anchor points 1007. As described further herein, similarity scores are calculated for the transformed subject digital image.

[0138] For purpose of example, and as embodied herein, performing a perspective transform can include identifying source anchor points and applying a perspective transform matrix to relocate the source anchor points to pre-determined destination anchor points. To generate a perspective transform matrix, a first subject digital image of a subject container can be taken and source anchor points in the first subject digital image can be identified. Exemplary source anchor points 906 and 1006 are illustrated in FIG. 9 and FIG. 10. Destination anchor points for the first subject digital image can be pre-determined by an algorithm developer. For purpose of example, destination anchor points can be selected to simulate a top view image of the first subject container. For example in transformed subject digital image 1002 the container 1003 appears generally rectangular, as if the transformed subject digital image were taken from directly above the subject container 1003. Additionally, and as further embodied herein, each of the A-pre-determined positions 1004 and the elements 1005 received therein appear more uniform in size in the transformed subject digital image 1002 as compared to the original image 1001 with perspective distortion. As described further herein, similarity scores for each of the N pre-determined positions can be more accurate when the element position is compared with same element position in a transformed template digital image. Exemplary destination anchor points 907 and 1007 are illustrated as circles in FIG. 9 and FIG. 10. Given four source anchor points and four destination anchor points, a perspective transform matrix can be generated. The perspective transform matrix can then be applied to subsequent subject digital images to define transformed subject digital images.

[0139] Although this disclosure describes performing a perspective transform based on detected source anchor points and pre-determined destination anchor points, this disclosure contemplates performing a perspective transform in any suitable manner. Although perspective transform has been described with respect to a subject digital image of a subj ect container, it is to be understood that perspective transform can also be performed on a template digital image of a template container.

[0140] 2.1 Detecting Source Anchor Points

[0141] Source anchor points in a subject digital image can be identified in any suitable way. For purpose of example and as embodied herein, source anchor points can correspond to the comers of a container. For purpose of example and as embodied herein, source anchor points can be identified using bounding box detection and template matching. As described above, the same techniques that can be used to detect source anchor points for perspective transform can be used to identify one or more calibration markers for calibration.

[0142] For purpose of example and not limitation, various image processing techniques can be used to detect source anchor points in an image, including for example, gamma correction and Otsu thresholding, as described above. For example, gamma correction can be used to improve image brightness. Additionally, or alternatively, Otsu thresholding can be used to separate foreground and background pixels on the image and to generate a binary image with black and white pixels. As embodied herein, bounding box detection can be performed on a binary image to detect XY coordinates of a box bounding the container in the digital image. The four corners of the bounding box can determine a search region for each corner of the subject container. As embodied herein, template matching can be used to compare the identified search regions in the subject image with one or more template images to find the best matched location. As embodied herein, template matching can output XY coordinates of four source anchor points. As embodied herein, the four source anchor points can correspond to four corners of a container in the subject digital image.

[0143] 2.1.1 Bounding Box Detection

[0144] Bounding box detection using projected sums can be used to detect a bounding box of a container in a digital image. As embodied herein, the bounding box can be used to determine a search region for identifying source anchor points. For example, a search region can be identified for each corner of a container in a digital image.

[0145] FIG. 11 illustrates example projected sums on a binary image 1101 of a container 1102. Otsu Thresholding can be used to prepare binary images, as described above. In the binary image 1101, pixels for background may be represented by zero, and pixels for foreground may be represented by one. A plot of the projected sum in Y direction, sum of image rows per column, on the binary image is shown at the bottom 1103. A column with a high value may suggest that a foreground object exists over the column. A column with a low value may suggest that no foreground object exists over the column. A plot of the projected sum in X direction, sum of image columns per row, on the binary image is shown at the right 1104. A row with a high value may suggest that a foreground object exists over the row. A column with a low value may suggest that no foreground object exists over the column. For the projected sum profile in Y direction, a middle value can be calculated from the maximum and minimum values of the profile. Then, starting a search from the left, an index close to the middle value can be found, called XI. starting a search from the right, an index close to the middle value can be found, called X2. For the projected sum profile in X direction, another middle value can be calculated from the maximum and minimum values of the profile. Then, starting a search from the right, an index close to the middle value can be found, called Y2.

[0146] The container 1102 in the example illustrated in FIG. 11 is a rack with 6 predetermined element positions each configured to receive a tube. As embodied herein, bounding box detection can be performed on an empty container, or on a container having one or more elements therein. For example and as embodied herein, bounding box detection can be performed on a rack with tubes or without tubes. Y1 of the bounding box can be inferred from XI, X2, and Y2. Given a fixed camera view angle, the aspect ratio of the bounding box for a container is fixed. XI and X2 define the width of a bounding box. The aspect ratio of the bounding box infers the height of a bounding box. Given Y2, Y1 can be calculated using the inferred height of the bounding box. Given (XI, Yl), and (X2, Y2), a bounding box can be drawn.

[0147] 2.1.2 Template Matching for Identifying Source Anchor Points

[0148] Template matching can be used, for example, for searching and finding the location of content in a subject digital image that is similar to content depicted in a template image. For example and as embodied herein, template matching can be used to identify coordinates of one or more source anchor points in a digital image. Template matching can include comparing a template image and each sub-region within a search region of a subject digital image. For example, template matching can be visualized as sliding the template image over a subject digital image and comparing the template image and sub region of the subject digital image.

[0149] FIG. 12 illustrates an example of template matching. The template matching algorithm can find a region in the subject image 1210 that matches the template image 1230 best. In the example illustrated in FIG. 12, the size of the template image 1230 is m pixels by n pixels. As embodied herein, the algorithm can identify a search region 1220 within the target image 1210, which would be larger than the template image 1230. In the example illustrated in FIG. 12, the search region 1220 is assumed to be 2m pixels by In pixels.

[0150] FIG. 13 illustrates example computations for the template matching scenario illustrated in FIG. 12. At the first iteration, a normalized 2-dimensional (2D) correlation coefficient between a first sub-region 1310 of the search region 1220 and the template image 1230 can be calculated. The first sub-region 1310 can be identified by coordinates (0, 0) of the upper-left comer of the first sub-region 1310. At the second iteration, a normalized 2D correlation coefficient between a second sub-region 1320 of the search region 1220 and the template image 1230 can be calculated. The second sub-region 1320 can be identified by coordinates (1, 0) of the upper-left comer of the second sub-region 1320. At the (m+2) iteration, a normalized 2D correlation coefficient between a (m+2) sub-region 1330 of the search region 1220 and the template image 1230 can be calculated. The (m+2) sub-region 1330 can be identified by coordinates (0, 1) of the upper-left corner of the second sub-region 1330. At the (m+2)(w+2) iteration, a normalized 2D correlation coefficient between a ( / 7?+2)( / / +2) sub-region 1340 of the search region 1220 and the template image 1230 can be calculated. The (m+2)(w+2) sub-region 1340 can be identified by coordinates ( / ??, ri) of the upper-left corner of the second sub-region 1340. The size of the sub-regions is m pixels by n pixels, the size of the template image 1230. After calculating the normalized 2D correlation coefficients, a sub-region having a highest normalized 2D correlation coefficient can be selected as the sub-region most similar to the template image 1230. A degree of similarity can be determined based on the normalized 2D correlation coefficient of the selected sub -region.

[0151] FIG. 14 illustrates an example method for identifying source anchor points in a subject image 1401 of a subject container 1402 using template matching. FIG. 14 shows four template images 1404 and four search regions 1403, one per corner, for template matching algorithm. The search region 1403 for each comer of the subject container 1402 can be determined using bounding box detection, as described above. Within each search region, the best matched location is found by comparing the template image 1404 and a subregion at an image coordinate inside the search region 1403.

[0152] FIG. 15 illustrates an example bounding box 1504 and source anchor points 1505 identified for the subject container 1503. The left picture 1501 of FIG. 13 shows a bounding box 1504. The right picture 1502 of FIG. 13 shows the best matched location per corner as dots. The dotted locations are the source anchor points 1505.

[0153] Calibration Target

[0154] For example, and not limitation, a calibration target can be used to identify source anchor points, as described further herein. Different perspective transform techniques can be used for different applications. For example, a calibration target can be used to identify source anchor points for use with each subject digital image of a given container type. Different calibration targets can be used for different container types. In certain embodiments, the camera calibration may be performed for each of one or more container types. The coordinates of the plurality of source anchor points may be registered per container type. Additionally, or alternatively, a perspective transform matrix used for the perspective transform can be pre-calculated and applied to each subject digital image for a given container type. Different perspective transform matrixes can be calculated for different container types. A calibration target and pre-calculated perspective transform matrixes can be used, for example, when subject containers are consistently positioned within the subject digital image.

[0155] FIG. 16 illustrates an exemplary calibration target. As embodied herein, the calibration target 1602 can be an empty container with a white plate on the top. As embodied herein, the calibration target 1602 includes different sets of source anchor points corresponding to different subject containers to be imaged. For various container types, four source anchor points corresponding to four corners of the respective container are marked on the white plate. Calibration targets 1602 including source anchor points for all containers to be imaged can be referred to as universal calibration targets. The source anchor points of the calibration target 1602 can be used to define a perspective transform matrix for each of the respective container types, as outlined above.

[0156] When source anchor points are identified using a calibration target 1602, it can be important to ensure that subject containers are placed consistently within the subject digital image. Machine vision methods described herein can be implemented in diagnostic instruments. As embodied herein, subject containers to be imaged can be, for example, trays for containing consumable items, such as reaction vessels, to be used in the diagnostic instrument. As embodied herein, the diagnostic instrument can be configured such that the portion of the diagnostic instrument configured to receive the subject container can reliably position the subject container in the same position. For example, and as described further herein, subject containers can have different weights depending on, for example, the number and type of consumable items present in the subject container. As embodied herein, the portion of the diagnostic instrument configured to receive the subject container can include one or more bracing structures to reliably position subject containers of different weights without sagging.

[0157] For purpose of example and not limitation, an exemplary bracing structure 1601 for supporting a subject container in a diagnostic instrument is depicted in FIG. 16. As embodied herein, the bracing structure 1601 includes a triangle fixture with 5 contact points. As embodied herein, each of the contact points can define an L shape and can extend from the triangle fixture. The spacing and configuration of the contact points can be selected according to the container types to be imaged. As embodied herein at least two contact points can be in contact with each subject container. For purpose of example and as embodied herein, the subject container can include ledges, and the ledges can mate with corresponding contact points.

[0158] Although this disclosure describes performing a perspective transform using pre-calculated transform matrix along with a calibration target in a certain manner, this disclosure contemplates performing a perspective transform using any suitable manner.

[0159] 2.1.3 Real-time Source Anchor Point Detection

[0160] As noted above, subject containers can have different contents and different weights. For example, and as embodied herein, a subject container can be a tray with predetermined positions for receiving, for example, consumable items to be used in a diagnostic instrument. As embodied herein, the subject container can be imaged at a portion of the diagnostic instrument. Depending on the configuration of the portion of the diagnostic instrument that receives the subject container, subject containers of different weight may have varying positions. For example, and not limitation, for the same subject container type, different weights of consumables can cause the container to sag relative to the camera different amounts.

[0161] FIG. 17 illustrates an exemplary container with different amounts of a consumable item therein. As embodied herein the container is a tub, and the consumable items received therein are wash vessels. As embodied herein, the position of the container in the digital image can vary depending on the number of elements in the container. For example and illustration and not limitation, as embodied herein for a full tub of wash vessels, the upper right comer of the tub is detected at Y = 605 pixels. The origin of Y axis is at the upper left corner of an image. As embodied herein, for a half-full tub of wash vessels, the upper right comer is detected at Y = 590 pixel. As embodied herein, for an empty tub, the upper right comer is detected at Y = 560 pixel. In this exemplary embodiment, the right side of the tub moves up gradually with the decreasing wash vessels. In this exemplary embodiment the physical dimension difference in vertical direction between the full tub and the empty tub is approximately 4 mm. The different positioning of the subject container within the image can cause varying source anchor points. For example, and as embodied herein, the location of source anchor points on the subject container can vary depending on the weight of consumables. With inconsistent source anchor point locations, a single perspective transform matrix may not provide consistent perspective transform results. Further, the change in positioning as a result of different numbers of elements in the container can be different for different container types. For example, a container configured to hold heavier elements may have a larger change in position relative to the camera (e.g., more sagging) as the container is filled with elements as compared to container configured to hold lighter elements.

[0162] FIG. 18 illustrates an example subject image 1801 of a subject container 1802 having TV pre-determined positions configured to receive consumables for a laboratory diagnostic instrument. As embodied herein one side of the subject container 1802 can be mounted on arm 1803, which can support the weight of the subject container 1802. As embodied herein, the subject container 1802 can be supported in a cantilevered configuration, with one side of the subject container 1802 supported by arm 1803 and an opposing side of the subject container free of additional support. As described above, the weight of the subject container 1802 can vary depending on the amount of consumables received therein. Exemplary overlay plots 1804 illustrating the location of source anchor points detected for subject container 1802 as loaded with three different consumable amounts are depicted in FIG. 18. For illustration and not limitation, and as embodied herein, the detected source anchor points for subject container 1802 shifted in the y direction by -3 mm, and -5 mm, respectively depending on the amount of consumables received therein. Accuracy of the source anchor points, such as for example whether the source anchor point accurately reflects a comer of the subject container, can affect accuracy of the perspective transformed image. Additionally, accuracy of the perspective transformed image can affect accuracy of similarity score calculations and the determination of subject attributes of the container.

[0163] For example, and as embodied herein, source anchor points and a corresponding perspective transform matrix can be identified for each subject digital image. Detecting source anchor points and a corresponding perspective transform matrix can, for example, be used instead of a calibration target and pre-determined perspective transform matrix in applications where a subject container can be located in different positions relative to the camera. For purpose of example, identifying source anchor points and a corresponding perspective transform matrix for each subject digital image can reduce overall system cost. As embodied herein, identifying source anchor points and a corresponding perspective transform matrix for each subj ect digital image can reduce the amount of bracing and other structure that could otherwise be required to ensure consistent positioning of containers of different weights. Identifying source anchor points and a corresponding perspective transform matrix for each subject digital image is referred to herein as real-time source anchor point detection.

[0164] For purpose of example, and not limitation, a subject image of a subject container can be taken and used to identify source anchor points. A perspective transform matrix can be generated for the subject image after the source anchor points are detected. As embodied herein, a second subject image of the subject container can be taken and used to calculate similarity scores, as described further herein. For example, a first subject image can include a first color channel for identifying source anchor points, and a second subject image can include a second color channel for calculating similarity scores. A perspective transform matrix can be calculated using the first subject image and applied to the second subject image. A perspective transform can be performed on the second image using the source anchor points and perspective transform matrix calculated using the first image. The transformed second image can be used to calculate similarity scores and determine subject attributes, as described further herein.

[0165] FIG. 19 illustrates an exemplary real-time source anchor point detection algorithm for identifying source anchor points of a subject container. As embodied herein, a subject container can be a tray. For illustration and not limitation, a subject digital image of the tray can be referred to as an input tray image. As embodied herein, gamma correction can be applied to the input tray image. Gamma correction can improve image brightness, as described above. As further embodied herein, Otsu thresholding can be applied to the input image. For purpose of example and as embodied herein, Otsu thresholding can be applied to the subject image after gamma correction. As described above, Otsu thresholding can separate foreground and background pixels on the image. As embodied herein, Otsu thresholding can generate a binary image with black and white pixels. As further embodied herein, bounding box detection using projected sums can be used to detect XY coordinates for a bounding surrounding the tray.

[0166] For example, and as embodied herein, bounding box detection can be performed on the binary image. The four corners of the bounding box can be used to determine a search region for each corner of the tray. As further embodied herein, template matching can be used to identify XY coordinates of four source anchor points, each source anchor point corresponding to a comer of the tray. As embodied herein, template matching can include comparing one or more template images with a sub-region within the search region identified for a corner of the tray to find the best matched location. As further embodied herein, the processing block, “Perspective Transform Matrix Creation”, can generate a perspective transform matrix, given the identified source anchor points and destination anchor points. As embodied herein the destination anchor points can be predetermined, as described above.

[0167] 3 Template Images

[0168] Methods in accordance with the disclosed subject matter include accessing one or more template digital images. The one or more template digital images include a first region of interest (RO I) having a first attribute and one or more second ROIs having a common second attribute. The first and second ROIs of the one or more template digital images can be used to calculate similarity scores and determine attributes of the subject digital image, as described further herein. The first and second ROIs can be configured as desired and to support determinations of any suitable attribute in the subject digital image.

[0169] In certain embodiments, the one or more template digital images can include one or more perspective transformed template container digital images of a template container comprising the N pre-determined element positions. The template container can include the same N pre-determined element positions as a respective subject container for which the template container will be used to calculate similarity scores. As embodied herein, the template container can be the same container type as the subject container. Each of the N pre-determined element positions in the one or more template digital images can be associated with a region of interest (ROI) and can have a template attribute that is uniquely associated with one of the one or more template digital images. The first ROI and the one or more second ROIs can be associated with the first element position of the N predetermined element positions in the one or more perspective transformed template container digital images. The one or more perspective transformed template container digital images can additionally include ROIs having the first attribute and the common second attribute associated with each of the N pre-determined element positions.

[0170] As described further herein, digital content in the ROI having the template attribute can be compared with corresponding digital content in a transformed subject image to calculate a similarity score and determine one or more subject attributes. The template attribute that is uniquely associated with a template digital image can be any attribute of interest. For purpose of example, and as embodied herein, the template attribute can include each of the N pre-determined element positions of the template container being empty. Additionally, or alternatively, the template attribute can include having elements positioned in each of the respective N pre-determined element positions. Additionally, or alternatively, the template attribute can include each of the N pre-determined element positions having the same type of element positioned therein. For purpose of example and as embodied herein, template digital images can be perspective transformed, such as for example using the exemplary perspective transform techniques described herein.

[0171] FIG. 20 illustrates an exemplary perspective transformed template container digital image. For purpose of example, and as embodied herein the template container can be a rack 2003 for receiving consumables to be used in a laboratory diagnostic instrument. As embodied herein, the rack 2003 can include 6 pre-determined element positions 2004. For example, and as embodied herein, rack 2003 can be configured to receive a tube at each of the pre-determined element positions. As embodied herein, the template attribute uniquely associated with the template digital image 2001 can include each of the 6 predetermined element positions 2004 of the template container being empty.

[0172] As embodied herein, each of the 6 pre-determined element positions in the exemplary template image 2001 is associated with an ROI. With continued reference to FIG. 20, exemplary image 2002 depicts exemplary template image 2001 with ROI 2007 denoted in dashed line for purpose of illustration, including for exemplary pre-determined element positions 2004(a), 2004(b), and 2004(c). For purpose of example and illustration, exemplary pre-determined element position 2004(a) can be considered the first element position. However, it is to be understood that any of the pre-determined positions can be considered the first element position. As embodied herein, ROI 2007 associated with predetermined element position 2004(a) can be a first ROI having a first attribute. As embodied herein, the first ROI can contain an empty element position, and the first attribute can indicate that the element position is empty. As embodied herein, the ROIs 2007 for each of the pre-determined element positions can include the first attribute.

[0173] As described further herein, ROI can be selected according to a certain application. For example, and not limitation, the location and dimensions of the ROI can be selected according to the container application. For example, pre-determined element positions can be of different dimensions in different containers, such as to receive elements of various sizes. As embodied herein, container 2003 can be configured to receive tubes of various types at each of the pre-determined element positions.

[0174] FIG. 21 illustrates another exemplary perspective transformed template container digital image. An exemplary template digital image 2101 depicts an exemplary template container 2103. For purpose of example, and as embodied herein the template container can be a rack 2103 for receiving consumables to be used in a laboratory diagnostic instrument. As embodied herein, rack 2103 can be the same type of rack as rack 2003, and can include 6 pre-determined element positions 2104 configured to receive various types of capped or uncapped tubes at each of the pre-determined element positions. As embodied herein, the template attribute uniquely associated with the template digital image 2101 can include the presence of an element of a first element type at each of the pre-determined positions. For example, and as embodied herein, an uncapped long tube 2105 can be positioned in each of the pre-determined element positions. The template attribute associated with the template digital image 2101 can also include an uncapped tube positioned in each of the pre-determined element positions.

[0175] As embodied herein, each of the 6 pre-determined element positions in the exemplary template image 2101 is associated with an ROI. With continued reference to FIG. 21, exemplary image 2102 depicts exemplary template image 2101 with ROI 2107 denoted in dashed line for purpose of illustration, including for exemplary pre-determined element positions 2104(a), 2104(b), and 2104(c). For purpose of example, and as embodied herein ROI 2107 and ROI 2007 can be different, even though the containers 2103 and 2003 are of the same type. For purpose of example and illustration, exemplary pre-determined element position 2104(a) can be considered the first element position. However, it is to be understood that any of the pre-determined positions can be considered the first element position. As embodied herein, ROI 2107 associated with pre-determined element position 2104(a) can be a second ROI having a common second attribute. The second ROI can contain an element of a respective element type having a common second attribute. As embodied herein, the second ROI can contain a long, uncapped, tube 2105. The common second attribute can include an uncapped tube. As embodied herein, the ROIs 2107 for each of the pre-determined element positions can include the common second attribute.

[0176] FIG. 22 illustrates another exemplary perspective transformed template container digital image. An exemplary template digital image 2201 depicts an exemplary template container 2203. For purpose of example, and as embodied herein the template container can be a rack 2203 for receiving consumables to be used in a laboratory diagnostic instrument. As embodied herein, rack 2203 can be the same type of rack as rack 2003 and rack 2103, and can include 6 pre-determined element positions 2204 configured to receive various types of capped or uncapped tubes at each of the pre-determined element positions. As embodied herein, the template attribute uniquely associated with the template digital image 2201 can include the presence of an element of a second element type at each of the pre-determined positions. For example, and as embodied herein, an uncapped long-slim tube 2205 can be positioned in each of the pre-determined element positions. The template attribute associated with the template digital image 2201 can also include an uncapped tube positioned in each of the pre-determined element positions, which may be a common template attribute associated with the template digital images 2201 and 2101.

[0177] As embodied herein, each of the 6 pre-determined element positions in the exemplary template image 2201 is associated with an ROI. With continued reference to FIG. 22, exemplary image 2202 depicts exemplary template image 2201 with ROI 2207 denoted in dashed line for purpose of illustration, including for exemplary pre-determined element positions 2204(a), 2204(b), and 2204(c). For purpose of example, and as embodied herein, ROI 2207 may be different from ROI 2007 and ROI 2107, even though the container 2203 is of the same type with the containers 2003 and 2103. For purpose of example and illustration, exemplary pre-determined element position 2204(a) can be considered the first element position. However, it is to be understood that any of the pre-determined positions can be considered the first element position. As embodied herein, ROI 2207 associated with pre-determined element position 2204(a) can be a second ROI having a common second attribute. The second ROI can contain an element of a respective element type having a common second attribute. As embodied herein, the second ROI can contain an uncapped long-slim tube 2205. The common second attribute can include an uncapped tube. As embodied herein, the ROIs 2207 for each of the pre-determined element positions can include the common second attribute.

[0178] FIG. 23 illustrates another exemplary perspective transformed template container digital image. An exemplary template digital image 2301 depicts an exemplary template container 2303. For purpose of example, and as embodied herein the template container can be a rack 2303 for receiving consumables to be used in a laboratory diagnostic instrument. As embodied herein, rack 2303 can be the same type of rack as rack 2003, rack 2103, and rack 2203. Rack 2304 may include 6 pre-determined element positions 2204 configured to receive various types of capped or uncapped tubes. As embodied herein, the template attribute uniquely associated with the template digital image 2301 can include the presence of an element of a third element type at each of the pre-determined positions.

[0179] For example, and as embodied herein, a short tube 2305 can be positioned in each of the pre-determined element positions. The template attribute associated with the template digital image 2301 can also include an uncapped tube positioned in each of the pre-determined element positions, which can be a common template attribute associated with the template digital images 2101, 2201, and 2301.

[0180] As embodied herein, each of the 6 pre-determined element positions in the exemplary template image 2301 is associated with an ROI. With continued reference to FIG. 23, exemplary image 2302 depicts exemplary template image 2301 with ROI 2307 denoted in dashed line for purpose of illustration, including for exemplary pre-determined element positions 2304(a), 2304(b), and 2304(c). For purpose of example, and as embodied herein, ROI 2307 may be different from ROI 2007, ROI 2107, or ROI 2207, even though the container 2303 is of the same type with the containers 2003, 2103, and 2203. For purpose of example and illustration, exemplary pre-determined element position 2304(a) can be considered the first element position. However, it is to be understood that any of the pre-determined positions can be considered the first element position. As embodied herein, ROI 2307 associated with pre-determined element position 2304(a) can be a second ROI having a common second attribute. The second ROI can contain an element of a respective element type having a common second attribute. As embodied herein, the second ROI can contain an uncapped short tube 2205. The common second attribute can include an uncapped tube. As embodied herein, the ROIs 2307 for each of the pre-determined element positions can include the common second attribute.

[0181] The one or more template digital images can comprise k second ROIs, each containing an element of a respective element type having the common second attribute, wherein Aus a number of possible element types. For purpose of example and as embodied herein, the one or more template digital images can comprise three second ROIs corresponding to ROIs 2107, 2207, and 2307 for long, long-slim, and short tubes, respectively. Although a set of template images representing three different element types is shown for purpose of illustration, it is to be understood that any desirable number of element types and ROIs can be selected.

[0182] Although perspective transformed template container images have been described, alternative template image configurations can be used with the machine vision methods disclosed herein. For example, in certain embodiments, the template images may not be perspective transformed. Additionally, or alternatively, a single ROI can be used to calculate similarity scores for each of the N pre-determined element positions in a subject container, as described further herein.

[0183] For purpose of example, digital content in a first ROI having a first attribute can be compared with corresponding digital content in each of the N pre-determined element positions of the subject container. For example, digital content in ROI 2007 associated with element position 2004(a) can be compared with corresponding digital content in each of the 6 pre-determined element positions in the subject container. Additionally, or alternatively, digital content in a second ROI having a common second attribute can be compared with corresponding digital content in each of the N pre-determined element positions of the subject container. For example, digital content in ROI 2107 associated with element position 2104(a) can be compared with corresponding digital content in each of the 6 predetermined element positions in the subject container. Additionally, or alternatively, first and second ROI can include first and common second attributes, respectively without being associated with a container.

[0184] In certain embodiments, for example if a template image is not perspective transformed, the template image can include a number of ROIs associated with different template attributes. For purpose of example, and as embodied herein a single template image that is not perspective transformed can include a certain template container that is of same type with the containers 2003, 2103, 2203, and 2303. A first element position among the 6 element positions of the certain template container may be empty. A first ROI associated with a first template attribute of an element position being empty may be prepared for the first element position among the 6 element positions of the certain template container. An element of a first type can be positioned at a second element position of the certain container in the template image. For example, an uncapped long tube may be positioned at a second element position among the 6 element positions of the certain container. A second ROI associated with second template attributes may be prepared for the second element position among the 6 element positions. The second template attributes may include an element position being filled with a long tube and the element position being filled with an uncapped tube. An element of a second type can be positioned at a third element position of the certain container in the template image.

[0185] For example, an uncapped long-slim tube may be positioned at a third element position among the 6 element positions. A third ROI associated with third template attributes may be prepared for the third element position among the 6 element positions. The third template attributes may include an element position being filled with a long-slim tube and the element position being filled with an uncapped tube. An element of a third type can be positioned at a fourth element position of the certain container in the template image. For example, an uncapped short tube can be positioned at a fourth element position among the 6 element positions. A fourth ROI associated with fourth template attributes may be prepared for the fourth element position among the 6 element positions. The fourth template attributes may include an element position being filled with a short tube and the element position being filled with an uncapped tube. Although a single template image having a number of ROIs associated with different element positions of a certain container is described, it is to be understood that other configurations can be used. For example, a single template image can include a number of ROIs associated with multiple different template containers and pre-determined element positions.

[0186] As embodied herein, regions of interest for the template digital image can be pre-determined. For example, regions of interest can be pre-determined during development, setup, or calibration. For example, and as embodied herein, ROI can be selected to depict a subject attribute to be determined. For example, and as embodied herein, if a subject attribute of the subject digital image to be determined includes whether a predetermined element position includes an element, an ROI for the respective pre-determined element position in the template container can be selected to depict an element or portion thereof in the pre-determined position. Additionally, or alternatively, an ROI can be selected to depict the pre-determined position empty, without an element received therein. Additionally or alternatively, an ROI can be selected to depict a number of elements received in a respective pre-determined position. Although exemplary ROI are described, it is to be understood that ROI can be selected to depict any attributes of interest.

[0187] As embodied herein, ROIs in the template image can be used to crop the template image. For example, the portion of the template image corresponding to an ROI can be cropped, and digital content in the transformed subject digital image can be compared with digital content in the cropped portion of the template image corresponding to the associated ROI.

[0188] Certain embodiments can include performing a perspective transform of the subject digital image. Exemplary perspective transform techniques are described above. In the transformed subject digital image, each of the N pre-determined element positions of the subject container and any elements positioned therein can be distorted in geometry. For example, and as embodied herein, the perspective transform process can result in distortion. Additionally, in the transformed subject digital image, each of the N pre-determined element positions can be distorted differently. For example, distortion in the transformed subject digital image can differ depending on the location of the pre-determined position within the image. Additionally, or alternatively, a subject digital image can include variation in reflection, such as reflection from flash, at different locations within the image.

[0189] For example, and as embodied herein, subject digital images taken using a camera offset from or angled relative to the subject container and with LED illumination of the subject container can include variation in reflection from the LEDs at each of the N predetermined positions. For example, the elements closer to the camera can have stronger reflection, and the elements farther from the camera can have weaker reflection.

[0190] In view of the potential variation at the different pre-determined element positions within the subject digital image, calculating a similarity score using a template image not having ROI with a common template attribute uniquely associated with the template image may not be robust enough to determine subject attributes for each predetermined element position in the subject container. For example, comparing digital content in an ROI of a template image associated with a first pre-determined element position with digital content in a subject digital image associated with a second, different, pre-determined element position, can result in inaccurate determinations. For example, the ROI of the template image associated with the first pre-determined element position may have been distorted differently during perspective transform than the digital content in the subject digital image associated with a second, different pre-determined element position.

[0191] Methods described herein can provide improved performance and address the challenges outlined above. As embodied herein, for each of the pre-determined element positions of the subject container, an ROI in a template image can be defined for a corresponding one of the pre-determined element positions.

[0192] 4 Calculating Similarity Scores

[0193] Methods in accordance with the disclosed subject matter further include calculating, for a first element position among the N pre-determined element positions in the subj ect digital image a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image. Methods further include calculating for the first element position, in response to a determination that the one or more first subject attributes associated with the first element position do not include the first attribute, one or more second similarity scores for the one or more second ROI having a common second attribute. In certain embodiments, subject similarity scores can be calculated for each of the N pre-determined element positions in the subject digital image of the subject container. For example, in certain embodiments, the subject image can be perspective transformed, and methods can include calculating, for each of the N predetermined element positions in the transformed subject digital image of the subject container, a similarity score for at least one of the one or more template digital images. In accordance with the disclosed subject matter, similarity scores are calculated comparing digital content in an associated ROI of the at least one template digital image and corresponding digital content in the transformed subject digital image. Comparing digital content in an associated ROI of the one or more template digital image and corresponding digital content in the subject digital image can be performed using any suitable technique. For purpose of example, template matching can be used to compare digital content in a template digital image and corresponding digital content in a subject digital image. Additionally, or alternatively, , dynamic template matching can be used. For example, dynamic template matching can be used to compare digital content in a template digital image and corresponding digital content in a perspective transformed subject digital image. As used herein, dynamic template matching refers to calculating similarity scores for each of the N pre-determined element positions in the transformed subject digital image of the subject container by comparing digital content in an associated ROI of the at least one template digital image and corresponding digital content in the transformed subject digital image. For example, dynamic template matching can include calculating a similarity score for a first pre-determined element position in the transformed subject digital image by comparing content in an ROI associated with the first predetermined element position of a template container with corresponding digital content associated with the first pre-determined element position of the transformed subject digital image. Dynamic template matching can further include calculating a similarity score for a second pre-determined element position in the transformed subject digital image by comparing content in an ROI associated with the second pre-determined element position of the template container with corresponding digital content associated with the second predetermined element position of the transformed subject digital image.

[0194] In certain embodiments, a pre-determined search region corresponding to a template digital image of the one or more template digital images can be defined for each of the N pre-determined element positions in the subject digital image. Each pre-determined search region can have one or more sub-regions of a size of the associated ROI. To calculate a similarity score of a certain element position among the N pre-determined element positions in the transformed subject digital image of the subject container for a first template digital image, an ROI within the first template digital image associated with the certain element position may be identified. Also, a search region corresponding to the first template digital image that is pre-defined for the certain element position may be identified. All subregions of the size of the associated ROI within the identified search region may be identified.

[0195] For example, (m+2)(n+2) sub-regions are identified within the search region 1220 in FIG. 13. Then, a normalized 2-dimensional (2D) correlation coefficient between each of the identified sub-regions and the associated ROI may be calculated. Finally, a highest normalized 2D correlation coefficient among the calculated normalized 2D correlation coefficients may be determined as the similarity score of the certain element position for the first template digital image. In certain embodiments one or more subject attributes are determined for each of the N pre-determined element positions in the transformed subject digital image of the subject container based on the calculated similarity scores.

[0196] As an example, and not by way of limitation, to determine one or more subject attributes for a first element position in the transformed subject digital image of the subject container, similarity scores can be calculated for the first element position for each of one or more template images. For calculating a similarity score of first element position for a certain template image, an ROI of first element position in the certain template image can be accessed. The similarity score may be calculated by comparing the ROI of a first element position in the certain template image with the first element position in the transformed subject digital image. The subject attributes of the first element position can be determined based on the calculated one or more similarity scores, each of which is calculated for each of the one or more template images. Although this disclosure describes determining one or more subject attributes of an element position in the transformed subject image based on one or more similarity scores calculated with one or more template images using dynamic template matching in a certain manner, this disclosure contemplates determining one or more subject attributes of an element position in the transformed subject image based on one or more similarity scores calculated with one or more template images using dynamic template matching in any suitable manner.

[0197] 5 Determining Subject Attributes

[0198] Methods in accordance with the disclosed subject matter further include determining subject attributes based on similarity scores. Methods include determining whether one or more first subject attributes associated with the first element position include a first attribute based on a first similarity score and determining whether the first subject attributes associated with the first element position include a common second attribute based on second similarity scores.

[0199] As embodied herein, subject attributes of a certain element position among the N pre-determined element positions in a subj ect digital image of a subj ect container may include one of two mutually exclusive primary sets of attributes. A certain primary set of attributes among the two primary sets of attributes may also be divided into two mutually exclusive secondary sets of attributes. One of the two mutually exclusive secondary sets of attributes may include a common attribute, while the other secondary set of attributes may not include the common attribute.

[0200] As embodied herein, a primary set of attributes that the subject attributes of the certain element position include may be determined by calculating a first similarity score of the certain element with a first ROI that belongs to one of the two mutually exclusive primary sets of attributes. Although this disclosure describes determining a primary set of attributes that belongs to subject attributes of an element position in a certain manner, this disclosure contemplates determining a primary set of attributes that belongs to subject attributes of an element position in any suitable manner.

[0201] As embodied herein, when the subject attributes are determined to include the certain primary set of attributes, a secondary set of attributes among the two mutually exclusive secondary sets of attributes that belongs to the subject attributes of the certain element position may be determined based on similarity scores of the certain element position for a plurality of second ROIs having a common attribute. One of the two mutually exclusive secondary sets of attributes may include the common attribute. The other secondary set of attributes may not include the common attribute. If any one of the calculated similarity scores of the certain element position for the plurality of second ROIs is greater than or equal to its respective threshold, the subject attributes may be determined to include the common attribute. If none of the calculated similarity scores of the certain element position for the plurality of second ROIs is greater than or equal to its respective threshold, the subject attributes may be determined not to include the common attribute. Although this disclosure describes determining whether subject attributes of an element position include a common attribute in a certain manner, this disclosure contemplates determining whether subject attributes of an element position include a common attribute in any suitable manner.

[0202] As embodied herein, the machine vision system may access one or more template digital images including a first region of interest (RO I) having a first attribute and one or more second ROIs having a common second attribute. A pre-determined search region corresponding to an ROI may be defined for a first element position among the N pre-determined element positions in the subject digital image. The pre-determined search region may have one or more sub-regions of a size of the ROI. As embodied herein, determining whether subject attributes of a certain element position among the N pre-determined element positions in a subject digital image of a subject container includes a first primary set of attributes among two mutually exclusive primary sets of attributes may comprise calculating a similarity score of the certain element position for a first ROI having a first attribute and comparing the calculated similarity score and a pre-determined threshold associated with the first ROI. For purpose of example and illustration and not limitation, the machine vision system may calculate for the certain element position a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image. The corresponding digital content in the subject digital image may be associated with the certain element position. In certain embodiments, the first ROI may contain an empty element position. The first attribute may indicate the element position being empty. One primary set of attributes among the two mutually exclusive primary sets of attributes may include the first attribute. The other primary set of attributes may not include the first attribute.

[0203] The machine vision system may determine whether subject attributes associated with the certain element position include the first attribute based on the first similarity score. Consequently, the machine vision system may determine which primary set of attributes belongs to the subject attributes associated with the certain element position. The system determines whether one or more first subject attributes associated with the first element position include the first attribute. As embodied herein, the system determines whether the one or more first subject attributes associated with the first element position indicate that the first element position is empty.

[0204] FIG. 24 illustrates exemplary comparison of an element position with an ROI containing a portion of an empty tube holder. Image 2401 depicts a portion of an exemplary subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 2403 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 2403 can be configured to receive different types of tubes. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 2401 depicts 3 pre-determined element positions 2404, including exemplary pre-determined element positions 2404(a), 2404(b), and 2404(c) configured to receive various types of tubes. In the example illustrated in image 2401, the element position 2404(a) is filled with an uncapped short tube 2405, and the element positions 2404(b) and 2404(c) are empty.

[0205] As embodied herein, the primary set of mutually exclusive attributes includes whether an element position is empty or not. An exemplary first ROI 2420 having a first attribute is shown. As embodied herein, the first attribute indicates the element position being empty, and ROI 2420 may contain a portion of an empty element position. The ROI 2420 may be associated with a template attribute that includes an element position being empty. As embodied herein the subject image depicted in image 2401 is not perspective transformed, and the ROI 2420 may be compared with a search region of each respective element position 2404 of the subject container to determine whether a subject attribute associated the respective element position 2404 includes the element position being empty.

[0206] In the example illustrated in FIG. 24, a similarity score between a search region 2410 associated with the element position 2404(b) corresponding to the ROI 2420 and the ROI 2420 is calculated. A size of the search region 2410 may be larger than a size of the ROI 2420. In the example illustrated in FIG. 24, the search region 2410 may have twice the size of the ROI 2420. In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tube holder, can be generated per sub-region within the search region 2410 with the ROI 2420. For the search region 2410, a maximum normalized correlation coefficient, y tube holder max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 2410 can be found. The y tube holder max estimates the highest similarity between the ROI 2420 and a sub-region of the same size within the search region 2410. For each element location 2404 in the non-pespective-transformed subject digital image 2401 including 2404(a), 2404(b) and 2404(c), y_tube_holder_max may be found as a similarity score. If y tube holder max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include the element position being empty. If y tube holder max is smaller than the pre-determined threshold, subject attributes of the element position may be determined to include the element position being not empty, and further subject attributes of the element position may be determined.

[0207] In response to a determination that the subject attributes associated with the certain element position include the certain primary set, for example, the element position being not empty, the machine vision system may determine which secondary set of attributes among two mutually exclusive secondary sets of attributes belongs to the subject attributes.

[0208] As embodied herein, the machine vision system calculates for the certain element position one or more second similarity scores for one or more second ROIs having a common second attribute. The common second attribute may be any attribute of interest to be determined and that is shared among all of the one or more second ROIs. For example, the one or more template digital images can comprise k second ROIs and each of the k second ROIs may contain an element of a respective element type having the common second attribute, with k being a number of possible element types. The machine vision system may determine whether the subject attributes associated with the certain element position include the common second attribute based on the second similarity scores. For example, the machine vision system may determine that the subject attributes associated with the certain element position do not include the common second attribute when the second similarity scores indicate that none of the k element types having the common second attribute are present at the certain element position. In certain embodiments, the common second attribute may include the element position being loaded by an uncapped tube. The one or more second ROIs can include uncapped tubes of each of k tube types, a determination that a position does not include an uncapped tube can be made when second similarity scores indicate that none of the k types of uncapped tubes are present at the position. As embodied herein, a determination that a position does not include an uncapped tube can be used to infer that a position includes a capped tube. In certain embodiments, the machine vision system may further determine which of the k element types is present at the certain element position based on the second similarity scores.

[0209] FIG. 25 illustrates an exemplary comparison of an element position with an exemplary second ROI containing a portion of an element of a first type having the common second attribute. As embodied herein, the first element type can be a long tube 2505, and the common second attribute can be the tube being uncapped. Image 2501 depicts a portion of an exemplary subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 2503 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 2503 can be configured to receive different types of tubes. Rack 2503 may be of same type of rack 2403. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 2501 depicts 3 pre-determined element positions 2504, including exemplary pre-determined element positions 2504(a), 2504(b), and 2504(c) configured to receive various types of tubes. In the example illustrated in image 2501, the element positions 2504(a), 2504(b) and 2504(c) are filled with elements of a first type. As embodied herein, the first element type can be an uncapped long tube 2505. Exemplary second ROI 2520 may contain a portion of an uncapped long tube positioned in an element position. The ROI 2520 may be associated with a template attribute that includes an element position being loaded with a long tube. ROI 2520 has a common second attribute. As embodied herein, the common second attribute of ROI 2520 includes an element position being loaded with an uncapped tube. As embodied herein the subject image depicted in image 2501 is not perspective transformed, and the ROI 2520 may be compared with search region of each element position 2504 of the subject container to determine whether subject attribute associated the element position 2504 include the element position being loaded with an uncapped tube. In the example illustrated in FIG. 25, a similarity score between a search region 2510 associated with the element position 2504(a) corresponding to the ROI 2520 and the ROI 2520 is calculated. A size of the search region 2510 may be larger than a size of the ROI 2520. In the example illustrated in FIG. 25, the search region 2510 may have twice the size of the ROI 2520. In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used.

[0210] A normalized 2D correlation coefficient, y tubeLong, can be generated per sub-region within the search region 2510 with the ROI 2520. For the search region 2510, a maximum normalized correlation coefficient, y tubeLong max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 2510 can be found. y tubeLong max estimates the highest similarity between the ROI 2520 and a sub-region of the same size within the search region 2510. For each element location 2504 in the non-perspective-transformed subject digital image 2501 including 2504(a), 2504(b) and 2504(c), y_tubeLong_max may be found as a similarity score. If y tubeLong max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a long tube and (ii) the element position being loaded with an uncapped tube.

[0211] FIG. 26 illustrates exemplary comparison of an element position with an exemplary second ROI containing a portion of an element of a second type having the common second attribute. As embodied herein, the second element type can be uncapped long-slim tube 2605 and the common second attribute can be the tube being uncapped. Image 2601 depicts a portion of an exemplary subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 2603 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 2603 can be configured to receive different types of tubes. Rack 2603 may be of same type of rack 2403 and rack 2503. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 2601 depicts 3 pre-determined element positions 2604, including exemplary pre-determined element positions 2604(a), 2604(b), and 2604(c) configured to receive various types of tubes. In the example illustrated in image 2601, the element positions 2604(a), 2604(b) and 2604(c) are filled with uncapped long-slim tubes.

[0212] Exemplary second ROI 2620 may contain a portion of an uncapped long- slim tube positioned in an element position. The second ROI 2620 has a common second attribute. As embodied herein, the common second attribute includes an element position being loaded with an uncapped tube. As embodied herein, the subject image depicted in image 2601 is not perspective transformed, and the ROI 2620 may be compared with search region of each element position 2604 of the subject container to determine whether subject attribute associated the element position 2604 include the element position being loaded with an uncapped tube. In the example illustrated in FIG. 26, a similarity score between a search region 2610 associated with the element position 2604(b) corresponding to the ROI 2620 and the ROI 2620 is calculated. A size of the search region 2610 may be larger than a size of the ROI 2620. In the example illustrated in FIG. 26, the search region 2610 may have twice the size of the ROI 2620. In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tubeLongSlim, can be generated per sub-region within the search region 2610 with the ROI 2620. For the search region 2610, a maximum normalized correlation coefficient, y tubeLongSlim max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 2610 can be found. The y tubeLongSlim max estimates the highest similarity between the ROI 2620 and a sub-region of the same size within the search region 2610. For each element location 2604 in the non-pespective-transformed subject digital image 2601 including 2604(a), 2604(b) and 2604(c), y_tubeLongSlim_max may be found as a similarity score. If y tubeLongSlim max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a long-slim tube and (ii) the element position being loaded with an uncapped tube.

[0213] FIG. 27 illustrates exemplary comparison of an element position with an exemplary second ROI containing a portion of an element of a third type having the common second attribute. As embodied herein, the third element type can be a short tube, and the common second attribute can be the tube being uncapped. Image 2701 depicts a portion of an exemplary subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 2703 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 2703 can be configured to receive different types of tubes. Rack 2703 may be of same type of rack 2403, rack 2503, and rack 2603. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 2701 depicts 3 pre-determined element positions 2704, including exemplary pre-determined element positions 2704(a), 2704(b), and 2704(c) configured to receive various types of tubes. In the example illustrated in image 2701, the element positions 2704(a) is filled with an uncapped short tube, and the element positions 2704(b) and 2704(c) are empty.

[0214] Exemplary ROI 2720 may contain a portion of an uncapped short tube positioned in an element position. The second ROI 2720 has a common second attribute. As embodied herein, the common second attribute includes an element position being loaded with an uncapped tube. As embodied herein, the subject image depicted in image 2701 is not perspective transformed, and the ROI 2720 may be compared with search region of each element position 2704 of the subject container to determine whether subject attribute associated the element position 2704 include the element position being loaded with an uncapped tube. In the example illustrated in FIG. 27, a similarity score between a search region 2710 associated with the element position 2704(a) corresponding to the ROI 2720 and the ROI 2720 is calculated. A size of the search region 2710 may be larger than a size of the ROI 2720.

[0215] In the example illustrated in FIG. 27, the search region 2710 may have twice the size of the ROI 2720. In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tubeShort, can be generated per sub-region within the search region 2710 with the ROI 2720. For the search region 2710, a maximum normalized correlation coefficient, y tubeShort max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 2710 can be found. The y tubeShort max estimates the highest similarity between the ROI 2720 and a sub- region of the same size within the search region 2710. For each element location 2704 in the non-pespective-transformed subject digital image 2701 including 2704(a), 2704(b) and 2704(c), y tubeShort max may be found as a similarity score. If y tubeShort max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a short tube and (ii) the element position being loaded with an uncapped tube.

[0216] As embodied herein, the subject attributes associated with the certain element position may be determined to include the element position being not empty based on a calculated similarity score of the certain element position with a first ROI containing a portion of an empty element position. If none of the similarity scores of the certain element position calculated for k second ROIs having the common template attributes is greater than or equal to its respective threshold, the subject attributes associated with the certain element position may be determined not to include the element position being loaded with an uncapped tube. As embodied herein, the subject attributes associated with the certain element position may be determined to include the element position being loaded with a capped tube if it is determined that the element position does not include any of the k uncapped tubes. For example, and not limitation, if similarity scores of a certain position of the subject container for ROIs 2520, 2620, and 2720 are less than their respective thresholds, the subject attributes associated with the certain position may be determined to include the element position being loaded with a capped tube.

[0217] In certain embodiments, the subject digital image can be perspective transformed, and the one or more template digital images may include one or more perspective transformed template container digital images of a template container comprising the N pre-determined element positions. Each of the N pre-determined element positions on a perspective transformed template container digital image may be associated with a ROI and have a template attribute uniquely associated with one of the one or more template digital images. The first ROI and the one or more second ROIs may be associated with the certain element position of the N pre-determined element positions in the one or more perspective transformed template container digital image. The one or more perspective transformed template container digital images may include a first perspective transformed template container digital image having each of the N pre-determined element positions empty. The one or more perspective transformed template container digital images may include a second perspective transformed template container digital image having each of the N pre-determined element positions loaded with an element of a respective element type having the common second attribute.

[0218] FIG. 28 illustrates an example method using dynamic template matching to detect capped tubes. As embodied herein, the exemplary method depicted in FIG. 28 includes Gamma correction and perspective transform of a subject image of a subject container. As embodied herein, a first similarity score can be calculated using dynamic template matching as indicated by processing block, “Dynamic Template Matching to Check Empty Holder.” As embodied herein, the “Dynamic Template Matching to Check Empty Holder” processing block may be applied to the perspective transformed subject image. As embodied herein, the first similarity score and a pre-determined threshold can be used to identify if a pre-determined element position is empty or occupied. If a predetermined element position is empty, then the pre-determined element position t may be determined to be empty. Thus, an index of this position may be set to “-1.” If a slot is occupied, then a second similarity score for the position can be determined. As embodied herein, the second similarity score can be calculated using dynamic template matching as indicated by the processing block, “Dynamic Template Matching to Check Uncapped Tube.” As embodied herein, the “Dynamic Template Matching to Check Uncapped Tube” processing block can be applied to detect whether the position includes an uncapped test tube. For example, and not limitation, three template images from three tube types, long, long-slim, and short, may be used to detect an uncapped test tube per slot. Then, a predetermined threshold per tube type may be used to identify if a slot has an uncapped test tube. If a slot has an uncapped test tube, then the index of this slot may be set 0. If a slot does not have an uncapped test tube, then this slot may be determined to have a capped test tube, and the index of this slot may be set 1.

[0219] FIG. 29 illustrates exemplary dynamic template matching with a perspective transformed template container digital image having each of the N pre-determined element positions empty. Image 2901 depicts a portion of an exemplary perspective transformed subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 2903 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 2903 can be configured to receive different types of tubes. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 2901 depicts 3 pre-determined element positions 2904, including exemplary pre-determined element positions 2904(a), 2904(b), and 2904(c) configured to receive various types of tubes. In the example illustrated in image 2901, the element position 2904(a) is filled with an uncapped short tube 2905, and the element positions 2904(b) and 2904(c) are empty.

[0220] Image 2902 shows a portion of an exemplary perspective transformed empty template container digital image depicting a portion of a template container 2923 comprising the N pre-determined element positions. For purpose of example and as embodied herein, image 2902 can be used to calculate similarity scores for the pre-determined element positions 2904 configured to receive tubes of various types. Each of the N pre-determined element positions in the template container digital image is associated with an ROI 2920 and has a template attribute uniquely associated with the template digital image. As embodied herein, an ROI 2920 may contain a portion of an empty element position. The template attribute may include that the element position is empty. For purpose of example and as embodied herein, dashed boxes are used to illustrate exemplary regions of interest for each of the N pre-determined element positions in the template container digital image, including for ROIs 2920(a), 2920(b), and 2920(c) corresponding to the pre-determined element positions 2904(a), 2904(b), and 2904(c) in the transformed subject digital image. As embodied herein, the portions of the template digital image 2902 associated with ROI can be cropped and used for comparison with corresponding digital content in the transformed subject digital image 2901.

[0221] As embodied herein, the transformed subject digital image in the image 2901 may include a pre-determined search region 2910 associated with the element position 2904(b) corresponding to the template digital image in the image 2902. The center coordinates of the pre-determined search region 2910 and the ROI 2920(b) in the template digital image in the image 2902 corresponding with position 2904(b) may be identical. A size of the pre-determined search region 2910 may be larger than a size of the ROI 2920(b). In the example illustrated in FIG. 29, the search region 2910 may have twice the size of the ROI 2920(b). In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tube holder, can be generated per sub-region within the search region 2910 with the ROI 2920(b). For the search region 2910, a maximum normalized correlation coefficient, y tube holder max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 2910 can be found. The y tube holder max estimates the highest similarity between the ROI 2920(b) and a sub-region of the same size within the search region 2910. For each element location 2904 in the perspective transformed subject digital image in image 2901, y tube holder max may be found as a similarity score with its corresponding ROI.

[0222] For example, a similarity score of element position 2904(a) may be calculated between a search region associated with the element position 2904(a) corresponding to the template digital image in the image 2902 and ROI 2920(a). A similarity score of element position 2904(b) may be calculated between the search region 2910 and ROI 2920(b). A similarity score of element position 2904(c) may be calculated between a search region associated with the element position 2904(c) corresponding to the template digital image in the image 2902 and ROI 2920(c). If y tube holder max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include the element position being empty. If y tube holder max is smaller than the pre-determined threshold, subject attributes of the element position may be determined to include the element position being not empty, and further subject attributes of the element position may be determined.

[0223] In response to a determination that the subject attributes associated with the certain element position do not include the first attribute, the machine vision system may determine which secondary set of attributes among two mutually exclusive secondary sets of attributes belongs to the subject attributes. As embodied herein, if a certain element position is determined not to be empty, one or more second similarity scores can be calculated to determine if the element position includes a common second attribute. The machine vision system may calculate for the certain element position one or more second similarity scores for the one or more second ROIs having a common second attribute. As embodied herein, the one or more template digital images can comprise k second ROIs. Each of k second ROIs may contain an element of a respective element type having the common second attribute, with k being the number of possible element types. The machine vision system may determine whether the subject attributes associated with the certain element position include the common second attribute based on the second similarity scores. The machine vision system may determine that the subject attributes associated with the certain element position do not include the common second attribute when the second similarity scores indicate that none of the k element types having the common second attribute are present at the certain element position. In certain embodiments, the common second attribute may include the element position being loaded by an uncapped tube. In certain embodiments, the machine vision system may further determine which of the k element types is present at the certain element position based on the second similarity scores.

[0224] FIG. 30 illustrates exemplary dynamic template matching with an exemplary perspective transformed template container digital image having each of the N predetermined element positions loaded with an element of a respective element type having the common second attribute. As embodied herein, the container in the perspective transformed template container digital image is fully-loaded with long tubes 3005, and the common second attribute includes the tubes being uncapped. Image 3001 depicts a portion of an exemplary perspective transformed subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 3003 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 3003 can be configured to receive different types of tubes. Rack 3003 may be of same type of rack 2903. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 3001 depicts 3 pre-determined element positions 3004, including exemplary pre-determined element positions 3004(a), 3004(b), and 3004(c) configured to receive various types of tubes. In the example illustrated in image 3001, the element position 3004(a), 3004(b), and 3004(c) are loaded with uncapped long tubes 3005. Image 3002 shows a portion of an exemplary perspective transformed template container digital image depicting a portion of a template container 3023 comprising the N pre-determined element positions loaded with uncapped long tubes. For purpose of example and as embodied herein, image 3002 can be used to calculate similarity scores for the pre-determined element positions 3004 configured to receive tubes of various types. Each of the N pre-determined element positions in the template container digital image is associated with an ROI 3020 and has a template attribute uniquely associated with the template digital image. As embodied herein, an ROI 3020 may contain a portion of an element position loaded with an uncapped long tube 3005. The template attribute may include that the element position is loaded with a long tube. The ROI 3020 may also be associated with a common template attribute that includes that the element position is loaded with an uncapped tube. For purpose of example and as embodied herein, dashed boxes are used to illustrate exemplary regions of interest for each of the N pre-determined element positions in the template container digital image, including for ROIs 3020(a), 3020(b), and 3020(c) corresponding to the pre-determined element positions 3004(a), 3004(b), and 3004(c) in the transformed subject digital image. As embodied herein, the portions of the template digital image 3002 associated with ROI can be cropped and used for comparison with corresponding digital content in the transformed subject digital image 3001.

[0225] As embodied herein, the transformed subject digital image in the image 3001 may include a pre-determined search region 3010 associated with the element position 3004(a) corresponding to the template digital image in the image 3002. The center coordinates of the pre-determined search region 3010 and the ROI 3020(a) in the template digital image in the image 3002 corresponding with position 3004(a) may be identical. A size of the pre-determined search region 3010 may be larger than a size of the ROI 3020(a). In the example illustrated in FIG. 30, the search region 3010 may have twice the size of the ROI 3020(a). In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tubeLong, can be generated per sub-region within the search region 3010 with the ROI 3020(a). For the search region 3010, a maximum normalized correlation coefficient, y tubeLong max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 3010 can be found. The y tubeLong max estimates the highest similarity between the ROI 3020(a) and a subregion of the same size within the search region 3010. For each element location 3004 in the perspective transformed subject digital image in image 3001, y tubeLong max may be found as a similarity score with its corresponding ROI. For example, a similarity score of element position 3004(a) may be calculated between the search region 3010 and ROI 3020(a). A similarity score of element position 3004(b) may be calculated between a search region associated with the element position 3004(c) corresponding to the template digital image in the image 3002 and ROI 3020(b). A similarity score of element position 3004(c) may be calculated between a search region associated with the element position 3004(c) corresponding to the template digital image in the image 3002 and ROI 3020(c). If y tubeLong max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a long tube and (ii) the element position being loaded with an uncapped tube.

[0226] FIG. 31 illustrates exemplary dynamic template matching with an exemplary perspective transformed template container digital image having each of the N predetermined element positions loaded with an element of another respective element type having the common second attribute. As embodied herein, the container in the perspective transformed template container digital image is fully-loaded with long-slim tubes 3105, and the common second attribute includes the tubes being uncapped. Image 3101 depicts a portion of an exemplary perspective transformed subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 3103 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 3103 can be configured to receive different types of tubes. Rack 3103 may be of same type of rack 2903 and rack 3003. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 3101 depicts 3 predetermined element positions 3104, including exemplary pre-determined element positions 3104(a), 3104(b), and 3104(c) configured to receive various types of tubes. In the example illustrated in image 3101, the element position 3104(a), 3104(b), and 3104(c) are loaded with uncapped long-slim tubes 3105.

[0227] Image 3102 shows a portion of an exemplary perspective transformed template container digital image depicting a portion of a template container 3123 comprising the N pre-determined element positions loaded with uncapped long-slim tubes. For purpose of example and as embodied herein, image 3102 can be used to calculate similarity scores for the pre-determined element positions 3104 configured to receive tubes of various types. Each of the N pre-determined element positions in the template container digital image is associated with an ROI 3120 and has a template attribute uniquely associated with the template digital image. As embodied herein, an ROI 3120 may contain a portion of an element position loaded with an uncapped long-slim tube 3105. The template attribute associated with an ROI 3120 may include that the element position is loaded with a long- slim tube. The ROI 3120 may also be associated with a common template attribute that includes that the element position is loaded with an uncapped tube. For purpose of example and as embodied herein, dashed boxes are used to illustrate exemplary regions of interest for each of the N pre-determined element positions in the template container digital image, including for ROIs 3120(a), 3120(b), and 3120(c) corresponding to the pre-determined element positions 3104(a), 3104(b), and 3104(c) in the transformed subject digital image. As embodied herein, the portions of the template digital image in the image 3102 associated with ROI can be cropped and used for comparison with corresponding digital content in the transformed subject digital image 3101.

[0228] As embodied herein, the transformed subject digital image in the image 3101 may include a pre-determined search region 3110 associated with the element position 3104(b) corresponding to the template digital image in the image 3102. The center coordinates of the pre-determined search region 3110 and the ROI 3120(b) in the template digital image in the image 3102 corresponding with position 3104(b) may be identical. A size of the pre-determined search region 3110 may be larger than a size of the ROI 3120(b). In the example illustrated in FIG. 31, the search region 3110 may have twice the size of the ROI 3120(b). In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tubeLongSlim, can be generated per sub-region within the search region 3110 with the ROI 3120(b). For the search region 3110, a maximum normalized correlation coefficient, y tubeLongSlim max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 3110 can be found. The y tubeLongSlim max estimates the highest similarity between the ROI 3120(b) and a sub-region of the same size within the search region 3110. For each element location 3104 in the perspective transformed subject digital image in image 3101, y tubeLongSlim max may be found as a similarity score with its corresponding ROI. For example, a similarity score of element position 3104(a) may be calculated between a search region associated with the element position 3104(a) corresponding to the template digital image in the image 3102 and ROI 3120(a). A similarity score of element position 3104(b) may be calculated between the search region 3110 and ROI 3120(b). A similarity score of element position 3104(c) may be calculated between a search region associated with the element position 3104(c) corresponding to the template digital image in the image 3102 and ROI 3120(c). If y tubeLongSlim max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a long-slim tube and (ii) the element position being loaded with an uncapped tube.

[0229] FIG. 32 illustrates exemplary dynamic template matching with an exemplary perspective transformed template container digital image having each of the N predetermined element positions loaded with an element of another respective element type having the common second attribute. As embodied herein, the container in the perspective transformed template container digital image is fully-loaded with short tubes 3205, and the common second attribute includes the tubes being uncapped. Image 3201 depicts a portion of an exemplary perspective transformed subject digital image depicting an exemplary subject container. As embodied herein, the subject container can be a rack 3203 configured to receive tubes for use in a laboratory diagnostic instrument. As embodied herein, rack 3203 can be configured to receive different types of tubes. Rack 3203 may be of same type of rack 2903, rack 3003, and rack 3103. For purpose of example and illustration and not limitation, the portion of the subject digital image depicted in image 3201 depicts 3 predetermined element positions 3204, including exemplary pre-determined element positions 3204(a), 3204(b), and 3204(c) configured to receive various types of tubes. In the example illustrated in image 3201, the element position 3204(a) is loaded with an uncapped short tube 3205, and the element positions 3204(b) and 3204(c) are empty.

[0230] Image 3202 shows a portion of an exemplary perspective transformed template container digital image depicting a portion of a template container 3223 comprising the N pre-determined element positions loaded with uncapped short tubes. For purpose of example and as embodied herein, image 3202 can be used to calculate similarity scores for the pre-determined element positions 3204 configured to receive tubes of various types. Each of the N pre-determined element positions in the template container digital image is associated with an ROI 3220 and has a template attribute uniquely associated with the template digital image. As embodied herein, an ROI 3220 may contain a portion of an element position loaded with an uncapped short tube 3205. The template attribute associated with an ROI 3220 may include that the element position is loaded with a short tube. The ROI 3220 may also be associated with a common template attribute that includes that the element position is loaded with an uncapped tube. For purpose of example and as embodied herein, dashed boxes are used to illustrate exemplary regions of interest for each of the N pre-determined element positions in the template container digital image, including for ROIs 3220(a), 3220(b), and 3220(c) corresponding to the pre-determined element positions 3204(a), 3204(b), and 3204(c) in the transformed subject digital image. As embodied herein, the portions of the template digital image in the image 3202 associated with ROI can be cropped and used for comparison with corresponding digital content in the transformed subject digital image 3201.

[0231] As embodied herein, the transformed subject digital image in the image 3201 may include a pre-determined search region 3210 associated with the element position 3204(a) corresponding to the template digital image in the image 3202. The center coordinates of the pre-determined search region 3210 and the ROI 3220(a) in the template digital image in the image 3202 corresponding with position 3204(a) may be identical. A size of the pre-determined search region 3210 may be larger than a size of the ROI 3220(a). In the example illustrated in FIG. 32, the search region 3210 may have twice the size of the ROI 3220(a). In certain embodiments, any suitable size of the search region that is larger than a size of a corresponding ROI may be determined to be used. A normalized 2D correlation coefficient, y tubeShort, can be generated per sub-region within the search region 3210 with the ROI 3220(a). For the search region 3210, a maximum normalized correlation coefficient, y tubeShort max, among the generated normalized 2D correlation coefficients corresponding to the sub-regions within the search region 3210 can be found. The y tubeShort max estimates the highest similarity between the ROI 3220(a) and a subregion of the same size within the search region 3210. For each element location 3204 in the perspective transformed subject digital image in image 3201, y tubeShort max may be found as a similarity score with its corresponding ROI. For example, a similarity score of element position 3204(a) may be calculated between the search region 3210 and ROI 3220(a). A similarity score of element position 3204(b) may be calculated between a search region associated with the element position 3204(b) corresponding to the template digital image in the image 3202 and ROI 3220(b). A similarity score of element position 3204(c) may be calculated between a search region associated with the element position 3204(c) corresponding to the template digital image in the image 3202 and ROI 3220(c). If y tubeShort max is larger than a pre-determined threshold, subject attributes of the element position may be determined to include (i) the element position being loaded with a short tube and (ii) the element position being loaded with an uncapped tube.

[0232] As embodied herein, the subject attributes associated with the certain element position may be determined to include the element position being not empty based on a calculated similarity score of the certain element position with a first ROI containing a portion of an empty element position. If none of the similarity scores of the certain element position calculated for k second ROIs having the common template attributes is greater than or equal to its respective threshold, the subject attributes associated with the certain element position may be determined not to include the element position being loaded with an uncapped tube. As embodied herein, the subject attributes associated with the certain element position may be determined to include the element position being loaded with a capped tube if it is determined that the element position does not include any of the k uncapped tubes. For example, and not limitation, if similarity scores of a certain position of the subject container for its corresponding ROIs in images 3002, 3102, and 3202 are less than their respective thresholds, the subject attributes associated with the certain position may be determined to include the element position being loaded with a capped tube.

[0233] FIG. 33 illustrates a first example of results of determining subject attributes based on dynamic template matchings with a plurality of digital template images. Image 3301 shows a first exemplary transformed subject digital image depicting an exemplary subject container 3303. Among 6 pre-determined element positions 3304, element positions 3304(a) and 3304(f) are loaded with uncapped short tubes, and element positions 3304(b), 3304(c), 3304(d), and 3304(e) are empty. Image 3302 shows annotated determined subject attributes for the pre-determined element positions based on dynamic template matchings with a number of digital template images. Boxes 3315 indicate that the corresponding element positions are empty. The element positions 3304(b), 3304(c), 3304(d), and 3304(e) are determined to be empty. Boxes 3318 indicate that the corresponding element positions are loaded with uncapped short tubes 3308. The element positions 3304(a) and 3304(f) are determined to be loaded with uncapped short tubes 3308.

[0234] FIG. 34 illustrates a second example of results of determining subject attributes based on dynamic template matchings with a plurality of digital template images. Image 3401 shows a second exemplary transformed subject digital image depicting an exemplary subject container 3403. All the element positions 3404(a), 3404(b), 3404(c), 3404(d), 3404(e), and 3404(f) of the subject container 3403 are loaded with capped tubes. As embodied herein, a determination can be made that an element position is loaded with a capped tube, without determining the type of the tube and the type of the cap. As described further herein, determining whether a tube is capped without determining the type of cap can provide efficiency benefits in certain applications. For example, if the relevant information for a process of a diagnostic instrument is whether the tube is capped or uncapped, determining the type of cap and / or type of tube can be unnecessary. Additionally, or alternatively, the number of possible caps that may be introduced in a diagnostic instrument can exceed the number of possible tubes. Calculating similarity scores for k ROIs containing each possible tube type with each ROI having an uncapped tube can be used to infer the presence of a cap as described above without the need to calculate additional similarity scores as would be required to identify a certain cap. Image 3402 shows annotated determined subject attributes for the pre-determined element positions based on dynamic template matchings with a number of digital template images. An element position loaded with a capped tube is not annotated with a box. Thus, no box is shown in image 3402 as all the element positions are loaded with capped tubes.

[0235] FIG. 35 illustrates a third example of results of determining subject attributes based on dynamic template matchings with a plurality of digital template images. Image 3501 shows a third exemplary transformed subject digital image depicting an exemplary subject container 3503. Among 6 pre-determined element positions 3504, element positions 3504(a), 3504(b), 3504(c), and 3504(d) are loaded with capped tubes, element position 3504(e) is loaded with an uncapped short tube 3508, and element position 3504(f) is loaded with an uncapped long tube 3506. Image 3502 shows annotated determined subject attributes for the pre-determined element positions based on dynamic template matchings with a number of digital template images. For the first four element positions 3504(a), 3504(b), 3504(c), and 3504(d), no box is annotated as the element positions 3504(a), 3504(b), 3504(c), and 3504(d) are loaded with capped tubes. Box 3518 indicates that the corresponding element position is loaded with uncapped short tubes 3508. The element position 3504(e) is determined to be loaded with uncapped short tube 3508. Box 3516 indicates that the corresponding element position is loaded with an uncapped long tube. The element position 3504(f) is determined to be loaded with an uncapped long tube 3506.

[0236] FIG. 36 illustrates a fourth example of results of determining subject attributes based on dynamic template matchings with a plurality of digital template images. Image 3601 shows a fourth exemplary transformed subject digital image depicting an exemplary subject container 3603. Among 6 pre-determined element positions 3604, element positions 3604(a) and 3604(b) are loaded with uncapped long-slim tubes, element positions 3604(c) and 3604(e) are loaded with capped tubes, element position 3604(d) is loaded with an uncapped short tube 3608, and element position 3604(f) is loaded with an uncapped long tube 3606. Image 3602 shows annotated determined subject attributes for the pre-determined element positions based on dynamic template matchings with a number of digital template images. Boxes 3617 indicate that the corresponding element positions are loaded with uncapped long-slim tubes 3607. The element positions 3604(a) and 3604(b) are determined to be loaded with uncapped long-slim tubes 3607. No boxes are shown for the element positions 3604(c) and 3604(e) because the element positions 3604(c) and 3604(e) are loaded with capped tubes. Box 3618 indicates that the corresponding element position is loaded with uncapped short tubes 3608. The element position 3604(d) is determined to be loaded with uncapped short tube 3608. Box 3616 indicates that the corresponding element position is loaded with an uncapped long tube. The element position 3604(f) is determined to be loaded with an uncapped long tube 3606.

[0237] FIG. 37 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including an empty template container. Image 3701 depicts histograms representing occurrences of y tube holder max in each data point range among 342 data points observed for the template container. A first distribution group 3702 represents non-empty element positions. A second distribution group 3703 represents empty element positions. Based on the results of tests, a proper threshold 3704 of y tube holder max for determining whether an element position configured to receive tubes of various types is empty may be 0.85. If y tube holder max is greater than or equal to the threshold 3704, the element position may be determined to be empty. If y tube holder max is less than the threshold 3704, the element position may be determined to be loaded a tube.

[0238] FIG. 38 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped long tubes. Image 3801 depicts histograms representing occurrences of y tubeLong max in each data point range observed in tests. A first distribution group 3802 represents element positions not loaded with uncapped long tubes. A second distribution group 3803 represents element positions loaded with uncapped long tubes. Based on the results of tests, a proper threshold 3804 of y tubeLong max for determining whether an element position configured to receive tubes of various types is loaded with an uncapped long tube may be 0.825. If y tubeLong max is greater than or equal to the threshold 3804, subject attributes associated with the element position may be determined to include (i) the element position being loaded with a long tube and (ii) the element position being loaded with an uncapped tube.

[0239] FIG. 39 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped long-slim tubes. Image 3901 depicts histograms representing occurrences of y tubeLongSlim max in each data point range observed in tests. A first distribution group 3902 represents element positions not loaded with uncapped long-slim tubes. A second distribution group 3903 represents element positions loaded with uncapped long-slim tubes. Based on the results of tests, a proper threshold 3904 of y tubeLongSlim max for determining whether an element position configured to receive tubes of various types is loaded with an uncapped long-slim tube may be 0.8. If y tubeLongSlim max is greater than or equal to the threshold 3904, subject attributes associated with the element position may be determined to include (i) the element position being loaded with a long-slim tube and (ii) the element position being loaded with an uncapped tube.

[0240] FIG. 40 illustrates an example histogram of maximum normalized correlation coefficients for the template container digital image including a template container loaded with uncapped short tubes. Image 4001 depicts histograms representing occurrences of y tubeShort max in each data point range observed in tests. A first distribution group 4002 represents element positions not loaded with uncapped short tubes. A second distribution group 4003 represents element positions loaded with uncapped short tubes. Based on the results of tests, a proper threshold 4004 of y tubeShort max for determining whether an element position configured to receive tubes of various types is loaded with an uncapped short tube may be 0.65. If y tubeShort max is greater than or equal to the threshold 4004, subject attributes associated with the element position may be determined to include (i) the element position being loaded with a short tube and (ii) the element position being loaded with an uncapped tube.

[0241] 6 Machine Vision Systems

[0242] The disclosed subject matter further includes machine vision systems. In accordance with an aspect of the disclosed subject matter, machine vision systems can be used in diagnostic instruments. Machine vision systems in accordance with an aspect of the disclosed subject matter include a camera having a field of view and at least a portion of a diagnostic instrument positioned within the field of view. The at least a portion of a diagnostic instrument is configured to receive the subject container comprising N predetermined element positions. Systems in accordance with an aspect of the disclosed subject matter further include a data processing unit operatively coupled to the diagnostic instrument, and a memory storing instructions which, when executed by the processing unit, cause the system to: capture, using the camera, a subject digital image including an image of the subject container; access one or more template digital images including a first region of interest (ROI) having a first attribute and one or more second ROIs having a common second attribute; calculate for a first element position among the N pre-determined element positions in the subject digital image a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image determine whether one or more first subject attributes associated with the first element position include the first attribute based on the first similarity score; calculate for the first element position, in response to a determination that the one or more first subject attributes associated with the first element position do not include the first attribute, one or more second similarity scores for the one or more second ROI having a common second attribute; and determine whether the first subject attributes associated with the first element position include the common second attribute based on the second similarity scores.

[0243] FIG. 41 illustrates an exemplary machine vision system 4100. The system includes a camera 4101. As embodied herein, the camera 4101 can be a digital camera. The camera 4101 can output full color images, grayscale or single channel (e.g., red, green, or blue) per image capture. For example, and as described further herein, a single channel output can be used, and the channel color can be selected based on the properties of the container to be imaged. The camera 4101 has a field of view, and at least a portion of a diagnostic instrument configured to receive a subject container is positioned within the field of view of the camera. As embodied herein, the camera can be mounted on a stationary portion of the diagnostic instrument. Additionally, or alternatively, and as described further herein, the camera can be configured to move. For example, the camera can be affixed to or incorporated in a robot within the diagnostic instrument. As embodied herein, the field of view of the camera 4101 can include a robot arm (not shown) holding a subject container 4102. For purpose of example, the subject container can be a tray or rack configured to receive one or more consumables used during operation of the diagnostic instrument. As embodied herein, the subject container 4102 can be a rack configured to receive a tube at each of the N pre-determined positions.

[0244] Any suitable camera configuration can be used. For purpose of example, a working distance between the camera and the subject container can be between about 100 mm and about 500 mm. Additionally, or alternatively, the working distance between the camera and the subject container can be between about 200 mm and about 400 mm. For purpose of example, and as embodied herein, the working distance between the camera and the subject container can be about 290 mm. The camera can include a flash. For purpose of example, and as embodied herein, the camera can include light-emitting diodes (LEDs). For example, and as embodied herein, the camera can include eight LEDs, with four LEDs on each side of the lens. For example and as embodied herein, the LEDs can flash once per image capture. Although a specific LED flash configuration is described, any suitable flash can be used.

[0245] In certain embodiments the camera can be configured to prevent liquids or other substances that may be present in the diagnostic instrument from contaminating the camera lens, which may reduce the quality of images captured by the camera. For example, the camera can include a vent-hood. The vent hood can be used to, for example, prevent liquid from splashing on the camera lens. The vent hood can have any suitable configuration. For example, the vent hood can have a cone shape, and the cone can extend out from the camera lens. The size of the cone can be selected so as not to obscure the camera’s field of view. Additionally, or alternatively, the vent hood can include a gutter. The gutter can, for example, direct liquid collected by the vent hood away from the camera lens. For example, a gutter can be included on an upper portion of an interior of a cone defining the vent hood. The gutter can prevent liquid from running down the interior surface of the cone onto the camera lens.

[0246] As embodied herein, the camera 4101 can have a tilted view angle of the subject container 4102. For example, and as embodied herein, the one or more areas of the diagnostic instrument configured to receive the subject container 4102 can define an area axis 4103, the area axis 4103 extending through a center of the subject container 4102 and perpendicular to a plane 4104 defined by a mouth of the subject container 4102. As embodied herein, the camera can be offset from the area axis 4103. As described further herein, images taken with a camera having a tilted view angle can include perspective distortion. The machine vision systems and methods described herein can be configured to accurately calculate similarity scores and determine subject attributes for subject images with perspective distortion.

[0247] As embodied herein, the system can be configured to capture subject images of different subject container types. Different source anchor points can be used for perspective transform, and different template images can be used to calculate similarity scores and determine subject attributes for different subject containers, as described further herein. Systems can use any suitable technique for identifying subject container type and selecting appropriate parameters for a certain subject container type, such as source anchor points and template images. For purpose of example and as embodied herein, a barcode reader 4105 can be used to read a barcode on the subject container to determine the subject container type. Systems in accordance with the disclosed subject matter can be used in any diagnostic instrument configuration. For purpose of example and not limitation, machine vision systems in accordance with the disclosed subject matter can be used in high- throughput nucleic acid testing systems, such as those described in International Application PCT / US2022 / 027067, published as International Publication Number WO2022 / 232601A1 the disclosure of which is hereby incorporated by reference in its entirety. For purpose of example and not limitation, a camera can be mounted to a robot or loader such as the robots and loaders described in PCT / US2022 / 027067. Additionally, or alternatively, machine vision systems can be used to determine subject attributes of subject containers, including for example, containers as described in PCT / US2022 / 027067. For example, and not limitation, machine vision systems can be used to determine whether sample tubes received in the system include caps. For example, sample tube racks can be received at one or more positions such as within a loading bay as described in PCT / US2022 / 027067. Machine vision systems can be used to determine whether sample tubes received in one or more sample tube racks are capped.

[0248] For purpose of example, during operation of a diagnostic instrument, a robot may pick up test tubes from a rack placed on an input deck such as within a loading bay. If a test tube has a cap on, the diagnostic instrument may not be able to uncap the tube for further processing, and handling error could occur. For example, if a robotic pipettor such as described in PCT / US2022 / 027067 attempts to pipette a sample from a capped tube, the robot may sense the change in torque from contacting the cap and a servo motor error may be triggered, or the robotic pipettor may be damaged. Therefore, automatically detecting if any test tube on a rack has a cap on before the probe contacts the cap can be important for operation of the certain diagnostic instruments. If an inadvertent capped tube is detected, the instrument may, for example, be able to trigger a process to uncap the tube. For example, the tube can be uncapped and the rack of test tubes with uncapped tubes can be refed back to the instrument.

[0249] FIG. 42 illustrates a portion of another exemplary diagnostic instrument 4201 and machine vision system 4200. The diagnostic instrument 4201 is configured to receive a first subject container 4202. As embodied herein, the first subject container 4202 is configured to be received at a consumable loading area 4203 of the diagnostic instrument. As embodied herein, the first subject container 4202 can be a tray having pre-determined element positions configured to receive reaction vessels and caps for use in the diagnostic instrument. As embodied herein, the reaction vessels and caps can be amplification vessels and caps as described in PCT / US2022 / 027067.

[0250] A camera (not shown) can be included on robot arm 4204. The camera can capture a subject container digital image of the tray 4202 and the machine vision methods as described herein can be used to determine one or more subject attributes for each of the pre-determined element positions in the subject digital image. For example and as embodied herein, the camera can capture a subject container image of the tray 4202 when the tray is introduced into the consumable loading area 4203. For example, and as embodied herein, the one or more subject attributes can include whether each element position is empty or loaded with an amplification vessel or cap, respectively. As described further below, the diagnostic instrument 4201 can use the subject attributes to, for example, guide the robot 4204 to pick the next available amplification vessel or cap from the tray 4202.

[0251] As embodied herein, the diagnostic instrument 4201 can be configured to receive a second subject container 4205 comprising pre-determined element positions at a second area of the diagnostic instrument 4201. For purpose of example, and as embodied herein, the second subject container 4205 can be a carousel 4206 having pre-determined element positions 4207. As embodied herein, the carousel can be an amplification and detection carousel as described in PCT / US2022 / 027067. As embodied herein, calibration methods such as ellipse-fitting as described above, can be used to calibrate the machine vision system with respect to the second carousel and pre-determined element positions therein.

[0252] As embodied herein, the robot 4204 can pick an amplification vessel from tray 4202 and place the amplification vessel in carousel 4206. The system can then use the camera to capture a second subject digital image of the carousel 4206. As embodied herein the machine vision methods described herein can be used to determine a subject attribute for each of the pre-determined element positions in the carousel. As embodied herein, the subject attribute can include whether a pre-determined element position is properly loaded with a reaction vessel after an attempted placement by the robot 4204.

[0253] As described above, systems in accordance with the disclosed subject matter include a data processing unit operatively coupled to the diagnostic instrument. In certain embodiments, the data processing unit can be a computer. For purpose of example and as embodied herein, the data processing unit can be used for multiple functions. For purpose of example, the data processing unit can be used to execute machine vision methods as described herein, as well as to execute additional operations within a diagnostic instrument. For purpose of example and not limitation, the processing unit can be shared with the mechanical sub-systems of the diagnostic instrument. As described further herein, machine vision methods that use data processing unit resources efficiently can be desirable, for example, to minimize the resources used by machine vision systems when resources are shared in the diagnostic instrument.

[0254] Although this disclosure describes certain configurations of the machine vision system, this disclosure contemplates any suitable configuration of the machine vision system. As described above, embodiments can include machine vision systems and methods for verifying whether a tube in the diagnostic instrument is capped. For example, a subject container can include a rack of tubes, such as for example calibrator control tubes, and the machine vision system can capture a subject digital image of the rack of tubes and determine subject attributes for each pre-determined element position of the rack of tubes. As embodied herein, the subject attribute can include whether a tube in the pre-determined element position has a cap. As embodied herein, if a calibrator control tube has a cap on, then the probe cannot uncap it automatically, and an error can be triggered.

[0255] 7 Examples

[0256] The following non-limiting examples are provided for purpose and illustration of the disclosed subject matter and not limitation.

[0257] Example 1

[0258] This example illustrates an exemplary determination of an exposure mode. In certain embodiments, a machine vision system may determine an exposure mode. The machine vision system may determine optimal exposure time and sensor gain for the determined exposure mode during a camera calibration period. As an example and not by way of limitation, a manual exposure mode may be implemented if a machine vision system is designed to be operating in an environment with constant illumination. An auto exposure mode should be implemented if a machine vision system is designed to be operating in an environment with varying illumination. To implement a manual exposure mode, the machine vision system may collect a few sample images of the same test target with various exposure time and determine the optimal exposure time and sensor gain during the calibration time. Then, an improved or optimal exposure time and sensor gain may be set to the default camera job per target type per application. A machine vision camera can store a lot of camera jobs and a vision control software can load a camera job per target type per application in real time.

[0259] A number of techniques for implementing an auto exposure mode may be available. First, the camera sensor may be shipped with an auto exposure algorithm. Certain camera sensors have embedded an auto exposure algorithm in the sensor chip. This option may require the camera vendor to enable the auto exposure algorithm via the camera firmware. Second, a camera processor chip may be shipped with an auto exposure algorithm. Certain machine vision camera models have implemented an auto exposure algorithm on its camera processor chip. The camera vendor may make the algorithm available for the external vision software to trigger the algorithm in real time. Third, vision control software may have an auto exposure algorithm: FIG. 1 illustrates an example auto exposure algorithm. Note that the sensor gain is assumed to be fixed in the algorithm illustrated in FIG. 1.

[0260] Example 3

[0261] This example illustrates an exemplary calibration in accordance with the disclosed subject matter. In certain embodiments, the calibration target may be a calibration target container. The one or more calibration markers may be one or more source anchor points of the calibration target container. In such a case, to determine current coordinates of one or more source anchor points of the calibration target container, the machine vision system may generate a brightness-improved calibration digital image by performing a gamma correction to the calibration digital image. The machine vision system may generate a binary calibration digital image by performing Otsu Thresholding algorithm on the brightness-improved calibration digital image. Each pixel in the binary calibration digital image may belong to a foreground or background. The machine vision system may detect a bounding box of the calibration target container based on projected sums on the binary calibration digital image. The machine vision system may also determine current coordinates of the one or more source anchor points by performing a template matching for each of one or more corners of the bounding box with a respective template image. A search region for each corner may be determined based on the bounding box. Causing the calibration target to be at a second relative location from the digital camera may include causing the calibration target container to be moved to the second relative location from the digital camera. As an example and not by way of limitation, a camera calibration may be required to register the optimal XY coordinates for picture taking because the unit-to-unit variation in camera FOV may be noticeable.

[0262] FIG. 5 illustrates an example unit-to-unit variation in camera FOV. In the left picture, the whole tray is included in the FOV of a first camera. The right edge of the tray is clipped by the FOV of a second camera as shown in the right picture. Note that the robot arm may adjust the tray position to left / right / forward / backward. Z position is preset to maintain the working distance at 290 mm. A full tray of Sample Tip may be used as a calibration target because the tray of Sample Tip is the largest tray among various tray types. A robot arm may transport a full tray of Sample Tip from Load Bay to the inventory station at the default XYZ position. The machine vision system may take a picture to detect XY coordinates at the upper left comer of the tray. There is a preset window that determines a region of the acceptable XY coordinates on the image. If the detected XY coordinates are outside this preset window, an iteration process may be performed between the robot arm and the vision system. After an image is processed, the vision system may ask the robot control software to move the robot arm to left / right / forward / backward by N mm. Then, the robot control software will move the robot arm and the tray to left / right / forward / backward by N mm. The machine vision system may take another picture to detect the XY coordinates at the upper left comer of the tray. If the XY coordinates of the image are within the preset window, the vision system will inform the robot control software to register the XY coordinates of the robot position during the picture taking for all the trays, tubs, and racks in this instrument unit. The algorithm to detect XY coordinates at the upper left corner of a tray may be a subset of the algorithm to detect XY coordinates at four corners of a tray as described below.

[0263] In certain embodiments, to detect coordinates of source anchor points corresponding to the corners of a container, the machine vision system may apply the processing block called “Gamma Correction” to an input image containing a container. The image brightness may be improved after this procedure. The machine vision system may apply the processing block called “Otsu Thresholding” to separate foreground and background pixels on the image. A binary image with black and white pixels is generated. Then, the machine vision system may apply the processing block called “Bounding Box Detection using Projected Sums” to the binary image and to detect XY coordinates for a bounding box of the container. The four comers of the bounding box determine a search region per corner for the next procedure. Then, the processing block called “Template Matching” may compare the template image and the patch image within the search region per corner on the gamma-corrected image to find the best matched location. The processing block, “Template Matching” may output XY coordinates of four source anchor points.

[0264] Otsu Thresholding may be used to perform automatic image thresholding. In a simple form, the algorithm returns a single intensity threshold that separate pixels into two classes: foreground and background. FIG. 4 illustrates a comparison between the input gamma-corrected image and the binary image after Otsu thresholding. The left image is the gamma-corrected image, and the right image is the binary image after Otsu thresholding. On the binary image, black pixels are the background pixels, and white pixels are the foreground pixels. Bounding box detection using projected sums may be used to detect a bounding box of a container. The bounding box would be used to determine a search region per source anchor point. FIG. 11 illustrates example projected sums on a binary image. A plot of the projected sum in Y direction, sum of image rows per column, on the binary image is shown at the bottom. A plot of the projected sum in X direction, sum of image columns per row, on the binary image is shown at the right. For the projected sum profile in Y direction, a middle value can be calculated from the maximum and minimum values of the profile. Then, starting a search from the left, an index close to the middle value can be found, called XI. Starting a search from the right, an index close to the middle value can be found, called X2. For the projected sum profile in X direction, another middle value can be calculated from the maximum and minimum values of the profile. Then, starting a search from the right, an index close to the middle value can be found, called Y2. The container in the example illustrated in FIG. 11 is a rack. Since the algorithm works for a rack with tubes or without tubes, Y1 of the bounding box may be inferred from XI, X2, and Y2. Given a fixed camera view angle, the aspect ratio of the bounding box for a rack is fixed. XI and X2 define the width of a bounding box. The aspect ratio of the bounding box infers the height of a bounding box. Given Y2, Y 1 may be calculated using the inferred height of the bounding box. Given (XI, Yl), and (X2, Y2), a bounding box may be drawn.

[0265] Template matching may be a method for searching and finding the location of a template image in a larger image. The algorithm simply slides the template image over the input image and compares the template and patch of input image. FIG. 12 illustrates an example scenario for a template matching. The template matching algorithm may find a region in the subject image 1210 that matches the template image 1230 best. In the example illustrated in FIG. 12, the size of the template image 1230 is m pixels by n pixels. First, the algorithm may identify a search region 1220 within the target image 1210, which would be larger than the template image 1230. In the example illustrated in FIG. 12, the search region 1220 is assumed to be 2m pixels by In pixels.

[0266] FIG. 13 illustrates example computations for the template matching scenario illustrated in FIG. 12. At the first iteration, a normalized 2-dimensional (2D) correlation coefficient between a first sub-region 1310 of the search region 1220 and the template image 1230 may be calculated. The first sub-region 1310 may be identified by coordinates (0, 0) of the upper-left corner of the first sub-region 1310. At the second iteration, a normalized 2D correlation coefficient between a second sub-region 1320 of the search region 1220 and the template image 1230 may be calculated. The second sub-region 1320 may be identified by coordinates (1, 0) of the upper-left corner of the second sub-region 1320. At the (m+2)- nd iteration, a normalized 2D correlation coefficient between a (m+2)-nd sub-region 1330 of the search region 1320 and the template image 1230 may be calculated. The (m+2)-nd sub-region 1330 may be identified by coordinates (0, 1) of the upper-left comer of the second sub-region 1330. At the (m+2)(w+2)-th iteration, a normalized 2D correlation coefficient between a (m+2)(w+2)-th sub-region 1340 of the search region 1220 and the template image 1230 may be calculated. The (m+2)(w+2)-th sub-region 1340 may be identified by coordinates (m, ri) of the upper-left corner of the second sub-region 1340. The size of the sub-regions is m pixels by n pixels, the size of the template image 1230. After calculating the normalized 2D correlation coefficients, a sub-region having a highest normalized 2D correlation coefficient may be selected as the sub-region most similar to the template image 1230. A degree of similarity may be determined based on the normalized 2D correlation coefficient of the selected sub-region.

[0267] In certain embodiments, an empty vessel holder on the carousal can be used as a calibration marker. The rim edge of an empty vessel can be fitted to an ellipse. The center pixel coordinates of the fitted ellipse can be used to calibrate the camera position in x and y directions. The center of the fitted ellipse should be detected near the preset pixel coordinate in x and y directions. If not, the camera position may have drifted. The major / minor axis half lengths of the fitted ellipse can be used to calibrate the camera position in z direction. If those lengths are not close to the preset lengths, then the camera position has drifted in z direction. The orientation of the fitted ellipse can be used to calibrate the camera view angle. If the measured ellipse orientation is not close to the preset orientation, then the camera view angle may have drifted.

[0268] FIG. 14 illustrates an example method for identifying source anchor points of a rack using template matching. FIG. 14 shows four template images and four search regions, one per comer, for template matching algorithm. The search region per corner may be determined by the bounding box corner of the rack. Within each search region, the best matched location is found by comparing the template image and a sub-region at an image coordinate inside the search region. FIG. 15 illustrates an example bounding box and source anchor points of a rack. The left picture of FIG. 15 shows a bounding box. The right picture of FIG. 15 shows the best matched location per corner as dots. The dotted locations are the source anchor points.

[0269] FIG. 43 illustrates an example calibration target that is a rack with a white board in the front, containing four corner marks. Four comer marks on the white board are corresponding to the four source anchor points of a rack, as shown in the left picture of FIG. 44. FIG. 44 illustrates an example perspective transform of an empty rack. During the camera calibration, a picture of the calibration rack is taken. Then, four comer marks of the white board are detected by a vision algorithm to register the XY coordinates of four source anchor points. During the development time, the destination anchor points are selected by the algorithm developer to generate the warped image through a perspective transform. The destination anchor points are shown as green circles in the right picture of FIG. 44. For the same rack type, the source and destination anchor points are fixed, so the perspective transform matrix is fixed, too. Therefore, a perspective transform matrix is generated per rack type during the camera calibration using four source anchor points and four destination anchor points, and it will be applied to every test-tube picture in the real time.

[0270] Example 4

[0271] This example illustrates another exemplary calibration. In certain embodiments, the calibration target may be a consumable loading station. The one or more calibration markers may include an empty element holder. Rim edge of the empty element holder may be fitted to an ellipse. In such a case, to determine current coordinates of the empty element holder, the machine vision system may generate a brightness-improved calibration digital image by performing a gamma correction to the calibration digital image. The machine vision system may detect the empty element holder in the brightness-improved calibration digital image by performing a template matching with a template image of an empty element holder. The machine vision system may calculate coordinates of a center of the empty element holder by performing a least squares ellipse-fitting method on a cropped region of the calibration digital image containing the detected empty element holder. The machine vision system may also determine a major-axis half length, a minor-axis half length, and an orientation of the fitted ellipse. Causing the calibration target to be at a second relative location from the digital camera may include adjusting a position and a view angle of the digital camera based on the calculated coordinates of the center of the empty element holder, the major-axis half length, the minor-axis half length, and the orientation of the fitted ellipse.

[0272] In certain embodiments, an empty vessel holder on the carousal may be used as a calibration marker. The rim edge of an empty vessel may be fitted to an ellipse. The center pixel coordinates of the fitted ellipse may be used to calibrate the camera position in x and y directions. The center of the fitted ellipse should be detected near the preset pixel coordinate in x and y directions. If not, the camera position has drifted. The major / minor axis half lengths of the fitted ellipse may be used to calibrate the camera position in z direction. If those lengths are not close to the preset lengths, then the camera position has drifted in z direction. The orientation of the fitted ellipse may be used to calibrate the camera view angle. If the measured ellipse orientation is not close to the preset orientation, then the camera view angle has drifted.

[0273] FIG. 6 illustrates an example camera calibration algorithm using an empty vessel holder. First, the processing block, “Gamma Correction”, may be applied to the input image. The image brightness can be improved after this procedure. Then, the processing block, “Vessel Holder Detection”, may be applied to determine if a vessel holder is present in the scene. If not, then the camera position has drifted greatly from the original field of view (FOV). An error flag for a calibration failure will be triggered. If the vessel holder is present in the scene, the next processing block, “Ellipse-Fitting of Vessel Holder”, may be applied to a small sub-image of the original input image. This processing block can output the following attributes of the fitted ellipse: (i) the center coordinates that can be used for inferring a camera position drift in x and y directions, (ii) the major / minor half axis length that can be used for inferring a camera position drift in z direction, and (iii) the rotation angle that can be used for inferring a drift in camera view angle. If any drift exceeds a threshold, then an error flag for a calibration failure may be triggered.

[0274] Template matching may be utilized to detect the vessel holder. FIG. 7 illustrates an example vessel holder detection using template matching. The right image shows a template of the empty vessel holder. In the example illustrated in FIG. 7, the size of the template image is 107 pixels by 85 pixels. The left image shows the input capper image. The box in the middle image shows the bounding box of the detected vessel holder by using the template. When no vessel holder presents at scene, the camera position has drifted away from the original FOV. The empty vessel holder should not be reflective to the LED flash device.

[0275] FIG. 8 illustrates example procedures for ellipse-fitting. First, the machine vision system may crop a region of interest (ROI) containing the vessel holder from the original image. The ROI may be identified based on the vessel holder detection. The machine vision system may enhance the contrast within the ROI image by applying a traditional computer vision algorithm called “Global Histogram Equalization” to the ROI image. The machine vision system may generate a binary edge image by applying a traditional computer vision algorithm called “Canny Edge Detector” to the contrast- enhanced image. The machine vision system may apply a traditional computer vision algorithm called “Connected-component Labeling” to the binary edge image. Each edge segment may be isolated, and statistics of each edge may be generated. Two longest edges, one near the top center x coordinate, and the other one near the lower center x coordinate, may be extracted based on the edge statistics. The machine vision system may erase some edge pixels in a known region of strong reflection in the ROI image. The machine vision system may apply a least squares ellipse-fitting method to the remaining edge pixels.

[0276] The fitted ellipse is shown in the right image of FIG. 8. The calculated X coordinate of the center of the ellipse may be used to adjust the camera position in X direction. The calculated Y coordinate of the center of the ellipse may be used to adjust the camera position in Y direction. The calculated major axis half-length and minor axis halflength may be used to adjust the camera position in Z direction. The calculated rotation angle may be used to adjust camera view angle. The vision system may ask the robot control software to move the robot arm to adjust camera position in X, Y, or Z direction and to adjust camera view angle based on the calculated ellipse properties. Although this disclosure describes performing a camera calibration using an empty vessel holder in a certain manner, this disclosure contemplates performing a camera calibration using an empty vessel holder in any suitable manner.

[0277] Example 5

[0278] This example illustrates exemplary subject images. FIG. 45 illustrates example color images, having red, green, blue (RGB) channels, taken from a simulation testbed. Upper left image is a tub of Wash Vessel. Upper center image is a tray of Lysis Tube (bigger diameter) and Transfer Tip (smaller diameter). Upper right image is a tray of Eluate Tip. Lower left image is a tray of Amplification Vessel (AV) and AV Cap. Lower center image is a tray of Sample Tip. Lower right image is a rack of Calibrator Control Tube. Perspective distortion is present on all the images due to a tilted view angle from the camera. This tilted view angle may be required to detect the number of wash vessels per slot. With perspective distortion on a wash tub, the deck height of wash vessels is proportional to the number of wash vessels per slot. If a machine vision system does not have to detect depth of consumables, the desired view angle of the camera may be at straight top view.

[0279] In certain embodiments, the machine vision system may use the digital camera to take a subject digital image of a subject container located at a pre-determined relative location from the digital camera with the determined optimal exposure time and sensor gain. The subject container may include N pre-determined element positions. The machine vision system may access the subject digital image of the subject container comprising the N pre-determined element positions. In certain embodiments, the subject digital image may comprise a single-color channel. The single-color channel producing highest contrast may be selected among red, green, blue (RGB) channels. FIG. 2 illustrates example single-channel subject images taken from a simulation testbed. The images presented in FIG. 2 are the input images to the machine vision algorithms for inventory verification. If consumables are bright or translucent, a darker tray or tub is preferred to the machine vision algorithms because a high contrast scene may improve detection accuracy. Vice versa, if consumables are dark, a bright tray or tub is preferred to the machine vision algorithms, because a high contrast scene can improve detection accuracy. Per color science, yellow objects absorb blue light. Thus, yellow objects look like dark objects in blue channel of a color image. Yellow objects do not absorb green light. Thus, yellow objects look like bright objects in green channel of a color image.

[0280] In FIG. 2, because wash vessels, Lysis tubes, transfer tips, amplification vessels (AVs), AV caps are bright or translucent, the blue-channel image may introduce a high contrast between the consumables and the tray or tub. Vice versa, because eluate tips and sample tips are black, the green-channel image may introduce a high contrast between the consumables and the tray. For the calibrator control tubes, the orange caps absorb blue light. Thus, the orange caps look like dark caps in blue channel. Because the calibrator control tube is translucent, the blue-channel image may introduce a high contrast between the tube and the cap.

[0281] To achieve a high contrast scene for this algorithm, if consumables are bright or translucent, then the tray must look darker. A yellow object absorbs blue light, so a yellow tray or tub looks darker in blue channel of a color image. If consumables are dark, then the tray must look brighter. A yellow object reflects green light, so a yellow tray or tub looks brighter in green channel of a color image. Therefore, the input tray image is taken in blue channel for the following three tray types: Tub of Wash Vessel, Tray of Lysis Tube and Transfer Tip, and Tray of AV and AV Cap. For Tub of Wash Vessel, a high image pixel resolution is required to provide a wider separation between every two distribution of vessel counts, so the full image resolution may be chosen. The high image pixel resolution may increase accuracy of inventory verification on Tub of Wash Vessel. Given the accurate source anchor points, the test results on the testbed show that accuracy of inventory map is 100% at the image resolution of 5 MP.

[0282] On the other hand, to reach an acceptable trade-off between the image pixel resolution and the processing time, the input image of the other tray types is down-sampled from 5 MP to 1.25 MP inside the camera processor. Given the accurate source anchor points, the test results on the testbed show that accuracy of inventory map is 100% at the image resolution of 1.2 MP.

[0283] Example 6

[0284] This example illustrates exemplary perspective transform. FIG. 19 illustrates an image processing pipeline of the real-time source anchor point detection algorithm for inventory verification. First, the processing block, “Gamma Correction”, is applied to the input tray image. The image brightness is improved after this procedure. Then, the processing block, “Otsu Thresholding”, is applied to separate foreground and background pixels on the image. A binary image with black and white pixels is generated. Then, the processing block, “Bounding Box Detection using Projected Sums”, is applied to the binary image and to detect XY coordinates for a bounding box of a tray. The four comers of the bounding box determine a search region per comer for the next procedure. Then, the processing block, “Template Matching”, compares the template image and the sub-region within the search region per comer on the gamma-corrected image to find the best matched location. “Template Matching” processing block outputs XY coordinates of four source anchor points. Finally, the processing block, “Perspective Transform Matrix Creation”, generates a perspective transform matrix, given the four source anchor points and the four destination anchor points. Four destination anchor points are pre-determined by the algorithm developer.

[0285] In certain embodiments, the machine vision system may perform a gamma correction to the subject digital image to improve brightness of the subject digital image. In certain embodiments, the machine vision system may perform a perspective transform of the subject digital image to define a transformed subject digital image. The perspective transform may include relocating a plurality of source anchor points corresponding to the subject digital image of the subject container to respective pre-determined destination anchor points. The tray picture taken from a tilted camera has perspective distortion. Through a perspective transform, the perspective distortion in the picture may be corrected.

[0286] FIG. 9 illustrates a first example of a perspective transform. The original image with source anchor points is shown on the left. The transformed (also called “warped”) image with pre-determined destination anchor points is shown on the right. FIG. 10 illustrates a second example of a perspective transform of a tray of Amplification Vessels (AVs) and Caps. The original image taken from a tilted camera is shown on the left. Four source anchor points are shown as circles. The transformed (warped) image with the user- selected destination anchor points (circles) is shown on the right. The right image in FIG. 10 is ready for the inventory verification algorithm. To generate a perspective transform matrix, a picture is taken to extract the source anchor points, shown as circles in the left picture of FIG. 9 or FIG. 10. The destination anchor points may be pre-determined by an algorithm developer. The destination anchor points are shown as circles in the right picture of FIG. 9 or FIG. 10. Given four source anchor points and four destination anchor points, a perspective transform matrix is generated. Then, a perspective transformed image or a warped image can be generated by applying the perspective transform matrix on the input image. Given the warped image per tray type, the tray shape is rectangular and the size of each consumable on the tray is identical to each other. This rectangular tray or tub facilitates the inventory verification algorithms.

[0287] In certain embodiments, a perspective transform matrix used for the perspective transform may be pre-calculated and applied to the subject digital image for a given container type in the subject digital image. An anti-sagging mechanism and a universal calibration target are deployed to address the inconsistent sagging issue. Both anti-sagging mechanism and the universal calibration target must be deployed at the same time.

[0288] FIG. 16 illustrates an example anti-sagging mechanism and an example universal calibration target. The left drawing shows a triangle fixture with 5 contact points, as an L shape. The fixture may use at least two contact points to hold one tub horizontally without any sagging during the image capture in the real-time operation. The right picture shows an example of the universal calibration target. It is an empty tub with a white plate on the top. For every tray type, four source anchor points corresponding to four comers are marked on the white plate. During the calibration time, when the anti-sagging mechanism holds the universal calibration target horizontally, a picture is taken. Four source anchor points per tray type can be detected and a perspective transfer matrix per tray type can be generated as described herein, given four destination anchor points per tray type. In the real-time operation, every tub will be held horizontally by the anti-sagging mechanism, so the perspective transfer matrix per tray type can be applied to all images of that tray type for perspective distortion correction.

[0289] In certain embodiments, a perspective transform matrix used for the perspective transform may be calculated based on current coordinates of the plurality of source anchor points in the subject digital image. While a tray arrives at the inventory station, one picture may be taken. Then, a perspective transform matrix may be generated in the real time after four source anchor points are detected. For some tray types, another picture at a different image channel may be taken. The perspective transform matrix is applied to this picture first. Then, the perspective corrected image is used to generate an inventory map. This option reduces cost to add an anti-sagging mechanism and a universal calibration target.

[0290] FIG. 18 illustrates example detected source anchor points of the tray of AV and AV Cap at 3 sagging positions in Y direction. Left shows the input image of Tray of AV and AV Cap. Right shows the overlay plot of the detected source anchor points at 3 sagging Y positions: horizontal, -3 mm, and -5 mm. Accuracy of the source anchor points affects accuracy of the perspective corrected image. Accuracy of the perspective corrected image affects accuracy of inventory verification.

[0291] FIG. 46 illustrates an example Gamma Correction on Tray of Lysis Tube and Transfer Tip. Left is an original image taken in green channel. Right is an image with gamma correction. FIG. 47 illustrates an example Gamma Correction on Tray of AV and AV Cap. Left is an original image taken in green channel. Right is the image with gamma correction. The gamma-corrected images are brighter and show more details.

[0292] FIG. 48 illustrates an example Otsu thresholding. FIG. 48 shows the input gamma-corrected image and the binary image after Otsu thresholding. On the binary image, black pixels are the background pixels, and white pixels are the foreground pixels. The images are from Tray of AV and AV Cap.

[0293] FIG. 49 illustrates example projected sums on a binary image. A plot of the projected sum in Y direction, sum of image rows per column, on the binary image is shown at the bottom. A plot of the projected sum in X direction, sum of image columns per row, on the binary image is shown at the right. The binary image is for Tray of AV and AV Cap. Note that the tub under the tray is in bright color that may affect the detection of Y2, so Y2 of the bounding box is inferred from XI, X2, and Y 1. Given a fixed camera view angle, the aspect ratio of the bounding box for a tray is fixed. XI and X2 define the width of a bounding box. The aspect ratio of the bounding box infers the height of a bounding box. Given Yl, Y2 is calculated using the inferred height of the bounding box. Given (XI, Yl), and (X2, Y2), a bounding box is drawn.

[0294] FIG. 50 illustrates another example projected sums on a binary image. At the right side of FIG. 50, a plot of the projected sum in Y direction, sum of image rows per column, on the binary image is shown at the bottom, and a plot of the projected sum in X direction, sum of image columns per row, on the binary image is shown at the right. At the left side of FIG. 50, the gamma-corrected tub image of Wash Vessel is shown. Note that the whole yellow tub is bright that may affect the detection of Y2. Thus, Y2 of the bounding box is inferred from XI, X2, and Yl. Given a fixed camera view angle, the aspect ratio of the bounding box for the horizontal top surface of a tub is fixed. XI and X2 define the width of a bounding box. The aspect ratio of the bounding box infers the height of a bounding box. Given Yl, Y2 is calculated using the inferred height of the bounding box. Given (XI, Yl), and (X2, Y2), a bounding box may be drawn.

[0295] FIG. 51 illustrates four template images and four search regions for template matching algorithm. The search region per comer is determined by the bounding box corner of the tray. Within each search region, the best matched location is found by comparing the template image and a sub-region at an image coordinate inside the search region.

[0296] FIG. 52 illustrates first example detected bounding boxes and detected source anchor points. Bounding box is drawn in a box and source anchor points are in dots. Left is a tray of AV and AV Cap. Right is a tub of Wash Vessel. FIG. 53 illustrates second example detected bounding boxes and detected source anchor points. Bounding box is drawn in a box and source anchor points are in dots. Left is a tray of Lysis Tube and Transfer Tip. Right is a tray of Eluate Tip.

[0297] FIG. 54 illustrates a third example detected bounding box and detected source anchor points. Bounding box is drawn in a box and source anchor points are in dots. The image is from a tray of Sample Tip.

[0298] FIG. 55 illustrates an example perspective correction on the tray of Lysis Tube and Transfer Tip. The image was taken in blue channel. The left picture is the tray picture taken from a tilted camera with perspective distortion. The source anchor points are displayed as the circles. The right picture is the tray picture after the perspective transform. The destination anchor points are displayed as the circles.

[0299] The tub of Wash Vessel has 6 Wash Vessels per slot. For a tub with multiple consumables per slot, a special relative position between the camera and the tub is designed. This relative position will allow the consumables to show difference with respect to the number of consumables per slot.

[0300] FIG. 56 illustrates an example perspective correction on the tub of Wash Vessel. The picture was taken in blue channel. The left picture is the tub picture taken from a tilted camera with perspective distortion. The source anchor points are displayed as the circles. The right picture is the tray picture after the perspective transform. The destination anchor points are displayed as the circles.

[0301] Example 7

[0302] This example illustrates exemplary determining subject attributes associated with an element position based on a plurality of template images having a common template attribute. If a similarity score of an element position for a corresponding ROI from any of the plurality of template images is greater than or equal to a respective threshold, the subject attributes associated with the element position may be determined to include the common template attribute. If not, the subject attributes associated with the element position may be determined not to include the common template attribute. The cap detection algorithm of test tubes is performed by a process of elimination. For each tube holder on a rack, there are 3 possibilities: empty, an uncapped test tube, or a capped test tube. Therefore, for each tube holder, if an algorithm can eliminate the first two possibilities, then a capped test tube must be in that holder. To eliminate an empty holder, a dynamic template matching algorithm with the image of the empty rack is applied to the perspective corrected image. To eliminate an uncapped test tube, a dynamic template matching algorithm with the images of various uncapped test tubes is applied to the perspective corrected image. Three types of test tubes are used to illustrate this algorithm: Long, long slim, and short.

[0303] The test-tube picture taken from a tilted camera has geometric distortion, even after perspective correction. Therefore, it may not be robust to use one fixed template image of the uncapped test tube to detect every tube holder on the rack. This is the motivation to utilize the dynamic template matching algorithm. For every location on the rack, a template image of an uncapped test tube was cropped from the corresponding location of a full rack, so it is a dynamic template.

[0304] FIG. 20 to FIG. 23 illustrate the first step of dynamic template matching algorithm: Preparation of pre-determined ROIs. During the development time, images of an empty rack, six uncapped long test tubes, six uncapped long slim test tubes, and six uncapped short test tubes are acquired. Then, Gamma correction and perspective correction are applied to those images. The destination anchor points are selected by the algorithm developer, and they are fixed per rack type, so the following pre-determined ROIs can be created during the development time. The pre-determined ROI for empty tube holders, as shown in the right picture of FIG. 20; The pre-determined ROI for six uncapped long test tubes, as shown in the right picture of FIG. 21; The pre-determined ROI for six uncapped long-slim test tubes, as shown in the right picture of FIG. 22; The pre-determined ROI for six uncapped short test tubes, as shown in the right picture of FIG. 23.

[0305] Note that all the pre-determined ROIs are fixed on the test-tube images per rack type, so they are used to crop the template image per location for the template matching algorithm in the real time. The optimal pre-determined ROI for those uncapped test tubes is selected to cover only the top half in the opening due to that those uncapped test tubes are transparent. Such pre-determined ROIs to cover the whole opening or to the lower half in the opening will cause misdetections frequently. If the target object to detect is not transparent, then the optimal pre-determined ROI should include edges of this whole object, as shown in the empty tube holders of FIG. 20.

[0306] FIG. 28 summarizes the algorithms for cap detection of test tubes. Gamma correction and perspective transform are applied to the input rack image. Then, the processing block, “Dynamic Template Matching to Check Empty Holder”, is applied to the perspective transformed (warped) image after the perspective transform. Then, a preset threshold is used to identify if a slot is empty or occupied. If a slot is empty, then the index of this slot is -1. If a slot is occupied, then the processing block, “Dynamic Template Matching to Check Uncapped Tube”, is applied to detect an uncapped test tube per slot. Three template images from three tube types are used to detect an uncapped test tube per slot. Then, a preset threshold per tube type is used to identify if a slot has an uncapped test tube. If a slot has an uncapped test tube, then the index of this slot is 0. If a slot does not have an uncapped test tube, then this slot has a capped test tube, and the index of this slot is 1. FIG. 29 shows the details in the processing block, “Dynamic Template Matching to Check Empty Holder”. The right picture 2902 shows a portion of an empty- rack image, which serves as a template image of the empty tube holders. Dashed boxes 2920 are used to crop one tube holder ROI per slot. The left picture 2901 shows a portion of the input test-tube image. The center coordinates of the dashed box 2910 and the dashed box 2920(b) of the tube holder are the same per slot. The dashed box 2910 has twice size of the dashed box 2920 per slot, and it search region to apply a tube holder template. One 2-D image surface of normalized cross-correlation, y tube holder, is generated per search region. Within each search region, y tube holder max is found, y tube holder max estimates the highest similarity between the tube holder ROI and a sub-region at the same size within the search region. If y tube holder max is less than a preset threshold, then this slot is occupied.

[0307] FIG. 30 shows the details in the processing block, “Dynamic Template Matching to Check Uncapped Tube” using a full-rack image of six uncapped long test tubes. The center coordinates of the dashed box 3010 and the dashed box 3020 of the uncapped long test tube are the same per location. The dashed box 3010 has twice size of the dashed box 3020 per location, and it is the search region to apply an uncapped long test tube template. One 2-D image surface of normalized cross-correlation, y tubeLong, is generated per search region 3010. Within each search region, y tubeLong max is found. The y tubeLong max estimates the highest similarity between the uncapped long test tube ROI and a sub-region at the same size within the search region 3010. If y tubeLong max is greater than a preset threshold, then this slot is occupied by an uncapped long test tube.

[0308] FIG. 31 shows the details in the processing block, “Dynamic Template Matching to Check Uncapped Tube” using a full-rack image of six uncapped long-slim test tubes. The center coordinates of the dashed box 3110 and the dashed box 3120 of the uncapped long-slim test tube are the same per location, the dashed box 3110 has twice size of the dashed box 3120 per location, and it is the search region to apply an uncapped long- slim test tube template. One 2-D image surface of normalized cross-correlation, y tubeLongSlim, is generated per search region. Within each search region, y tubeLongSlim max is found. The y tubeLongSlim max estimates the highest similarity between the uncapped long-slim test tube ROI and a sub-region at the same size within the search region. If y tubeLongSlim max is greater than a preset threshold, then this slot is occupied by an uncapped long-slim test tube.

[0309] FIG. 32 shows the details in the processing block, “Dynamic Template Matching to Check Uncapped Tube” using a full-rack image of six uncapped short test tubes. The center coordinates of the dashed box 3210 and the dashed box 3220 of the uncapped short test tube are the same per location. The dashed box 3210 has twice size of the dashed box 3220 per location, and it is the search region to apply an uncapped short test tube template. One 2-D image surface of normalized cross-correlation, y tubeShort, is generated per search region. Within each search region, y tubeShort max is found. The y tubeShort max estimates the highest similarity between the uncapped short test tube ROI and a sub-region at the same size within the search region. If y tubeShort max is greater than a preset threshold, then this slot is occupied by an uncapped short test tube.

[0310] FIG. 37 shows a histogram of y tubeHolder max using 342 data points, where the highlighted data points 3702 are for slots occupied by test tubes. For each slot, if y tubeHolder max is greater than a preset threshold, then this slot is empty. Otherwise, this slot is occupied by a test tube. Based on the histogram, a proper preset threshold is 0.85.

[0311] FIG. 38 to FIG. 40 show how to determine a preset threshold per tube type. FIG. 38 shows a histogram of the maximum normalized cross-correlations for the uncapped long test tube templates. The failure cases, which mean a slot is not occupied by an uncapped long test tube, are highlighted. Two groups of data points are well-separated. A proper preset threshold is 0.825.

[0312] FIG. 39 shows a histogram of the maximum normalized cross-correlations for the uncapped long-slim test tube templates. The failure cases, which mean a slot is not occupied by an uncapped long-slim test tube, are highlighted. Two groups of data points are well-separated. A proper preset threshold is 0.8.

[0313] FIG. 40 shows a histogram of the maximum normalized cross-correlations for the uncapped short test tube templates. The failure cases, which mean a slot is not occupied by an uncapped short test tube, are highlighted. Two groups of data points are well-separated. A proper preset threshold is 0.65.

[0314] 8 Image Quality Monitoring Functions for Machine Vision System

[0315] Various factors can degrade the image quality in a machine vision system. Images at poor quality can, for example, result in determination of faulty subject attributes. For example, poor quality images can fail the loading anomaly detection algorithm or the inventory verification algorithm.

[0316] FIG. 57 illustrates an example flowchart of image quality monitoring functions. Below is a list of exemplary factors and the corresponding image quality monitoring function, respectively. The design and implementation of image quality monitoring functions can be used to provide a robust machine vision system.

[0317] Instrument shaking: If an instrument is shaken such as for example due to an earthquake or road construction outside, then motion blur can appear on the image. An image quality monitoring function can be used to detect blurry images.

[0318] Portions of a diagnostic instrument having the camera or a subject container can be in motion during image capture. For example, a carousal, or trays, or robot arm, can move during image capture: If the carousal or tray or robot arm is moving during the image capture, then motion blur can appear on the image. An image quality monitoring function can be used to detect blurry images.

[0319] Wrong focus distance, for example, due to failure in switching camera job files during image capture: If a wrong focus distance is set on the camera, then the image can be out of focus, or look blurry. An image quality monitoring function can be used to detect blurry images.

[0320] Dirty camera: If, for example, the camera’s lens is covered by splashes, such as from reagents or other liquids within the diagnostic instrument, then images can be blurry. An image quality monitoring function can be used to detect blurry images.

[0321] LED broken or dim: If the illumination device is aged or broken, then images can be dark. An image quality monitoring function can be used to detect dark images.

[0322] Unexpected subject containers or elements, such as for example, counterfeit consumables or trays: If, for example, counterfeit consumables or trays are used in the instrument, the properties of the counterfeit consumables or trays may not align with the properties of trays used to capture template images. This can, for example, cause false alarms in the loading anomaly detection, or miscounts in the inventory verification. An image quality monitoring function can be necessary to detect counterfeit consumables or trays.

[0323] 9 Machine Vision System Software Design

[0324] This machine vision system can use any suitable software. For purpose of example, and as embodied herein, the camera and the robot software can be shared. For example, in real time, the robot software or the barcode scanning camera software informs the vision software about what object type is at scene. If the object type is a rack of tubes, the vision software will apply the cap detection algorithm to the captured image. The software design for the inventory machine vision system is disclosed herein. For purpose of example, and as embodied herein, a consumables inventory vision controller software module is utilized by a reagent sample consumables manager transport (RSCM) robot to determine the number of consumables in a consumable tray and the locations of the available consumable in the tray and the number of consumables available in a slot if the consumable is stacked. The availability matrix generated by the consumable inventory machine vision system is passed to the Consumable Pick and Place robot when a consumable tray is loaded for pick and place. This machine vision system can use a digital camera mounted at the consumable inventory station to capture images and use inventory vision algorithm to generate map of consumable availability at the designated locations in a tray.

[0325] FIG. 58 illustrates an example architecture of the machine vision software. The Machine Vision software can be implemented using a client - server architecture over an inter-process communication mechanism of message-passing between consumable loader software, machine vision camera driver and inventory vision controller. The camera driver can communicate with the camera via Hypertext Transfer Protocol (HTTP) Restful application programming interface (REST API) over Transport Control Protocol / Internet Protocol (TCP / IP).

[0326] FIG. 59 illustrates an example sequence diagram. The sequence diagram shows how a consumable loader robot control software can utilize the machine vision camera driver and the vision controller that implements the inventory algorithm to detect consumable availability for pick and place for various commodities used in the instrument. As embodied herein, the algorithm can generate a matrix for every type of consumables. For example and as embodied herein, in the trays where there are paired consumables, two matrices can be generated, one of each type of consumable in the tray. As further embodied herein, the matrices can be collapsed together to make a single availability matrix of usable consumable pairs before the result is sent back to the RSCM robot.

[0327] An exemplary method to determine the next picking spot, such as for consumables that are not stacked, can include a 2D matrix. For example, the vision algorithm can output an inventory map per consumable type, which can be a 2D matrix, matrix l. There are only 0’s and l’s in this matrix. “0” means that a spot is empty. “1” means that a spot is occupied. Given a map of picking order for the same consumable type, which is also a 2D matrix, matrix_2, there are integers from 1 to N in this matrix, N is the total number per tray. In the real time, to get the next picking spot, the method is to multiply two matrices element-wisely. The output matrix can have zeros and non-zeros. The location with the lowest non-zero value will be the next picking spot.

[0328] In addition to the specific embodiments claimed below, the disclosed subject matter is also directed to other embodiments having any other possible combination of the dependent features claimed below and those disclosed above. As such, the certain features presented in the dependent claims and disclosed above can be combined with each other in other manners within the scope of the disclosed subj ect matter such that the disclosed subj ect matter should be recognized as also specifically directed to other embodiments having any other possible combinations. Thus, the foregoing description of specific embodiments of the disclosed subject matter has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosed subject matter to those embodiments disclosed.

[0329] It will be apparent to those skilled in the art that various modifications and variations can be made in the method and system of the disclosed subject matter without departing from the spirit or scope of the disclosed subject matter. Thus, it is intended that the disclosed subject matter include modifications and variations that are within the scope of the appended claims and their equivalents.

Claims

CLAIMS1. A machine vision method for a subject container, comprising: accessing a subject digital image of the subject container comprising N pre-determined element positions; accessing one or more template digital images including a first region of interest (ROI) having a first attribute and one or more second ROIs having a common second attribute; calculating for a first element position among the N pre-determined element positions in the subj ect digital image a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image; determining whether one or more first subject attributes associated with the first element position include the first attribute based on the first similarity score; calculating for the first element position, in response to a determination that the one or more first subject attributes associated with the first element position do not include the first attribute, one or more second similarity scores for the one or more second ROI having a common second attribute; and determining whether the first subject attributes associated with the first element position include the common second attribute based on the second similarity scores.

2. The method of Claim 1, wherein a pre-determined search region corresponding to an ROI is defined for the first element position among the N pre-determined element positions in the subject digital image, the pre-determined search region having one or more sub-regions of a size of the ROI, wherein calculating a similarity score of the first element position for the ROI comprises: calculating, for each of the one or more sub-regions, a normalized 2-dimensional (2D) correlation coefficient between the sub -region and the ROI; anddetermining a highest normalized 2D correlation coefficient among the calculated normalized 2D correlation coefficients as the similarity score of the first element position for the ROI.

3. The method of Claim 1, wherein the first ROI contains an empty element position, and wherein the first attribute indicates the element position being empty.

4. The method of Claim 1, wherein the one or more template digital images comprise k second ROIs, each containing an element of a respective element type having the common second attribute, wherein & is a number of possible element types, and wherein the method further includes determining which, if any, of the k element types is present at the first element position based on the second similarity scores.

5. The method of Claim 4, wherein a determination that the one or more first subject attributes associated with the first element position do not include the common second attribute is made when the second similarity scores indicate that none of the k element types having the common second attribute are present at the first element position.

6. The method of Claim 1, wherein the subject container is configured to receive tubes at each of the N pre-determined positions, the tubes being either capped or uncapped, and wherein the common second attribute includes an uncapped tube.

7. The method of Claim 1, further comprising: determining an exposure mode;determining optimal exposure time and sensor gain for the determined exposure mode during a camera calibration period; and taking, using a digital camera, the subject digital image of the subject container located at a pre-determined relative location from the digital camera with the determined optimal exposure time and sensor gain.

8. The method of Claim 7, wherein a camera calibration of the digital camera is performed during the camera calibration period.

9. The method of Claim 8, wherein the camera calibration comprises: taking a calibration digital image of a calibration target container at a pre-determined relative location from the digital camera; determining coordinates of a plurality of source anchor points of the calibration target container within the calibration digital image; registering the coordinates of the plurality of source anchor points; and creating a perspective transform matrix that linearly relocates the plurality of source anchor points to respective pre-determined destination anchor points.

10. The method of Claim 1, further comprising: performing, before calculating the first similarity score, a perspective transform of the subject digital image to define a transformed subject digital image.

11. The method of Claim 10, wherein the one or more template digital images include one or more perspective transformed template container digital images of a template container comprising the N pre-determined element positions, each associated with a ROIand having a template attribute uniquely associated with one of the one or more template digital images, and wherein the first ROI and the one or more second ROIs are associated with the first element position of the N pre-determined element positions in the one or more perspective transformed template container digital image.

12. The method of Claim 11, wherein the one or more perspective transformed template container digital images include ROIs having the first attribute and the common second attribute associated with each of the N pre-determined element positions.

13. The method of Claim 12, comprising calculating, for each of the N pre-determined element positions in the transformed subject digital image, a first attribute similarity score for at least one of the one or more perspective transformed template container digital images by comparing digital content in an associated ROI of the one or more perspective transformed template container digital image having the first attribute and corresponding digital content in the transformed subject digital image; and determining, for each of the N pre-determined element positions in the transformed subject digital image of the subject container, whether one or more first subject attributes associated with the respective element position includes the first attribute based on the first similarity score.

14. The method of Claim 13, further comprising calculating, for each of a subset of theN pre-determined element positions for which a determination was made that the position does not include the first attribute, one or more second similarity scores for the associated ROI having the common second attribute; anddetermining whether the first subject attributes associated with each of the subset of the N pre-determined element positions include the common second attribute based on the second similarity scores.

15. The method of Claim 11, wherein the one or more perspective transformed template container digital images includes a first perspective transformed template container digital image having each of the N pre-determined element positions empty and a second perspective transformed template container digital image having each of the N predetermined element positions loaded with an element of a respective element type having the common second attribute.

16. The method of Claim 10, wherein the subject digital image is with perspective distortion because the subject digital image is taken by the digital camera with a tilted view angle, and wherein the perspective transform corrects the perspective distortion of the subject digital image.

17. The method of Claim 9, wherein a perspective transform matrix used for the perspective transform is calculated based on current coordinates of the plurality of source anchor points in the subject digital image.

18. The method of Claim 9, wherein a perspective transform matrix used for the perspective transform is pre-calculated based on the registered coordinates of the plurality of source anchor points and applied to the subject digital image.

19. The method of Claim 9, wherein the camera calibration is performed for each of one or more container types, and wherein the coordinates of the plurality of source anchor points are registered per container type.

20. The method of Claim 9, wherein determining coordinates of a plurality of source anchor points of the calibration target container comprises: generating a brightness-improved calibration digital image by performing a gamma correction to the calibration digital image; generating a binary calibration digital image by performing Otsu Thresholding algorithm on the brightness-improved calibration digital image, each pixel in the binary calibration digital image belonging to a foreground or background; detecting a bounding box of the calibration target container based on projected sums on the binary calibration digital image; and determining the coordinates of the plurality of source anchor points by performing, for each corner of the bounding box, a template matching with a respective template image, wherein a search region for the comer is determined based on the bounding box.

21. The method of Claim 1, wherein the subject digital image comprises a single-color channel, and wherein the single-color channel producing highest contrast is selected among red, green, blue (RGB) channels.

22. The method of Claim 1, further comprising: performing a gamma correction to the subject digital image to improve brightness of the subject digital image before calculating the first similarity score.

23. A machine vision system for a subject container, the system comprising: a camera having a field of view; at least a portion of a diagnostic instrument positioned within the field of view and configured to receive the subject container, the subject container comprising N predetermined element positions; a data processing unit operatively coupled to the diagnostic instrument, and a memory storing instructions which, when executed by the processing unit, cause the system to: capture, using the camera, a subject digital image including an image of the subject container; access one or more template digital images including a first region of interest (ROI) having a first attribute and one or more second ROIs having a common second attribute; calculate for a first element position among the N pre-determined element positions in the subject digital image a first similarity score for the first ROI by comparing digital content in the first ROI and corresponding digital content in the subject digital image; determine whether one or more first subject attributes associated with the first element position include the first attribute based on the first similarity score; calculate for the first element position, in response to a determination that the one or more first subject attributes associated with the first element position do not include the first attribute, one or more second similarity scores for the one or more second ROI having a common second attribute; and determine whether the first subject attributes associated with the first element position include the common second attribute based on the second similarity scores.

24. The system of claim 23, wherein the subject container is a rack, the rack being configured to receive a tube at each of the N pre-determined positions.

25. The system of claim 23, wherein the first ROI contains an empty element position, and wherein the first attribute indicates the first element position being empty.

26. The system of claim 24, wherein tubes can be either capped or uncapped, and wherein the common second attribute includes an uncapped tube.

27. The system of claim 26, wherein the one or more template digital images comprise k second ROIs, each containing a tube of a respective tube type having the common second attribute, wherein & is a number of possible tube types.

28. The system of claim 27, wherein a determination that the one or more first subject attributes associated with the first element position do not include the common second attribute is made when the second similarity scores indicate that none of the k tube types having the common second attribute are present at the first element position.

29. The system of claim 28, wherein a determination that the one or more first subject attributes associated with the first element position do not include the common second attribute includes a determination that a tube received in the first element position is capped.

30. The system of claim 23, wherein the camera has a tilted view angle of the subject container.

31. The system of claim 23, further comprising one or more areas defining an area axis, the area axis extending through a center of a subject container received at the area, and perpendicular to a plane defined by a mouth of the subj ect container, and wherein the camera is offset from the area axis.

32. The system of claim 23, wherein the at least a portion of the diagnostic instrument is configured to receive the subject container at a first area and to receive a second subject container comprising O pre-determined element positions at a second area, and wherein the instructions, when executed by the processing unit, cause the system to: capture, using the camera, a second subject digital image including an image of the second subject container; and determine, for each of the O pre-determined element positions in the image of the second subject container, one or more second container subject attributes.

33. The system of claim 23, wherein the instructions cause the system to perform, before calculating the first similarity score, a perspective transform of the subject digital image to define a transformed subject digital image.

34. The system of claim 23, wherein the one or more template digital images include one or more perspective transformed template container digital images of a template container comprising the N pre-determined element positions, each associated with a ROI and having a template attribute uniquely associated with one of the one or more template digital images, and wherein the first ROI and the one or more second ROIs are associated with the first element position of the N pre-determined element positions in the one or more perspective transformed template container digital image.

Images