Methods and systems for tracking labware

Abstract

The present disclosure relates to a system for tracking a labware product in an ART process. The system includes one or more cameras that generate images of one or more areas of a facility. The images of the one or more areas of the facility are processed to determine a transfer of the labware product from a first location within the facility to a second location within the facility. Information associated with the transfer of the labware product is outputted.

Description

METHODS AND SYSTEMS FOR TRACKING LABWARECLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 608,459 filed on December 11, 2023, the entire contents of which are hereby incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure relates to methods and systems for tracking labware in an assisted reproductive technology (ART) process.BACKGROUND

[0003] To prevent mistakes during assisted reproductive technology (ART) processes, automated witnessing systems may be used. The automated witnessing systems generally involve assigning each piece of labware a unique identifier (e.g., a unique number), typically by labelling the labware with a machine-readable tag (e.g., a radio frequency identification (RFID) tag) and having readers in areas in the lab that can read these tags and thus identify the dishes. The unique identifier of the lab ware can be linked to a patient associated with a biological material undergoing the ART process.SUMMARY

[0004] The present disclosure is directed to systems and methods for tracking labware during an assisted reproductive technology (ART) process. The systems disclosed herein can provide automated tracking that reduces instances of error and the need for a witness each time a labware product is moved from one location within a medical facility, such as an ART lab, to another.

[0005] In one aspect, a system for tracking a labware product in an assisted reproductive technology (ART) process includes one or more cameras, one or more computers, and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for tracking the labware product, the operations comprising: receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, determining, based on the processed images ofthe one or more areas of the facility, a transfer of the labware product from a first location within the facility to a second location within the facility, and outputting information associated with the transfer of the lab ware product.

[0006] In another aspect, a labware product for use in an assisted reproductive technology (ART) process includes an optically transparent body for holding a biological material and an upward facing visual identifier on a bottom surface of the optically transparent body. The upward facing visual identifier includes a machine-readable code that is processable for determining one or both of a unique identifier for the labware product and a location of the labware product.

[0007] In a further aspect, a labware product for use in an assisted reproductive technology (ART) process includes an optically transparent body for holding a biological material and an upward facing visual identifier on a bottom surface of the optically transparent body. The upward facing visual identifier includes a machine-readable code that is processable for determining one or both of a unique identifier for the labware product and one or more subjects associated with the labware product.

[0008] In yet another aspect, a method performed by one or more computers for tracking a labware product in an assisted reproductive technology (ART) process includes receiving, from one or more cameras, images of one or more areas of a facility; processing the images of the one or more areas of the facility; determining, based on the processed images of the one or more areas of the facility, a transfer of the labware product from a first location within the facility to a second location within the facility; and outputting information associated with the transfer of the labware product.

[0009] In an additional aspect, a system for tracking one or more labware products in an assisted reproductive technology (ART) process includes one or more cameras, one or more computers, and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for tracking the one or more labware products, the operations comprising: receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, processing data from a witnessing system, determining, based on the processed images of the one or more areas of the facility and the data from the witnessing system, whether any of the one or more labware products were imaged by the one or more cameras but were notrecognized by the witnessing system, and outputting information associated with a location of the labware products.

[0010] In another aspect, a method performed by one or more computers for tracking one or more labware products in an assisted reproductive technology (ART) process includes receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, processing data from a witnessing system, determining, based on the processed images of the one or more areas of the facility and the data from the witnessing system, whether any of the one or more labware products were imaged by the one or more cameras but were not recognized by the witnessing system, and outputting information associated with a location of the labware products.

[0011] In a further aspect, a system for monitoring an assisted reproductive technology (ART) process includes one or more cameras; one or more computers; and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, determining, based on the processed images of the one or more areas of the facility, whether a cleaning process of the one or more areas of the facility has been carried out, and outputting information associated with cleaning process.

[0012] In yet another aspect, a method performed by one or more computers in an assisted reproductive technology (ART) process includes receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, determining, based on the processed images of the one or more areas of the facility, whether a cleaning process of the one or more areas of the facility has been carried out, and outputting information associated with the cleaning process.

[0013] The above aspects can include one or more of the following features.

[0014] In some embodiments, the one or more cameras comprises a first camera configured to generate a first set of images of the first location and a second camera configured to generate a second set of images of the second location.

[0015] In some embodiments, the one or more cameras comprise at least one of a wide- view camera or a camera with an auto adjustable zoom lens.

[0016] In some embodiments, the system includes a workstation having a worksurface comprising the first and second locations.

[0017] In some embodiments, the first location is within a heated portion of the worksurface, and the second location is within an unheated portion of the worksurface.

[0018] In some embodiments, the first location is within a portion of the worksurface that is monitored by a witnessing system and the second location is outside the portion of the worksurface that is monitored by the witnessing system.

[0019] In some embodiments, the witnessing system comprises a radio frequency identifier (RFID) reader configured to read an RFID tag on the labware product when the labware product is within the area that is monitored by the witnessing system.

[0020] In some embodiments, the witnessing system comprises a barcode reader configured to read a barcode on the labware product when the labware product is within the area that is monitored by the witnessing system.

[0021] In some embodiments, the one or more cameras are attached to the workstation and are configured to be oriented at a plurality of angles relative to the worksurface.

[0022] In some embodiments, at least one of the one or more cameras comprise at least one camera positioned outside the workstation.

[0023] In some embodiments, the labware product comprises an upward facing label that is readable by at least one of the one or more cameras, and processing the images of the one or more areas of the facility comprises processing the upward facing label to determine an identifier of the labware product.

[0024] In some embodiments, the upward facing label comprises a code indicative of the identifier of the labware product, wherein the code is a machine generated code.

[0025] In some embodiments, the machine generated code comprises a bar code or a two- dimensional matrix barcode.

[0026] In some embodiments, the labware product comprises a RFID tag, and the system comprises a RFID reader configured to read the RFID tag of the labware product and to transmit data associated with the RFID tag to the one or more computers.

[0027] In some embodiments, the data associated with the RFID tag is associated with the labware product being tracked.

[0028] In some embodiments, the labware product comprises a barcode, and the system comprises a barcode reader configured to read the barcode of the labware product and to transmit data associated with the barcode to the one or more computers.

[0029] In some embodiments, the data associated with the barcode is associated with the labware product being tracked.

[0030] In some embodiments, the system includes a database storing a standard operating protocol, wherein the operations comprise determining whether the transfer of the labware product from the first location to the second location is in compliance with the standard operating protocol stored in the database.

[0031] In some embodiments, outputting the information comprises generating an alert signaling non-compliance with the standard operating protocol.

[0032] In some embodiments, the system includes a sensor configured to detect one or more environmental parameters at the first location or the second location, the one or more environmental parameters comprising a temperature, a gas concentration, or a humidity level.

[0033] In some embodiments, the operations further comprise determining, based on the processed images of the one or more areas of the facility, an additional transfer of the labware product within the facility from the second location to a third location.

[0034] In some embodiments, the one or more cameras track the transfer of the labware product from the first location to the second location to the third location.

[0035] In some embodiments, the one or more cameras comprises a first camera configured to image the first location, a second camera configured to image the second location, and a third camera configured to image the third location.

[0036] In some embodiments, the one or more cameras and the one or more computers are configured to record a cleaning process of at least one of the one or more areas of the facility.

[0037] In some embodiments, the one or more cameras and the one or more computers are configured to recognize and track wipes used to carry out the cleaning process.

[0038] In some embodiments, the one or more cameras and the one or more computers are configured to track movement of the wipes during the cleaning process to identity portions of the at least one of the one or more areas of the facility that were not cleaned.

[0039] In some embodiments, the portions of the at least one of the one or more areas of the facility that were not cleaned are displayed and marked on a user interface to assist a user to complete the cleaning process.

[0040] In some embodiments, the one or more cameras and the one or more computers are configured to read a barcode on the labware product and associate the barcode with the labware product.

[0041] In some embodiments, the one or more cameras and the one or more computers are configured to automatically read the barcode when the labware product is moved into a viewing area of the one or more cameras.

[0042] In some embodiments, the one or more cameras and the one or more computers are configured to read a barcode on packaging of the labware product and associate the barcode with the labware product.

[0043] In some embodiments, the one or more cameras and the one or more computers are configured to automatically read the barcode when the packaging of the labware product is moved into a viewing area of the one or more cameras.

[0044] In some embodiments, the one or more computers are configured to monitor an inventory of labware products.

[0045] In some embodiments, the one or more computers are configured to record the labware product as being used in the ART process being carried out.

[0046] In some embodiments, the machine-readable code is readable by at least one camera positioned above the labware product.

[0047] In some embodiments, the upward facing visual identifier comprises a code indicative of the unique identifier of the labware product, wherein the code is a machine generated code.

[0048] In some embodiments, the machine-generated code comprises a bar code or a two- dimensional matrix barcode.

[0049] In some embodiments, the upward facing visual identifier is an upward facing label adhered to a bottom surface of the lab ware product.

[0050] In some embodiments, the upward facing visual identifier is etched in or printed on a bottom surface of the labware product.

[0051] In some embodiments, the labware product comprises a radio frequency tag that comprises a radio frequency identifier and is configured to be detected by a radio frequency identifier reader.

[0052] In some embodiments, the radio frequency identifier can be processed for one or both of identifying the labware product or identifying one or more subjects associated with the labware product.

[0053] In some embodiments, the labware product includes a visual characteristic configured to be processed for determining the location of the labware product.

[0054] In some embodiments, the method includes determining the location of the labware product by reading a RFID tag associated with the labware product.

[0055] In some embodiments, the one or more cameras comprise at least one of a wide- view camera or a camera with an auto adjustable zoom lens.

[0056] In some embodiments, the method includes generating a reconstructed image of the one or more areas of the facility by combining portions of the images acquired by the one or more cameras.

[0057] In some embodiments, the first and second locations are parts of a worksurface of an ART workstation.

[0058] In some embodiments, at least one of the one or more cameras is attached to the workstation and configured to be oriented at a plurality of angles relative to the ART worksurface.

[0059] In some embodiments, at least one of the one or more cameras is positioned outside the ART workstation.

[0060] In some embodiments, the method includes determining whether the transfer of the labware product from the first location to the second location is in compliance with a standard operating protocol stored in a database of a memory.

[0061] In some embodiments, outputting the information associated with the transfer of the labware product comprises generating an alert signaling non-compliance with the standard operating protocol.

[0062] In some embodiments, the labware product is a first labware product that is associated with one or more persons related to the ART process, and the alert signals that the first labware product is in proximity to a second labware product that is not associated with the one or more persons with whom the first labware product is associated.

[0063] In some embodiments, the alert signals that the labware product has been outside of the first location for greater than a predetermined period of time.

[0064] In some embodiments, the first location is a heated area of a workstation in the facility.

[0065] In some embodiments, the alert signals that the labware product has been in the second location for greater than a predetermined period of time.

[0066] In some embodiments, the second location is an unheated area of a workstation in the facility.

[0067] In some embodiments, the method includes determining one or more environmental parameters of the first location or the second location, the one or more environmental parameters comprising a temperature, a gas concentration, or a humidity level.

[0068] In some embodiments, the method includes receiving, from a reader, data associated with a tag of the labware product, and determining, based on the data associated with the tag of the labware product, (i) one or more persons associated with the labware product and (ii) that the labware product is in the first location.

[0069] In some embodiments, the labware product is a first labware product, and outputting the information associated with the transfer of the first labware product comprises generating an alert signaling that the first labware product is in proximity to a second labware product that is not associated with the one or more persons with whom the first labware product is associated.

[0070] In some embodiments, the reader is a RFID reader, and the tag is a RFID tag.

[0071] In some embodiments, the reader is part of a witnessing system.

[0072] In some embodiments, the method includes recording, using the one or more cameras, a cleaning process of at least one of the one or more areas of the facility.

[0073] In some embodiments, the method includes recognizing and tracking, using one or more computers, wipes used to carry out the cleaning process.

[0074] In some embodiments, the method includes tracking, using the one or more computers, movement of the wipes during the cleaning process to identity portions of the at least one of the one or more areas of the facility that were not cleaned.

[0075] In some embodiments, the method includes displaying and marking, on a user interface, the portions of the at least one of the one or more areas of the facility that were not cleaned to assist a user to complete the cleaning process.

[0076] In some embodiments, the method includes reading a barcode on the labware product and associating the barcode with the labware product.

[0077] In some embodiments, the method includes automatically reading the barcode when the labware product is moved into a viewing area of the one or more cameras.

[0078] In some embodiments, the method includes reading a barcode on packaging of the labware product and associating the barcode with the labware product.

[0079] In some embodiments, the method includes automatically reading the barcode when the packaging of the labware product is moved into a viewing area of the one or more cameras.

[0080] In some embodiments, the method includes monitoring an inventory of labware products based on the reading of the barcode.

[0081] In some embodiments, the method includes recording that the labware product is used in the ART process being carried out.

[0082] In some embodiments, the one or more cameras and the one or more computers are configured to recognize and track wipes used to carry out the cleaning process.

[0083] In some embodiments, the one or more cameras and the one or more computers are configured to track movement of the wipes during the cleaning process to identity portions of the at least one of the one or more areas of the facility that were not cleaned.

[0084] In some embodiments, the system includes a user interface, wherein the portions of the at least one of the one or more areas of the facility that were not cleaned are displayed and marked on the user interface to assist a user to complete the cleaning process.

[0085] In some embodiments, determining whether the cleaning process of the one or more areas of the facility has been carried out comprises recognizing and tracking wipes used to carry out the cleaning process.

[0086] In some embodiments, the method includes tracking movement of the wipes during the cleaning process to identity portions of the at least one of the one or more areas of the facility that were not cleaned.

[0087] In some embodiments, the method includes displaying and marking on a user interface the portions of the at least one of the one or more areas of the facility that were not cleaned to assist a user to complete the cleaning process.

[0088] Implementations described herein can provide one or more of the following advantages.

[0089] In some implementations, labware products can be automatically tracked through a greater portion of the ART process than prior ART systems generally allowed because the tracking is not limited to areas within range of a reader (e.g., an RFID reader or a barcode reader). The automatic tracking can decrease the number of people required to witness movements of the labware products from one location to the next throughout the ART process and can reduce the likelihood of labware associated with certain persons going through the ART process being mistakenly used in procedures for other persons.

[0090] In certain implementations, the labware products can be tracked across the entire surface of an ART workstation and / or throughout an entire ART lab space. This can help to ensure that biological samples are not inadvertently mixed up during the ART process and that standard operating protocols are followed.

[0091] In some implementations, the tracking systems include a witnessing system (e.g., a radio frequency identifier (RFID) witnessing system or a barcode witnessing system) that can detect tagged lab ware products within one area of the ART lab and can identify persons (e.g., ART clients, patients, etc.) associated with those labware products. The one or morecameras can track the labware products both within the area of the ART lab in which the witnessing system is operable and within other areas of the ART lab. As such, even when the labware products are outside the range of the witnessing system, the labware products can still be tracked using the one or more cameras. This can help to ensure that only labware products associated with desired persons can enter designated areas of the ART lab (e.g., areas of the ART lab in which biological material associated with those desired persons is being analyzed or processed). The system may, for example, alert lab personnel when a labware product that is not associated with those persons enters the designated areas of the ART lab.

[0092] An advantage of implementations of the tracking assembly that include a witnessing system is that the camera system does not necessarily need to read tags for initiating tracking of labware. Instead, the camera-based tracking system can be configured to follow around labware that has been identified by the witnessing system. For example, a camera can generate an image of labware that comes into a reading area of the medical facility. A reader (e.g., an RFID reader or barcode reader) detects an identifier of the labware (e.g., an alphanumeric identifier) when the labware is moved into the reading area of the medical facility. The reader transmits the identifier of the labware to the tracking system, which can be configured to associate the identifier with the imaged labware, maintaining the association with the identifier as the labware moves across various locations (including areas outside the reading range of the reader). This arrangement may allow for the use of less expensive cameras for tracking the labware, as compared to cameras that must also read identifiers on the labware.

[0093] A technical advantage of certain implementations of the tracking system includes an adaptive data storage mechanism that enables automatic selection of a minimal amount of the acquired data for processing and extraction of processed data for storage without affecting the performance of the existent computing systems of the ART lab.

[0094] Other aspects, features, and advantages of the present disclosure will be apparent from the following detailed description, figures, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The accompanying drawings, which are incorporated in and constitute a part of this specification, show certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations.

[0096] FIG. 1 A illustrates a system for tracking labware during an assisted reproductive technology (ART) process.

[0097] FIG. IB illustrates a portion of the system of FIG. 1 A including a heated worksurface area and an unheated worksurface area.

[0098] FIG. 1C illustrates another portion of the system of FIG. 1 A including an incubator and a camera configured for tracking labware moved into and out of the incubator.

[0099] FIG. 2 is a flow chart showing various stages of the ART process relative to labware tracking.

[0100] FIG. 3 is a perspective view of a workstation configured for tracking labware using a single camera.

[0101] FIG. 4 is a flowchart of a process for tracking labware.

[0102] FIG. 5A is a perspective view of a labware product in the form of a dish including an upward facing label that allows the dish to be tracked via one or more cameras.

[0103] FIG. 5B is a top view of another labware product in the form of a dish that includes an upward facing label, an RFID tag, and a marking that can be used for tracking the dish.

[0104] FIG. 6 is a flowchart of a process for tracking a labware product including an upward facing label.

[0105] FIG. 7 is a schematic diagram of another system for tracking labware products, in accordance with some example implementations.

[0106] FIG. 8 schematically illustrates a portion of a medical facility configured for tracking a labware product, in accordance with some example implementations.DETAILED DESCRIPTION

[0107] Implementations of the present disclosure are generally directed to tracking labware during an assisted reproductive technology (ART) process. More particularly, implementations of the present disclosure are directed to camera assisted tracking of labware (e.g., dishes, receptacles, pipettes, tubes, catheters, etc.) being moved between multiple different areas of a medical facility during the ART process. Certain systems and methods described herein enable continuous monitoring of identified labware throughout multiple areas of a medical facility (e.g., an ART lab), including areas that are outside the readable area of an automated witnessing system.

[0108] FIG.1 A illustrates an example system 100 for tracking labware products, such as a dish 102 and a pipette 103, within a portion of a medical facility (e.g., an ART lab) 104. Asillustrated, the lab 104 includes an ART workstation 110 and an auxiliary medical system 108 that are configured to support ART processes that involve moving multiple pieces of labware (e.g., the dish 102, the pipette 103, etc.) around the lab 104 and transferring biological material, such as sperm, eggs, and / or embryos, between different pieces of labware.

[0109] The example system 100 includes cameras 118A, 118B, 118C, 118D configured to capture images of labware (e.g., the dish 102 and the pipette 103) positioned within and moved between different areas 114A, 114B, 114C of the lab. For example, the cameras 118A, 118B, 118C can be attached to a hood of the workstation 110 to capture images of the labware within areas 114 A, 114B of the worksurface 116, and the camera 118D can be attached to the auxiliary medical system 108 to capture images of labware within the area 114C of the auxiliary medical system 108 (e.g., an area around a micromanipulator 119).

[0110] The example system 100 includes a computing system 132 disposed near the workstation 110 (e.g., underneath or beside the worksurface 116). The computing system 132 is communicatively coupled to the cameras 118A, 118B, 118C, 118D to receive images recorded by the cameras 118A, 118B, 118C, 118D, and to track multiple labware products through the different areas 114A, 114B, 114C during one or more ART processes. The computing system 132 can seamlessly track the labware products as they are moved between the different areas 114 A, 114B, 114C by processing images received from the cameras 118A, 118B, 118C, 118D.[oni] The cameras 118A, 118B, 118C, 118D can continuously generate images of the areas 114 A, 114B, 114C of the lab within their field of view. Alternatively, the cameras 118A, 118B, 118C, 118D can generate images of the areas in their fields of view in response to receiving a recording trigger generated by the computing system 132 (e.g., in response to the computing system 132 receiving an indication that an identifier of a labware product was read by a reader).

[0112] The cameras 118A, 118B, 118C, 118D are oriented to image different portions of the worksurface 113 of the workstation 110 and the auxiliary medical system 108. One or more of the cameras 118A, 118B, 118C, 118D, for example, can be configured to image labware products, such as dishes, deposited in and extracted from an incubator 120 integrated in the workstation 110. The incubator 120 can be configured to keep biological samples in dishes that aren’t being processed in a low oxygen atmosphere and at an optimal temperature (e.g., 37°C±0.1°C). Similarly, one or more of the cameras 118A, 118B, 118C, 118D can be configured to image labware products, such as dishes, moved into the vicinity of a microscope 112 of the workstation 110.

[0113] As another example, one or more of the cameras 118A, 118B, 118C, 118D can be configured to image labware products moved from the workstation 110 to the auxiliary medical system 108. In the illustrated implementation, the camera 118D, attached to the auxiliary medical system 108, is configured to image labware products moved to the micromanipulator 119. The camera 118D is mounted above the micromanipulator 119 to optimize visualization of the labware product holding the biological material being processed by the micromanipulator 119. The camera 118D may also image the area in which an incubator 121.

[0114] In the illustrated configuration, the cameras 118A, 118B, 118C are integrated in a hood 134 of the workstation 110 for tracking the labware products as they are moved to different locations on the worksurface 116. The cameras 118A, 118C are located in opposite comers of the hood 134 and are directed towards the worksurface 116. The camera 118B is in the center of the hood 134 or above the heated area 114A of the worksurface 116 that includes the microscope 112. The camera 118B is directed downwards towards the area 114A of the worksurface 116 for continuously imaging any labware placed within the heated area 114A.

[0115] In some cases, the cameras 118A, 118B, 118C, 118D are oriented to together image the entire area of the ART lab through which labware is expected to pass during the ART process. The cameras 118A, 118B, 118C, 118D can be configured to acquire a series of time-sequential images (e.g., at a set frequency, such as 5-10 Hz) that enable continuous tracking of the labware throughout the ART process. The cameras 118A, 118B, 118C, 118D can include ultra-high-definition cameras configured to generate high resolution images of the labware (and identification tags on the labware in some cases).

[0116] The cameras 118A, 118B, 118C can be arranged to have overlapping wide fields of view (FOV) to enable reconstruction of images of the labware with fully visible (unobstructed) views of the labware located on any portion of the worksurface 116, including transition areas. The cameras 118A and 118B may, for example, be oriented so that the FOV of the camera 118A overlaps partially with the FOV of the camera 118B such that labware can be continuously imaged (and thus tracked) over the entire right half of the worksurface 116. Similarly, the cameras 118B and 118C may be oriented so that the FOV of the camera 118B overlaps partially with the FOV of the camera 118C such that labware can be continuously imaged (and thus tracked) over the entire left half of the worksurface 116. The cameras 118A and 118D can also be oriented to have overlapping FOVs to allow labware to be continuously tracked between the workstation 110 and the auxiliary medical system 108.

[0117] The system 100 can be configured for monitoring conditions of the tracked labware, relative to different areas 114A, 114B, 114C of the system 100. The different areas 114 A, 114B, 114C can be used for different parts or procedures of the ART process, according to a standard protocol. For example, the area 114A under the microscope 112 (and around the microscope 112) can be heated to a set temperature, so that labware products, such as the dish 102, and the biological material held therein are maintained at an optimal temperature (e.g., 37°C) while the biological material is being viewed under the microscope 112 or waiting to be viewed under the microscope 112. The areas 114B, 114C may be unheated since the dish 102 is not expected to be placed in those areas or is not expected to be placed in those areas for extended periods of time.

[0118] As will be described in greater detail below, by tracking the labware throughout the ART lab, the system 100 can help a user, such as a lab technician, to avoid breaching certain ART protocols. The computing system 32 can, for example, store protocols that relate to the movement of labware through the ART lab and can use imaging data received from the cameras 118A, 118B, 118C, 118D to ensure compliance with those protocols. As an example, where an ART protocol prohibits labware from two different parties (e.g., two different patients, two different donors or donor pairs, etc.) from being located within the same workspace, the computing system 32 can cause the system 100 to emit an alert (e.g., an audible or visual alert) to inform the user when such an event occurs.

[0119] The labware products to be tracked by the system 100 can include radio frequency identifier (RFID) tags that can be detected by an RFID reader 126 to match the labware to persons (e.g., ART clients, patients, etc.) associated with the labware. The dish 102, for example, includes an RFID tag 125 that is readable by the RFID reader 126. Thus, as the dish 102 is placed in the heated area 114A that is monitored by the RFID reader 126, the system matches the dish 102 to the persons in the computing system’s database associated with the RFID tag 125 on the dish 102. At the same time, the camera 118A and / or the camera 118B image the dish 102 within the area 114A. The cameras 118A, 118B, 118C, 118D can continue to image the dish 102 as it is moved across various parts of the system 100 during the ART process. Because the computing system 132 previously matched the dish 102 to particular persons in the database associated with the RFID tag 125 of the dish 102, the system 100 can help to ensure that the dish 102 is not used with labware associated with other persons.

[0120] In some cases, the pipette 103 is equipped with an RFID tag and can be matched to particular persons within the database and can be tracked using the process describe above.In other cases, the pipette 103 may not include an RFID tag. In such cases, the pipette 103 can simply be monitored by the cameras 118A, 118B, 118C, 118D. If, based on the images provided by the cameras 118A, 118B, 118C, 118D, the computing system 132 determines that the pipette 103 was used with labware connected to certain persons and it appears that the embryologist may be attempting to use the pipette 103 with labware connected to other persons, then the system 100 can activate an alarm to inform the embryologist of the potential mix up. Similarly, if the pipette 103 is inadvertently left on the workstation 110 after being used for an ART procedure that has been completed, the system 100 can activate an alarm instructing the embryologist to remove the pipette 103. This can reduce the risk of the pipette 103 being inadvertently used with other persons’ labware and biological material in a subsequent ART procedure.

[0121] In some cases, the system 100 further includes one or more sensors 122 A, 122B, 122C, 122D that are configured to measure a temperature, a humidity, an air flow, an air concentration or other parameters within the ART lab. The sensors can transmit the measured parameters to the computing system 132, during labware tracking, to enable verification, by the computing system 132, of labware storage and movement in and between the different areas 114 A, 114B, 114C of the system 100 to ensure compliance with the ART process protocols related to storage conditions of the labware and biological material therein for particular periods of time. For example, the ART protocol may prohibit labware products containing biological material from being outside of the heated area 114 or the incubators 120, 121 for more than an allotted period of time. In such cases, the system 100 may alert the user as the allowed period of time comes to an end. The allowed time, as determined by the computing system 132, may differ depending on the actual temperature monitored by the sensors in the unheated area in which the labware and biological material are located. The system 100 may similarly alert the user when the labware and biological material have been exposed to certain other conditions, such as a less than ideal humidity and / or gas concentrations, for a certain period of time.

[0122] As another example of the usage of the cameras 118A, 118B, 118C, 118D for verification compliance with the ART protocol, one or more of the cameras 118A, 118B, 118C, 118D can be configured to image a cleaning protocol that can be associated with an executed or planned ART process. For example, the workstation 110 can be required to be thoroughly cleaned, with wipes, before and after handling biopsy samples for genomic testing. The cameras 118A, 118B, 118C, 118D and the example computing system 116 can be configured to record a cleaning process of the workstation 110 that can be processed bythe computing system 132. By recognizing and tracking the wipes, using the cameras 118A, 118B, 118C, 118D and the computing system 132, the trajectory followed by the wipes during the cleaning process can be tracked to identity any potentially missed areas of the workstation 110 that have not been cleaned. The areas of the workstation 110 identified as not cleaned during the cleaning process can be displayed as marked (highlighted, encircled, or annotated) on a user interface 128A, 128B to assist a user to complete the cleaning protocol.

[0123] The images captured by the cameras 118A, 118B, 118C, 118D and, optionally, the sensor data recorded by the sensors 122 A, 122B, 122C, can be processed by the computing system 132 to determine compliance with the standard protocol of the ART process. The tracking results generated by the computing system 132 can trigger a display of a feedback on one or more user interfaces 128A, 128B and / or sound an alarm using a speaker 130, if the movement of the dish 102 deviates from a standard protocol of the ART process. For example, if the dish 102 includes biological material associated with one set of subjects (e.g., parents) and enters a work area, such as the area around and under the microscope 112, that is being used to process biological material associated with another set of subjects, the lab technician can be alerted to the detected breach of standard protocol. Similarly, if the dish 102 remains outside of the heated area 114A for greater than an allotted period of time, the lab technician can be alerted to the detected breach or impending breach of standard protocol.

[0124] Even though, the system 100 illustrated in FIG. 1 A only shows the dish 102 and the pipette 103 as examples of labware tracked by the system 100, any of various other types of labware can be tracked. The computing system 132 using images recorded by the cameras 118A, 118B, 118C, 118D can, for example, track dishes, portions of dishes (e.g., lids and bases), pipettes, tubes, test tubes, microtubes, PCR tubes, PCR tube racks including multiple PCR tubes, catheters, and any other labware product that may be used in an ART procedure. The computing system 132 can be configured to detect (by processing images recorded by the cameras 118A, 118B, 118C, 118D) a labware product (e.g., dish 102) as a whole. In addition, in cases in which a labware product is separated into multiple components (e.g., a dish base and a lid), the individual components can be separately tracked (e.g., to avoid a situation in which a component from one labware product is inadvertently coupled with an unrelated labware component).

[0125] Further details regarding the tracking of labware will now be described with reference to FIGS. IB and 1C, which illustrate certain portions of the system 100. FIG. IB illustrates a configuration 100B of a portion of the workstation 110. In the configuration100B, the worksurface 116 of the workstation 110 is illustrated as including the RFID reader 126. The RFID reader 126 can be configured to be near (e.g., under) the microscope 112. The RFID reader 126 detects the RFID tag 125 associated with a respective lab ware product (e.g., dish 102) placed near (e.g., on) the RFID reader 126, which is within the heated area 114A. The RFID tag 125 can be configured to identify the dish 102. The data collected by the RFID reader 126 can be transmitted to the computing system 132 and / or a server system for further processing (conversion to a human readable format) and for being added in a human readable format (such as an alpha-numeric format), as annotations or tags to the displayed images of the respective labware product (e.g., dish 102). In some implementations, the image tracking of the dish 102, performed by the cameras 118A, 118B, 118C, 118D, can be initiated by a trigger generated by the computing system 132, in response to the detection of the RFID tag 125 by the RFID reader 126. Alternatively, the cameras 118A, 118B, 118C, 118D can continuously take images of their designated areas regardless of whether a piece of labware is witnessed in the area 114 A.

[0126] The system 100, using images recorded by cameras 118A, 118B, 118C, 118D and a tracking algorithm implemented in the computing system 132, can be configured to track the dish 102 anywhere on the worksurface 116, not just in a limited space, such as the heated area 114A. For example, after the dish 102 has been identified by the RFID reader 126, the RFID reader 126 transmits the (numeric / alphanumeric) identifier of the dish 102 to the computing system 132. The computing system 132 can be configured to associate the identifier with the imaged dish 102, maintaining the association with the identifier as the dish 102 moves across different areas of the workspace (including areas outside the reading range of the reader) and can be configured to provide for display, to the user interface 128A, 128B, the imaged dish 102 with the identifier, displayed as a label annotation. The example configuration 100B provides an advantage over certain previously existing workstations in which a dish would be invisible to the system when transferred to a non-heated area outside the reader range. The computing system 132 can generate a record specifying which dish was outside the heated area 114A and for how long. The computing system 132 can also cause an alert to be generated if the dish remains outside of the heated area 114A for greater than an allowed period of time. In some cases, the alert is generated immediately upon the dish being moved outside of the heated area 114A. This can help to ensure that biological material within the dish is maintained at an appropriate temperature.

[0127] FIG. 1C illustrates another configuration 100C of a portion of the system 100 for tracking, using the camera 118C, dishes deposited in and extracted from the incubator 120.The camera 118C is oriented towards (above) the incubator 120. The incubator 120 includes multiple compartments for storing dishes 102 with biological material between operations at a desired temperature and gas concentration. The camera 118C can be configured to monitor dishes 102 transferred relative to each compartment of the incubator 120. For example, the system 100 can be configured to track a dish 102 moved into and out of a compartment of the incubator 120. The tracking, using the camera 118C, of the dishes 102 deposited in or removed from any compartment of the incubator 120 can be compared to a standard protocol of an ART process. An alert may be generated if the standard protocol is breached. For example, an alert may be generated if the dish 102 is detected to be outside the incubator 120 for greater than an allowed period of time.

[0128] While the example configuration 100C has been described with respect to the incubator 120 of the workstation 110, it may also be applicable to the auxiliary medical system 108 described with reference to FIG. 1A. For example, the camera 118D of the auxiliary medical system 108 (or an additional camera associated with the auxiliary medical system 108 and not shown in FIG. 1 A) can be used to track dishes being move into and out of the compartments of the incubator 121 in much the same way as described above.

[0129] While the labware discussed above has been described as utilizing RFID tags to allow the system to identify the labware and connect that labware with persons associated with the ART procedure, other types of machine-readable identifiers can be used on the labware to identify the labware and connect that labware with persons associated with the ART procedure. Other examples of machine-readable identifiers include barcodes, 2D barcodes, or QR codes. In other cases, markings on the labware can be analyzed in images taken by the cameras for identifying the labware and connecting that labware with persons associated with the ART procedure.

[0130] FIG. 2 illustrates an example flow chart of a standard protocol of an ART process 200, during which one or more labware products (e.g., dishes 102, pipettes 103, etc.) can be tracked. The example flow chart of the ART process 200 is described with reference to the system 100 described above.

[0131] Initially, the process 200 begins at a first stage I 204A where sperm from a semen sample 206A from patient A, initially stored in a tube 102A, is transferred to a dish 102B to fertilize a single or multiple eggs from patient B, initially stored in a dish 102C. A portion of the fertilization process (e.g., the transfer of the biological material between dishes) can be performed in a designated area 214A of the ART system 100 (e.g., under the microscope 112 in the area 114A shown in FIG. 1 A). The dish 102B with the fertilized eggs can then bestored in an incubator (e.g., the incubator 120 shown in FIG. 1 A) to promote development of embryos 206C. Any movement of the dish 102B out of the incubator can be tracked using the tracking system discussed above with regard to FIG. 1 A.

[0132] At a second stage II 204B, embryos 206C are transferred to a culture dish 102D where multiple embryos 206C reside in a single drop 224 A, 224B. In the illustrated example, the culture dish 102D includes two drops 224A, 224B each containing multiple embryos 206C. The number of embryos can vary by subject. The second stage 204B of the ART cycle includes storing the culture dish 102D including the embryos 206C in an incubator (e.g., the incubator 120 that can be monitored as described with reference to FIG. 1 C) located in a particular area 214B of the ART system that includes cameras configured to track the culture dish 102D. The tracking can include a camera tracked transfer of the embryos from the incubator 120 to a holding dish 102E.

[0133] At a third stage III 204C, in response to determining that the embryos 206C developed to a desired maturity, the embryos 206C are transferred from the culture dish 102D to the holding dish 102E, where each embryo 216A, 216B, 216C is disposed in a drop 234 and is assigned an identity (e.g., numbers, letters, or combination thereof) in the dish 102E. The example system 100 is configured to identify and track the dish 102E using an identifier associated with the dish 102E. The computing system 132 tracks, using a camera (e.g., the camera 118A, 118B, 118C, 118D) the location of the dish 102E as it moves from a first area 214B to a second area 214C and, further to a third area 214D.

[0134] With continued reference to FIG. 4, a fourth stage IV of the ART process 200 includes a transfer of an embryo (e.g., embryo 216A) from the holding dish 102E to a biopsy dish 102F (e.g., to remove a few cells from the embryo that will be used for genetic testing). In some cases, the transfer of the embryo from the holding dish 102E to the biopsy dish 102F occurs at the workstation 110. The cameras 118A, 118B, 118C, 118D of the system 100 can be used to track the dishes 102E, 102F throughout this process to ensure that the embryo is transferred to the correct biopsy dish (i.e., a biopsy dish that is associated with the same persons as the holding dish).

[0135] Afterwards, the biopsy dish 102F is taken away from the worksurface to take a biopsy from the embryo 216A. Typically, the biopsy is carried out using the micromanipulator 119 at the auxiliary medical system 108. The cameras 118A, 118B, 118C, 118D of the system 100 can be used to track the biopsy dish 102F from the workstation 110 to the auxiliary medical system 108. After the biopsy process, the biopsy dish 102F is typically returned to the area 214D (e.g., an area of the workstation 110) with the dish 102Fholding both the embryo 216A and a biopsy of the embryo 216A. The cameras 118A, 118B, 118C, 118D of the system 100 can be used to track the biopsy dish 102F from the auxiliary medical system 108 back to the workstation 110. The system 100 can alert the user at this point if labware associated with any other persons are located at the workstation 110.

[0136] At a fifth stage V 204E, shown in FIG. 2, the embryo 216A, after having the biopsy taken from it, is transferred back to the holding dish 102E, and the holding dish 102 is returned to the incubator 120. Again, the tracking of the dishes 102E, 102F by the system 100 can help to ensure that the dishes 102E, 102F are associated with the same persons.

[0137] The embryologist can return to the biopsy dish 102D — now containing a single drop 434 with only the biopsy of the embryo 206CA — to remove the biopsy from the biopsy dish 102D and transfer the biopsy to a wash dish 102G to, for example, prepare the biopsy for genetic testing, as described below.

[0138] At a sixth stage VI 204F, the biopsy from the biopsy dish 102D is transferred to a wash dish 102G having three separate washing drops 234A, 234B, and 234C that the embryologist uses to wash the biopsy.

[0139] At a seventh stage VII 204G of the ART process 200, the washed biopsy 254 is transferred to a PCR tube 258 having a unique identifier 262 (e.g., barcode, 2D barcode, QR code, numbers, letters, or a combination thereof attached to a label or a tag).

[0140] At an eighth stage VIII 204H, the PCR tube 458 containing the biopsy is then sent to genetic testing. If a PCR tube 458 has been identified but the identifier 462 cannot be seen, the computing system prompts the embryologist to rotate the tube 458 until the identifier 462 can be seen. The unique identifier 462 can be a pre-printed 2D barcode that is imaged by the camera and processed by the computing system. The computing system records the transfer of the biopsy from the wash dish 102E to the PCR tube 458 and records the location of the biopsy 454 with the unique identifier 462 of the PCR tube 458.

[0141] At a ninth stage IX 2041, the embryo 216A from the holding dish 102E is transferred to a pre-vitrification dish 102H where the embryo will be placed into a series of drops 234D-234I to prepare the embryo for vitrification in a cry opreservation device. The cry opreservation device (not shown in FIG. 1 A) may be located in the vicinity of the workstation 110 and may be imaged by the cameras 118A, 118B, 118C, 118D or an additional camera that is trained on the cryopreservation device.

[0142] At a tenth stage X 204J, the embryo 216A is transferred from the pre-vitrification dish 102H to a vitrification tube 102F, and the vitrification tube 102F is placed in the vitrification device for storage.

[0143] Throughout the various stages of the above-described ART process, the cameras 118A, 118B, 118C, 118D (and possibly additional cameras) of the system 100 are used to track the various different labware products (i.e., dishes, tubes, pipettes, etc.). In much the same way that the RFID reader 126 was described above as reading the RFID tag 125 of the dish 102 to connect the dish 102 to particular persons associated with the ART procedure, the various labware (e.g., dishes, tubes, pipettes, etc.) used to carry out the ART process illustrated in FIG. 2 can include RFID tags, barcodes, or other unique identifiers that are used to identify the labware and connect the labware to persons associated with the ART procedure such that the computing system 132 can utilize the tracking data of the cameras, along with the personal data provided by the RFID tag, to alert the embryologist to any potential use of one person’s labware with another person’s lab ware or other breaches of standard operating protocol.

[0144] The ART process 200 of FIG. 2 is an example process that can include more or fewer stages. For example, the ART process 200 was described as tracking multiple dishes 102A-102H associated with a single embryo 246 and the biopsy taken from the embryo 246. However, in some examples, the imaging system can be configured to track multiple dishes 102A-102H associated with multiple embryos processed at a time, relative to a prestored protocol.

[0145] FIG. 3 illustrates a perspective view of another example configuration of an ART system 300 including a workstation 310 for tracking dishes during an ART process. In the configuration 300 illustrated in FIG. 3, the worksurface 316 of the workstation 310 is the same as the worksurface 116 of the workstation 110 described above and thus will not be described in detail. The workstation 310 includes a single camera 318 for tracking the dishes 102 as they are transferred between different areas of the worksurface 316. The single camera 318 can be an adjustable camera that is attached to a hood 334 of the workstation 310. The camera 318 can be coupled to an adjustable support 302. The adjustable support 302 can be configured to automatically adjust a height and an angle of the camera 318 to optimize continuous unobstructed tracking of the dish 102 on the worksurface 316, as the dish 102 is moved between different areas, and to deliver the images to the computing system 132 and for display on a user interface of the system 300. For example, the adjustable support 302 can include a retractable bar to adjust a distance between the camera and the dish 102. Further, the adjustable support 302 can include a rotatable bar to adjust an orientation of the camera to enable tracking of the dish 102 throughout multiple areas of the worksurface 316. In some cases, the height and orientation angle of the camera 318 can be adjusted to view an upwardfacing label of a dish (as illustrated in FIGS. 5A and 5B below) at any location on the worksurface 316.

[0146] FIG. 4 depicts a flow chart representing an example process 400 for tracking labware during an ART process, as described with reference to FIG. 2. The process 400 is performed by the computing system of the system 100 described with reference to FIG. 1 or a modified version thereof.

[0147] At 402, data associated with a machine-readable tag attached to a labware product before the labware product is used is received. This machine-readable tag can include an identifier that can be automatically read and associated with a respective labware product (e.g., dish), the association being maintained throughout the ART procedure to be performed. The machine-readable tag (e.g., tag with a barcode, a tag with a QR code, RFID tag) can be an upward facing visual identifier printed on the labware product, etched on the labware product, or attached to a portion (e.g., lid or base) of the labware product (e.g., a dish) that can be tracked in multiple areas of a medical facility, as described with reference to FIG. 1.

[0148] At 404, a labware product (e.g., an empty dish or a dish carrying the biological material associated with the persons) with an attached RFID tag is detected, using a reader (e.g., an RFID reader), in a first area of a workspace. The detected labware product is automatically identified and automatically classified using a mapping to the received data associated with a machine-readable tag attached to the labware product. For example, a tag (e.g., RFID tag) of the labware product including a biological material can be detected, by a tag reader (e.g., RFID reader), upon entering an area (e.g., near a microscope) of the workspace where an initial stage of the ART process (e.g., stage II of the ART process described with reference to FIG. 2) is being performed.

[0149] At 406, a tracking of the labware product is initiated. For example, after the identification of the labware product using the reader, images recorded by cameras can be used to continuously track the identified labware product as it is transferred between multiple areas of the ART lab (even outside the detection range of the reader). The tracking can be setup according to one or more tracking rules associated with the ART process described with reference to FIG. 2. Customizing the dish tracking can include a definition of a frequency (e.g., 10-20 Hz) of image recording by the cameras, selection of data processing (e.g., selection of image recognition algorithms) for event detection, definition of events (and event types), one or more conditions for storing acquired data and other parameters that optimize data acquisition and processing for labware product tracking. 1

[0150] At 408, images of the labware product are received, by a computing system, from one or more cameras (e.g., cameras 118 described with reference to FIGS. 1 and 3). The images can include high resolution images of the lab ware product. In some implementations, the images can include metadata including time stamps indicative of a recording time of each image that enables a chronological ordering of the images.

[0151] At 410, the images of the labware product are processed, by the computing system. The image processing can include processing of video data streams from cameras using clustering algorithms and machine vision algorithms (e.g., combination of continuous video frame differencing and background subtraction, video stream object movement noise filtering with labware location estimation via deep neural networks, video stream object movement noise filtering with density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture-based background / foreground segmentation for video stream object drop detection in front of other moving objects, Farneback optical flow for video stream movement speed segregation in special cases, image color detection, image segmentation, pixel clustering, pixel thresholding, morphology operations using erode and dilate and / or find contour, video frame processing using cascade filters, windowing, skipping frames, and down sampling, image enhancement using Gaussian blur, kernelling, Laplacian filter, change of color space, image feature detection using Hough Transform, measurement using Euclidean distance and mahalanobis distance, stereovision, histogram, and watershed). The processing can include applying a region of interest detection algorithm to identify, reconstruct, and extract a region of interest (e.g., the dish). In response to determining that an image includes an incomplete (obstructed) view of the region of interest (e.g., the dish), two or more images can be combined to reconstruct the region of interest (e.g., the dish) by recreating a complete view of the region of interest (e.g., the dish). The image processing can be adjusted based on the stage of the ART process being performed. For example, images of the labware product, received by the computing system from a selected camera associated with a particular area of the workstation can be processed during a particular stage of the ART process (e.g., the ART process 200 illustrated in FIG. 2).

[0152] At 412, a transfer of the labware product from a first to a second location is determined by the computing system. The location of the labware product can be within any area of the ART lab as described with reference to FIGS. 1 A and 2. The location identification can be performed at various stages of the example ART process.

[0153] At 414, the detected transfer of the labware product is correlated, by the computing system, to a standard protocol (e.g., the standard protocol described with referenceto FIG. 2) of the ART process that can be obtained (actively retrieved) from a database. The computing system can retrieve rules associated with transferring the identified labware product from the first location (e.g., an area within a range of a reader) to the second location (e.g., a second area outside the range of the reader), such as a maximum duration of holding the identified labware product containing a biological material at the detected locations and / or a temperature range appropriate for holding the identified labware product and biological material at the detected locations relative to temperature values measured by temperature sensors measuring temperature at the respective locations. The labware tracking data including the detected movement of labware products can be correlated by the computing system with an operating protocol retrieved from a data store to generate detailed correlation results indicative of the multi-parameter analysis including allowed times for each area and recorded sensor parameters (temperature, humidity, air flow, air composition, etc.) during the detection of the labware product within a respective area. For example, before the labware product is delivered to the second location, the computing system determines whether the second location correlates with SOP stored in a database of the memory. The correlation results indicate compliance or deviation of the detected event from the SOP and associated medical procedure rules. The data correlation can include a comparison of deviations to respective deviation thresholds to determine a risk of a potential detrimental effect of the labware product transfer (e.g., on the biological sample held within the moved lab ware product).

[0154] At 416, feedback data is generated, by the computing system, for display. The feedback data is generated based on the correlation results to indicate compliance or deviation of the detected labware product transfer between areas of the lab according to the SOP and associated medical procedure rules. In some implementations, the detection of an incorrect event can trigger the display of an alert on the user interface, to sound an acoustic alert on a speaker, and / or to activate one or more light sources to highlight a correct location to guide the medical staff (embryologist) to transfer the labware product to the correct area of the ART lab.

[0155] At 418, the labware product transfer data associated with the ART process is stored by the computing system. The data storage can be configured to minimize data storage resources such that a selected (single key) reconstructed image of the region of interest is recorded for each location where the labware product associated with the identifier was detected.

[0156] As noted above, while the dish 102 illustrated in FIG. 1 A has been described as including the RFID tag 125 that can be read by the RFID reader 126 of the system 100, other types of tags, such as barcodes, 2D barcodes, QR codes, etc., can alternatively or additionally be used on the labware described herein to allow that labware to be connected to persons involved with the ART process. FIGS. 5A and 5B are views of example holding dishes that can be used to hold biological material (e.g., embryos 216) and can be tracked during an ART process. The example holding dish 502 shown in FIG. 5A includes an upward facing clear label 124 that is attached to a bottom surface of the dish 502. The label 124 enables a reader or a camera to identify the dish, the identification being transmitted by the reader to the computing device and maintained, by the computing device, while tracking the dish within the workspace of a medical facility (e.g., an ART lab). While holding dish 502 illustrated in FIG. 5A includes an upward facing label 124 that is adhered to the bottom surface of the dish 502, other types of upward facing visual identifiers can alternatively be used. For example, the upward facing visual identifier can alternatively be laser etched into the bottom surface of the dish 502 or printed onto the bottom surface of the dish 502. The upward facing visual identifiers can be configured to allow a clear view of wells that are formed in the dish 502 for holding biological material.

[0157] The holding dish 502 can also include an optically transparent lid 502 covering for covering the wells. The lid can also include a label attached to the lid for tracking purposes. The label can be laser etched into the lid, printed onto a surface of the lid, or adhered to a surface of the lid using a clear label.

[0158] As shown in FIG. 5B, another holding dish 552 includes the RFID tag 125, the label 124, and a marking 538. While the marking 538 on the dish 552 is illustrated in the form of a line, other visual characteristics, such as dots, symbols, letters, numbers, and shapes created by printing, etching, molding, labeling, or otherwise marking the dish. The RFID tag 125 can be read by an RFID reader, as described above, to match the dish 552 to a particular persons involved with the ART process. The label 124 can alternatively or additionally be read by a suitable reader (e.g., a barcode reader) or a camera for matching the dish 552 to the particular persons involved with the ART process and can be displayed by the user interface as a label. Alternatively, or additionally, the marking 538 can be viewed by one or more of the cameras of the system for matching the dish 552 to the particular persons involved with the ART process. In addition to using the RFID tag 125, the label 124, and the marking 538 to match particular persons to the dish 552, they can be used by the cameras during the dish tracking process to distinguish the dish 552 from other dishes that may be within view of thecameras. In certain cases, for example, the cameras are capable of zooming in to view the various tags, labels, and markings of the dish 552 to assist in distinguishing the dish 552 from other lab ware products.

[0159] FIG. 6, depicts a flow chart representing a process 600 for tracking a dish (optionally holding a biological material) during an ART process. The process 600 may be performed by the computing system 132 described with reference to FIG. 1, using dishes including labels, as discussed with reference to FIGS. 5A and 5B. The process 600 can be executed by a server system, such as the server system 824 described with reference to FIG. 8 or a different computing device with processing capabilities.

[0160] At 602, images of a labware product (e.g., the dish 552) (optionally holding biological material) are received, by a computing system (e.g., the computing system 132), from one or more cameras (e.g., cameras described with reference to FIGS. 1-3). The images can include high resolution images of the labware product.

[0161] At 604, the images of the labware product are processed, by the computing system, to detect a position of the labware product and, optionally, a label of the labware product including the labware product identifier. The image processing can include processing algorithms as described with reference to FIGS. 2, 4 and 5.

[0162] At 606, the labware product is monitored while being at a location within the ART lab. The labware product monitoring can be performed according to rules corresponding to various stages of the example ART process, described with reference to FIG. 2, defining optimal parameter ranges for keeping the labware product within a particular area of the ART lab. The optimal parameter ranges include a maximum duration of holding the identified labware product (including a biological material) at the detected location and / or a temperature range appropriate for holding the identified labware product (including the biological material) at the detected location.

[0163] At 608, an event is detected, by processing images acquired by one or more cameras. The detected event can include a transfer of the labware product from the detected (first) location to a new (second) location and / or detection of an additional labware product (e.g., another dish, pipette, tube, etc.) near the tracked labware product. The event detection can include continuous tracking of the identified labware product to enable differentiation from the additional labware product. The continuous tracking can include a derivation of labware product displacement trajectory relative to a timeline, such that each location of the tracked labware product is determined and stored with a corresponding time while the labware product is within the medical facility.

[0164] At 610, the detected event data is correlated by the computing system with a SOP, retrieved from a data store, to generate correlation results that are transmitted and stored using the labware product identifier.

[0165] At 612, feedback data is generated, by the computing system for display. The feedback data is generated based on the correlation results to indicate compliance or deviation of the detected event from the SOP and associated medical procedure rules. The feedback data provides real-time feedback to an embryologist that the labware product being used is appropriate for the ART procedure being carried out.

[0166] At 614, the data associated with the event is stored by the computing system. The data storage can be configured to minimize data storage resources such that a selected (single key) reconstructed image of the region of interest is recorded for each location where the identified labware product was detected.

[0167] While the example processes 400 and 600 were described for tracking labware within ART facilities, the example processes 400 and 600 can also be applied for tracking labware within other types of medical facilities, such as a medical facility 702, described with reference to FIG. 7, and using a tracking system 802, described with reference to FIG. 8.

[0168] FIG. 7 illustrates an example system 700 for tracking a labware product (e.g., dish 552 illustrated in FIG. 5B) (optionally containing biological material) being transferred between multiple areas 714A, 714B of the medical facility 702, in accordance with some example implementations. The first area 714A and the second area 714B can be within the same room of the medical facility 702 or in separate rooms of the medical facility 702. In some cases, the first area 714A and the second area 714B are both within a single workstation. For example, the first area 714A can correspond to the area 114A in FIG. 1 A, and the second area 714B can correspond to the area 114B in FIG. 1 A. In other cases, the first area 714A can be within the workstation and the second area 714B can be outside the workstation. For example, the first area 714A can correspond to the area 114A in FIG. 1 A, and the second area 714B can correspond to the area 114C in FIG. 1 A. In some implementations, the first area 714A can be at a first temperature (e.g., the first area 714A can be heated) and the second area 714B can be at a second temperature (e.g., the second area 714B can be unheated). The first area 714A and the second area 714B can include one or more components of the example system 100, such as the cameras 118A, 118B, 118C, one or more microscopes 112, one or more sensors 122 A, 122B, 122C, one or more RFID readers 126, one or more computing systems 132A, 132B, one or more user interfaces 128, and a server system 704, described in further details with reference to FIG. 8.

[0169] The dish 552 can be continuously tracked, by the example system 700, within the first area 714A, within the second area 714B, and while being transferred from the first area 714A to the second area 714B, using label detection by processing images acquired by from cameras 118A, 118B, 118C and / or using RFID detection, performed by the RFID readers 126. For example, the dish 552 can be recognized, by the RFID reader 126 in the first area 714A and can then be continuously tracked, by cameras 118A, 118B, 118C, as it is transferred to other locations within the first area 714A or outside the first area 714A, such as within the second area 714B, without requiring reidentification.

[0170] In another example, the cameras 118A, 118B, 118C can have high resolution for both identification and tracking of the dishes, anywhere on the worksurface through upward facing codes included in the label 124 attached to the dish 552. The codes, as discussed above, can be laser etched underneath the dish 552, printed on the dish 552, or adhered to an underside surface of the dish 552 using a clear label. If the resolution of the cameras 118A, 118B, 118C is not high enough for dish identification, at least one camera can have a zoom lens mounted on a gimble that could direct itself towards unidentified dishes and identify the dishes anywhere on the worksurface. In some examples, the identification by the cameras 118A, 118B, 118C can be extended to recognizing users (e.g., embryologists) through codes on ID cards, lanyards, wristbands, or other worn objects.

[0171] The system can be configured to detect and track potential breaches of medical procedure protocol by detecting operations associated with the tracked displacement of the dish 552 within any of the first and second areas 714A, 714B. Systematic collection, by the example system 700, of potential adverse events, within the medical facility 702, during actual medical procedures provides a spatially complete set of information. The collected information enables comparison and analytics that can be used to conduct assessments and prevent adverse events using real-time notifications and feedback, such as audio and / or visual alerts.

[0172] Still referring to FIG. 7, in some cases, the computing system can use an image recognition software (e.g., computer vision, machine vision, etc.) and / or machine learning to image and process properties of the dish 552 identified. The example computing system tracks and keeps records of the dish 552 as an embryologist moves the dish 552 between different locations within the same area 714A or between different areas 714A, 714B. The example computing system is configured to receive an image of the dish 552 having a visual characteristic and is configured to process the image using a tracking module (described withreference to FIG. 1). The example computing system can also record visual characteristics that can be used to identify that specific dish and distinguish it from other dishes.

[0173] The example computing systems 132A, 132B are communicatively coupled to the camera(s) 118A, 118B, 118C within (or focused on) a respective area 714A or 714B and receive the images from the respective camera(s). The computing system can process images to identify the dish 552. The one or more data storage devices memory 830 of the example computing system defines the tracking module and a data store 828 containing, for example, subject information, a plurality of visual characteristics, a plurality of types of biological material, and standard operating protocols (SOPs) for the ART process. After receiving the image from the camera, the example computing system processes the image and compares the image with information stored in the database. The computing system is communicatively coupled to the camera by a wired and / or wireless connection, such as via Bluetooth™, or radio communication (e.g., Wi-Fi). The computing system is configured to deliver real-time feedback in the form or prompts and / or alerts to the embryologist at each stage of the ART process. The real-time feedback is delivered through the user interface 128, within the respective area, and through an audible feedback device, which is coupled to the computing system 132B.

[0174] As another example, the images recorded by the cameras 118A, 118B, 118C including a portion of a respective area 714A or 714B (e.g., worksurface 114 A), can be augmented to include annotations indicative of an availability of virtual (remote) display control. The annotations can include markers of regions, such as arrows pointing to portions of the workstation 110. The virtual (remote) display control can include preset (predefined) control actions or gestures. The control actions or gestures can be defined to include a movement of a hand or a finger over a particular area or object in the workstation 110. For example, images displayed, by the user interface 128A, 128B, can include arrows pointing to portions of the workstation 110. The recording and recognition of a hand being moved over the physical area displayed as including an arrow can be trigger an action associated with the images recorded by one or more of the cameras 118A, 118B, 118C and displayed by the user interface 128A, 128B. The triggered action can include a modification of a display of the images. In some implementations, a particular hand gesture can be associated to a particular triggered action affecting the display. For example, a pinch gesture can be associated to a modification of a zoom level of the displayed image. In other cases, a hand gesture applied to a particular area or object in the workstation 110 can include scrolling through commandsaffecting displayed images and a modification of the displayed image based on a selected command. The hand gestures can be used as inputs for the witnessing system.

[0175] FIG. 8 illustrates a diagram of an example system 800 for tracking labware (e.g., dish 552) (optionally containing biological material) within a medical facility. The example system 800, as illustrated in FIG. 8, includes a combination of devices coupled with each other for continuous tracking the labware throughout multiple areas of a medical facility, including (but not limited to an ART lab, as described with reference to FIG. 1). In particular, the example system 800 includes tracking system 802, one or more cameras 118A, 118B, 118C, one or more microscopes 112, sensors 122 A, 122B, 122C, one or more RFID readers 126, one or more computing systems 132, a user interface 128, a computing device 820, a network 822, and a server system 824.

[0176] The tracking system 802 can include an assembly configured for one or more operations of processes, as described in detail with reference to FIGS. 4 and 6. The tracking system 802 can include or be communicatively coupled to one or more components of the example system 800, such as the cameras 118A, 118B, 118C, the one or more microscopes 112, the sensors 122 A, 122B, 122C, the one or more RFID readers 126, the one or more computer boards 816, the computing system 132, and the computing device 132.

[0177] The cameras 118A, 118B, 118C can be configured for real-time locating of dish 552 within different areas of a medical facility. For example, the cameras 118A, 118B, 118C can include digital (ultra-high definition) cameras. The digital cameras can include zoom, angle, and / or position adjustable cameras. The digital cameras can include cameras with optical character recognition (OCR). The cameras 118A, 118B, 118C can be adapted to read bar codes and / or QR codes in some cases included in the label of a dish 102. The cameras 118A, 118B, 118C can be integrated in and / or coupled to one or more tracking system 802. The cameras 118A, 118B, 118C can be configured for continuous (e.g., video stream) recording or selective image acquisition in response to set triggers (e.g., signals received from the one or more sensors 122A, 122B, 122C, and the RFID reader 126).

[0178] As noted above, in some cases, the cameras 118A, 118B, 118C are configured to read bar codes of consumables (e.g., dishes, pipettes, etc.) or the packaging of such consumables. This can enable the system to monitor the inventory of consumables and otherwise record the particular consumables that are used in a procedure. Consumables used as part of the ART process typically have date and lot codes that get recorded during a procedure. Certain embodiments of the systems described herein allow the barcodes to be automatically read and associated with each dish, thereby reducing a step that otherwisemight need to be performed by the technician or clinician (e.g., by manually scanning the barcode on the packaging of the consumable). For example, the cameras 118A, 118B, 118C may automatically read the barcodes on any consumables or consumable packaging that is moved into the viewing area of those cameras.

[0179] The one or more microscopes 112 can be configured for imaging the biological material used during ART processes. The one or more microscopes 112 can include any computer-controlled microscope that is equipped for digital image acquisition, storage and analysis of biological material. In some implementations, the one or more microscopes 112 can include stereo zoom microscopes, and inverted microscopes that fit inside the tracking system 802.

[0180] The sensors 122 A, 122B, 122C can include sensors configured to detect one or more parameters of a biological material and / or of an environment, where a dish 102 including biological material can be stored. For example, the sensors 122A, 122B, 122C can include environmental sensor(s). The sensor 122 A can sense temperature of an area of the medical facility, where biological materials can be stored for any period of time during the ART process. The sensor 122B can sense a load (weight of a receptacle) on a surface (e.g., worksurface of a workstation 110) assigned for operations executed during the ART process. The sensor 122C can be a gas sensor configured to sense pressure, humidity, static pressure, dynamic pressure, and presence of one or more particles with particular dimensions (e.g., between 0.5 pm and 1.0pm). Additional sensors are also possible, including for example, air flow sensors and air particle count sensors.

[0181] The one or more RFID readers 126 can be configured to interrogate one or more RFID tags attached to dishes storing biological material that are within a threshold distance from the RFID readers 126. For example, RFID readers 126 can sequentially interrogate each position in an area, determining which positions have a wireless transponder tagged dishes, and identity of each dish at each position that has a RFID tag. The RFID readers 126 can include a two-dimensional array of antennas arranged to be in registration with respective positions of one or more tagged dishes.

[0182] While the system 800 of FIG. 8 has been described and illustrated as having one or more RFID readers 126, it should be understood from the discussion above that the system 800 can alternatively or additionally include any of various other types of readers that are capable of reading machine-readable identifiers on labware. Examples of other types of readers include barcode readers as well as image processing systems that are capable of recognizing identifiers on labware.

[0183] The computing system 132 can include a special purpose computing device configured for tracking dish including biological material during an ART process. The special-purpose computing device is hard-wired to perform the techniques or includes digital electronic devices such as one or more application- specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques or can include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine custom hardwired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. In various implementations, the special-purpose computing devices are desktop computer systems, portable computer systems, handheld devices, network devices or any other device that incorporates hard-wired and / or program logic to implement the process described in detail with reference to FIG. 8.

[0184] The computing system 132 can include a localization module 826, a data store 828, a memory 830, a processor 832, a messaging system 834, and a controller 836. The localization module 826 determines the position of dishes by using data from the sensors 122A, 122B, 122C and data from the data store 828 (e.g., tracking data) to calculate a position of the dish holding the biological material. The data store 828 provides mass storage for the computing system 132. In some implementations, the data store 828 is a computer- readable medium. In some implementations, the data store 828 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device. In an implementation, data, stored by the data store 828 and used by the localization module 826 includes high-precision maps of the medical facility, geometric properties of work surfaces within tracking system 802, maps describing RFID reader ranges, maps describing physical properties (such as optimal operation temperatures, humidity levels, gas concentrations, or combinations of them), and maps describing the spatial locations of sensors relative to workspaces or other sensor signals of various types.

[0185] The computing system also includes a main memory 830, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to the bus for storing information and instructions to be executed by the processor 832. In one implementation, the main memory is used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor. The example computing system can store images and image data generated by the camera in the memory 830. The memory 830 can be, for example, a physical data storage device or a logical data storage areaof the computing system. Additionally, the example computing system stores data that defines the locations of drops of media in each dish in the memory 830.

[0186] In an implementation, the computing system further includes a read only memory (ROM) or other static storage device coupled to the bus for storing static information and instructions for the processor. The hardware processor 832 is, for example, a general-purpose microprocessor. A storage device, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional cross point memory 830 is provided and coupled to the messaging system 834, which controls transmission of data for storing information for tracking dishes. The controller 836 receives data representing a target output for tracking an identified receptacle. The target output typically includes an information type characterizing an event associated with the dish (e.g., transfer of biological material during ART process from one dish to another). In an implementation, the controller 836 receives feedback that a camera is adjusted for acquiring updated images of labware containing the biological material. For example, if the localization module identifies an obstruction of a view of the labware containing the biological material, the image acquisition of the camera is adjusted for the desired output (unobstructed view of the labware containing the biological material). In an implementation, any measured output is provided to the controller 836 so that image acquisition adjustments are performed, e.g., based on the differential between expected labware border detection and determined labware border detection in one or more images. In some implementations, the computing system is implemented in or as a part of other computing systems of a medical facility to acquire and / or to send medical imaging data for display on one or more user interfaces 128 and computing devices 820, that collect and logically assemble data to track medical procedures (e.g., ART processes) and prevent medical staff from performing incorrect operations and / or alarm on adverse contributors in real-time while a medical procedure is ongoing from the computing system (e.g., an admin console), which can be secured by an internal network 822.

[0187] The user interface 128 can include a display, such as cathode ray tube (CRT), a liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user. The user interface 128 can include alphanumeric and other keys coupled to a bus for communicating information and command selections to the processor. Another type of user input device is a cursor controller, such as a mouse, a trackball, a touch-enabled display, or cursor direction keys for communicating direction information and command selections to the processor and for controlling cursor movement on the display. The user interface 128 canenable receipt of user input and display information associated to dish tracking throughout the medical facility, providing information and alerts related to operations involving the tracked receptacle. Specifically, the user interface 128 includes a speaker 840 that is configured to play a sound to deliver a “correct” message and a different sound to deliver an “incorrect” message when prompted by the computing system. Additionally, the user interface 128 is configured to temporarily flash a message or color on a screen of the user interface 128 to deliver “correct” and “incorrect” messages when prompted by the computing system. For example, before the embryologist transfers the dish to a different location, the computing system signals to the user interface 128 to display a first color or text on the screen to deliver the “correct” message or display a second color or text on the screen to deliver the “incorrect” message.

[0188] The computing device 820 can be a processor-based device including, for example, a computing system such as the described computing system. The computing device 820 can be a smartphone, a tablet computer, a wearable apparatus, a virtual assistant, an Internet-of-Things (loT) appliance, and / or the like. The computing device 820 can be coupled to or communicate with the tracking system 802 via the network 822. For example, the computing systems 132 and / or computing devices 820 (services consumers) are connected to the server system 704 through network links and network adapters.

[0189] The network 822 can be a wired network and / or a wireless network. In an implementation, the network 822 represents any combination of one or more local networks, wide area networks, or internetworks coupled using wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network 822, is transferred using any number of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc. Furthermore, in implementations where the network 822 represents a combination of multiple sub-networks, different network layer protocols are used at each of the underlying sub-networks. In some implementations, the network 822 represents one or more interconnected internetworks, such as the public Internet.

[0190] The server system 704 includes one or more server devices configured to provide a “cloud” computing environment for tracking dishes in medical facilities. Cloud computing services enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services). In typical cloud computing systems, one or more large cloud data centers house the machines used to deliver the services providedby the cloud. The server system 704, configured as a cloud computing environment, includes cloud data centers that are interconnected through the network 822. Data centers provide computing services to the computing system 132 and the computing device 820 connected to the server system 704 over the network 822. The server system 704 includes one or more data centers. In general, a data center, for example a cloud data center, refers to the physical arrangement of servers that make up a cloud, for example the cloud, or a particular portion of a cloud. For example, servers are physically arranged in the datacenter into rooms, groups, rows, and racks. The datacenter has one or more zones, which include one or more rooms of servers. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementation, servers in zones, rooms, racks, and / or rows are arranged into groups based on physical infrastructure requirements of the datacenter facility, which include power, energy, thermal, heat, and / or other requirements. In an implementation, the server nodes are similar to the described computing system 132. The data center has multiple computing systems distributed through many racks. The server system 704 includes data centers along with the network 822 and networking resources (for example, networking equipment, nodes, routers, switches, and networking cables) that interconnect the data centers and help facilitate the computing systems 132 access to remote (cloud) computing services.

[0191] Although an example processing system has been described in FIG. 8, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0192] The imaging systems described above with respect to the cameras 118A, 118B, 118C operate on a computing system and each include one or more cameras, a memory 830, and tracking module 826. However, in another example, the imaging system can have more or fewer components. In another example, the memory and or detection model of the imaging system can be integrated with one or more cameras 118A, 118B, 118C, the microscope 112, the user interface 128, or a connection to the server system 704 instead of the local computing system. In the example system 100 of FIG. 1A, the cameras 118A, 118B, 118C are mounted to the workstation 110. However, in other examples, one or more of the cameras 118A, 118B, 118C can be directly mounted on a wall or a ceiling of a room of the medical facility to enable imaging of any areas within the respective room. In other implementations, an imaging system can include additional multiple cameras set up through the lab space(attached to and unattached to workstations) to track multiple dishes. For example, a plurality of spaced apart cameras can be perpendicularly disposed relative to a horizontal work surface to image all dishes, for example, under a lab hood.

[0193] In some examples, the imaging systems can provide a digital, visual guidance to provide feedback during the ART process. In one example, a transparent LCD screen can be disposed under the receptacle, which could provide visual feedback and guidance to the embryologist while the embryologist is viewing the dish under the microscope. In another example, the example computing system can include a microscope with an integrated graphical overlay that provides feedback and guidance while viewing the dish through the microscope. Specifically, graphical overlay can incorporate augmented reality (AR) technology. For example, the microscopes 112 can incorporate AR by providing a transparent screen disposed between an embryologist’s eye and what is being read with the microscope. The AR technology can be coupled with the imaging systems to give visual commands to the embryologist (e.g., highlighting a drop on the examined dish to identify where the drop should be deposited, crossing out drops that already contain biological material, crossing out entire dishes to indicate the incorrect dish is under the microscope, etc.). In another example, the embryologist could use a microscope configured with a display screen instead of eyepieces. In this case, graphical information could be overlaid onto that display screen.

[0194] In some examples, the visual characteristics can include dish details, information added to the receptacle, information around the drops, and / or layout of different visual references relative to each other. For example, a dish can have an RFID tag on a bottom surface and is specifically placed adjacent to a first drop location. The drops on the dish can be identified by their relative locations to the RFID tag.

[0195] The tracking assemblies can be used to track dishes outside of the ART process. Further, while the imaging systems described herein are used in an ART process to track dishes in an ART lab, the imaging systems can be used to track different labware used in different processes. In such examples, different technicians can be working in the lab using different dishes that are simultaneously tracked by the tracking system to avoid and prevent a mix-up of dishes.

[0196] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to performparticular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0197] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0198] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0199] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0200] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0201] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0202] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0203] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0204] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0205] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.

[0206] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0207] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0208] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In someimplementations, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0209] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosure or of what can be claimed, but rather as descriptions of features that can be specific to particular examples of particular disclosures. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable sub combination. Moreover, although features can be described herein as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a sub combination or variation of a sub combination.

[0210] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the examples described herein should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single product or packaged into multiple products.

[0211] Particular examples of the subject matter have been described. Other examples are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

WHAT IS CLAIMED IS:

1. A system for tracking a labware product in an assisted reproductive technology (ART) process, the system comprising: one or more cameras; one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for tracking the labware product, the operations comprising: receiving, from the one or more cameras, images of one or more areas of a facility in which at least a portion of the ART process is carried out, processing the images of the one or more areas of the facility, determining, based on the processed images of the one or more areas of the facility, a transfer of the labware product from a first location within the facility to a second location within the facility, and outputting information associated with the transfer of the labware product.

2. The system of claim 1, wherein the one or more cameras comprises a first camera configured to generate a first set of images of the first location and a second camera configured to generate a second set of images of the second location.

3. The system of claim 1 or claim 2, wherein the one or more cameras comprise at least one of a wide-view camera or a camera with an auto adjustable zoom lens.

4. The system of any one of claims 1 through 3, comprising a workstation having a worksurface comprising the first and second locations.

5. The system of claim 4, wherein the first location is within a heated portion of the worksurface, and the second location is within an unheated portion of the worksurface.

6. The system of claim 4, wherein the first location is within a portion of the worksurface that is monitored by a witnessing system and the second location is outside the portion of the worksurface that is monitored by the witnessing system.

7. The system of claim 6, wherein the witnessing system comprises a radio frequency identifier (RFID) reader configured to read an RFID tag on the labware product when the labware product is within the area that is monitored by the witnessing system.

8. The system of claim 6, wherein the witnessing system comprises a barcode reader configured to read a barcode on the labware product when the labware product is within the area that is monitored by the witnessing system.

9. The system of claim 6, wherein the one or more cameras are attached to the workstation and are configured to be oriented at a plurality of angles relative to the worksurface.

10. The system of claim 6, wherein at least one of the one or more cameras comprise at least one camera positioned outside the workstation.

11. The system of any one of claims 1 through 7, wherein the labware product comprises an upward facing label that is readable by at least one of the one or more cameras, and processing the images of the one or more areas of the facility comprises processing the upward facing label to determine an identifier of the labware product.

12. The system of claim 11, wherein the upward facing label comprises a code indicative of the identifier of the labware product, wherein the code is a machine generated code.

13. The system of claim 11, wherein the machine generated code comprises a bar code or a two-dimensional matrix barcode.

14. The system of any one of claims 1 through 10, wherein the labware product comprises a RFID tag, and the system comprises a RFID reader configured to read the RFID tagof the labware product and to transmit data associated with the RFID tag to the one or more computers.

15. The system of claim 14, wherein the data associated with the RFID tag is associated with the labware product being tracked.

16. The system of any one of claims 1 through 10, wherein the labware product comprises a barcode, and the system comprises a barcode reader configured to read the barcode of the labware product and to transmit data associated with the barcode to the one or more computers.

17. The system of claim 16, wherein the data associated with the barcode is associated with the labware product being tracked.

18. The system of any one of the preceding claims, comprising a database storing a standard operating protocol, wherein the operations comprise: determining whether the transfer of the labware product from the first location to the second location is in compliance with the standard operating protocol stored in the database.

19. The system of claim 18, wherein outputting the information comprises generating an alert signaling non-compliance with the standard operating protocol.

20. The system of any one of the preceding claims, comprising a sensor configured to detect one or more environmental parameters at the first location or the second location, the one or more environmental parameters comprising a temperature, a gas concentration, or a humidity level.

21. The system of any one of the preceding claims, wherein the operations further comprise determining, based on the processed images of the one or more areas of the facility, an additional transfer of the labware product within the facility from the second location to a third location.

22. The system of claim 21, wherein the one or more cameras track the transfer of the labware product from the first location to the second location to the third location.

23. The system of claim 21 or claim 22, wherein the one or more cameras comprises a first camera configured to image the first location, a second camera configured to image the second location, and a third camera configured to image the third location.

24. The system of any one of the preceding claims, wherein the one or more cameras and the one or more computers are configured to record a cleaning process of at least one of the one or more areas of the facility.

25. The system of claim 24, wherein the one or more cameras and the one or more computers are configured to recognize and track wipes used to carry out the cleaning process.

26. The system of claim 25, wherein the one or more cameras and the one or more computers are configured to track movement of the wipes during the cleaning process to identity portions of the at least one of the one or more areas of the facility that were not cleaned.

27. The system of claim 26, wherein the portions of the at least one of the one or more areas of the facility that were not cleaned are displayed and marked on a user interface to assist a user to complete the cleaning process.

28. The system of any one of the preceding claims, wherein the one or more cameras and the one or more computers are configured to read a barcode on the labware product and associate the barcode with the labware product.

29. The system of claim 28, wherein the one or more cameras and the one or more computers are configured to automatically read the barcode when the labware product is moved into a viewing area of the one or more cameras.

30. The system of any one of the preceding claims, wherein the one or more cameras and the one or more computers are configured to read a barcode on packaging of the labware product and associate the barcode with the labware product.

31. The system of claim 30, wherein the one or more cameras and the one or more computers are configured to automatically read the barcode when the packaging of the labware product is moved into a viewing area of the one or more cameras.

32. The system of any of claims 28-31, wherein the one or more computers are configured to monitor an inventory of labware products.

33. The system of any of claims 28-32, wherein the one or more computers are configured to record the labware product as being used in the ART process being carried out.

Images