Device and method for training a sample characteristic evaluation algorithm in a diagnostic examination room system

Abstract

A method of updating training of an annotation generator of a diagnostic examination room system, the method comprising providing an imaging device in the diagnostic examination room system, wherein the imaging device is controllably movable within the diagnostic examination room system; capturing a first image within the diagnostic examination room system using the imaging device, wherein the first image is captured under at least one imaging condition; performing an annotation of the first image using the annotation generator to generate a first annotated image; and updating the training of the annotation generator using the first annotated image. Other methods and systems are disclosed.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 368,456, entitled "DEVICES AND METHODS FOR TRAINING SAMPLE CHARACTERIZATION ALGORITHMS IN DIAGNOSTIC LABORATORY SYSTEMS," filed on July 14, 2022, the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] Embodiments of the present disclosure relate to devices and methods for training sample characterization algorithms in a diagnostic laboratory system.

Background Art

[0003] Diagnostic laboratory systems perform clinical chemistry tests or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. Samples can be received in and / or transported in sample containers throughout the laboratory system. Many laboratory systems process large numbers of sample containers and the samples contained within the sample containers.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Some inspection room systems facilitate sample processing and sample container identification using machine vision and machine learning. This can be based on the characterization and / or classification of the sample containers. For example, a vision-based machine learning model (e.g., an artificial intelligence (AI) model) is adapted to provide a fast and non-invasive method for sample container identification and characterization. However, the training costs for supporting a new type of sample container or new imaging conditions using a machine learning model can be high. This is because a large amount of training data is required to retrain or adapt the machine learning model to characterize a new type of sample container or to adapt the machine learning model to function under new imaging conditions. Therefore, there is a need for inspection room systems and methods that improve the training of machine vision systems in inspection room systems.

Means for Solving the Problems

[0005] According to a first aspect, a method for updating the training of a sample characterization algorithm of a diagnostic inspection room system is provided. The method includes providing an imaging device to the diagnostic inspection room system, where the imaging device is controllably movable within the diagnostic inspection room system; capturing a first image within the diagnostic inspection room system using the imaging device, where the first image is captured under certain imaging conditions; performing an annotation of the first image using an annotation generator of the diagnostic inspection room system to generate a first annotated image; and updating the training of the annotation generator using the first annotated image.

[0006] In another aspect, a method for training a sample characteristic evaluation algorithm of a diagnostic laboratory system is provided. The method includes providing an imaging device to the diagnostic laboratory system, where the imaging device is controllably movable within the diagnostic laboratory system; capturing a first image of a sample container using the imaging device, where the first image is captured under a first imaging condition; performing annotation of the first image to generate a first annotated image; changing the first imaging condition to a second imaging condition; capturing a second image of the sample container under the second imaging condition using the imaging device; performing annotation of the second image to generate a second annotated image; training an annotation generator of the diagnostic laboratory system using at least the first annotated image and the second annotated image; changing the second imaging condition to a third imaging condition; capturing a third image of the sample container under the third imaging condition using the imaging device; performing annotation of the third image using the annotation generator to generate a third annotated image; and further training the annotation generator using at least the third annotated image.

[0007] In a further aspect, there is provided a diagnostic examination room system comprising: (1) an imaging device that is controllably movable within the examination room system and is configured to capture images within the examination room system under various imaging conditions; (2) a processor coupled to the imaging device; and (3) a memory coupled to the processor, where the memory includes an annotation generator trained to annotate images captured by the imaging device, and the processor, when executed by the processor, causes the processor to: (a) receive first image data of a first image captured by the imaging device using at least one imaging condition; (b) cause the annotation generator to execute an annotation of the first image to generate a first annotated image; and (c) update the training of the annotation generator using the first annotated image.

[0008] Further aspects, configurations, and advantages of the present disclosure may become readily apparent from the following description and illustrations of several exemplary embodiments, including the best mode contemplated for carrying out the present disclosure. The present disclosure is also capable of other different embodiments, and some details thereof may be changed in various respects without departing from the scope of the present disclosure.

[0009] The drawings described below are provided for illustrative purposes and are not necessarily drawn to scale. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive. The drawings are not intended to limit the scope of the present disclosure in any way.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12-1

Figure 12-2

Figure 13-1

Figure 13-2

DETAILED DESCRIPTION OF THE INVENTION

[0011] As described above, the diagnostic laboratory system performs clinical chemistry tests and / or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. The sample is collected in a sample container and then delivered to the diagnostic laboratory system. The sample container is then loaded into the sample handler of the laboratory system. Next, the sample container is transferred by a robot to a sample carrier, which transports the sample container to the instruments and components of the diagnostic laboratory system where the sample is processed and analyzed.

[0012] The diagnostic laboratory system can capture images of the sample container and / or the contents of the sample container (e.g., the biological sample) using a vision system. The captured images are then used to identify the sample container and / or the contents of the sample container. For example, the diagnostic laboratory system can include a vision - based AI model configured to provide a fast and non - invasive method for sample container characterization or classification. The AI model can be trained to annotate images of various types of sample containers, as well as deformations of the tube portion and / or cap of the sample container. The annotated images can then be used for the purpose of sample identification.

[0013] When a new type of sample container is introduced into a diagnostic laboratory system, the AI model used must be updated or "trained" to be able to annotate the new sample container type. Retraining the AI model in a conventional diagnostic laboratory system is costly and time-consuming because multiple different types of sample containers need to be imaged and manually annotated to retrain the AI model. The AI models used in machine vision systems are typically trained using images of samples and / or sample containers captured in an ideal environment such as a studio setting with ideal imaging conditions. Capturing images in these ideal environments is expensive and time-consuming. Additionally, these ideal conditions rarely exist within the constraints of a "deployed" laboratory system. For this reason, machine vision systems may not be accurately trained due to the mismatch between the images used to train the machine vision system and the actual images captured during the use of the machine vision system within a deployed diagnostic laboratory system. Therefore, there is a need for systems and methods to improve the training of machine vision systems in diagnostic laboratory systems.

[0014] Embodiments of the systems and methods described herein overcome problems related to the training of sample container identification and classification AI models by capturing training images of sample containers under actual conditions within a deployed laboratory system and, in some cases, automatically annotating these training images. The training images can be used to train or retrain an AI model (e.g., an annotation generator) in a diagnostic laboratory system.

[0015] In some embodiments, the diagnostic laboratory system and method disclosed herein utilize a robot used to move and / or position a sample container. An imaging device is coupled to the robot and can be used to capture training images of the sample container within the diagnostic laboratory system. Use of the robot enables specific movement between the imaging device and the sample container, such that the images of the sample container can include a predetermined variation between the images. The variation between the images can include various poses, illumination intensities, illumination spectra, exposure times, and other imaging conditions. As such, a number of varying training images can be obtained within the deployed diagnostic laboratory system.

[0016] Furthermore, in some embodiments, once an image is annotated, the annotated image can be used to retrain how future images will be annotated. That is, annotate a first set of images obtained under a first set of conditions (e.g., lighting, pose, motion profile, etc.), and then use this to train how the AI model of the examination room system will annotate a subsequent second set of captured images obtained under a different second set of conditions (e.g., different lighting, pose, motion profile, etc.). For example, train the AI used to annotate images, which is referred to herein for convenience as an "annotation generator", to annotate a well - lit sample container imaged using a controllably movable imaging device (e.g., an imaging device attached to a robot). When the annotation generator is trained using well - lit sample container images, a first set of images of the tray of sample containers can be obtained under the same well - lit conditions, and the annotation generator can annotate the first set of images. Since the annotation generator is trained using images under well - lit conditions, the annotation of the first set of images should be accurate. Thereafter, the lighting condition can be reduced (e.g., to half intensity or some other reduced intensity), and a second set of images can be captured by the imaging device. The precise control provided by the robot enables the imaging device to be positioned at exactly the same viewing position so that it obtains the second set of images on exactly the same tray of sample containers. Since all conditions except the lighting intensity were the same during the capture of the first and second sets of images, the annotation used for the first set of images can serve as the annotation for the second set of images. Using the annotation and the second set of images, the annotation generator can be refined (e.g., retrained) to annotate images obtained under reduced lighting conditions (e.g., half intensity).Therefore, the annotation generator itself can be made part of the sample characteristic evaluation algorithm and can be repeatedly trained to handle even more variations. That is, with the controllable movement of the imaging device, using the annotations of the first set of images acquired under the first set of conditions, it becomes possible to annotate a second set of images acquired under a second set of conditions (e.g., different lighting intensities, different lighting spectra, different motion profiles, different sample container positions, etc.). Using the above process, the annotation generator can be trained to annotate sample containers, images of sample container holders, or other image features using a variety of imaging conditions within an actual deployed diagnostic examination room system.

[0017] These and other systems and methods are described in more detail below with reference to FIGS. 1 - 13F.

[0018] Referring now to FIG. 1, which shows a block diagram of an exemplary embodiment of a diagnostic examination room system 100. The examination room system 100 can include a plurality of devices 102 configured to process sample containers 104 (some of which are labeled) and perform an assay or test on a sample disposed within the sample container 104. The examination room system 100 can have a first device 102A and a second device 102B. Other embodiments of the examination room system 100 can include more or fewer devices.

[0019] The sample placed within the sample container 104 can be various biological specimens collected from an individual such as a patient being evaluated by a medical professional. The sample can be collected from the patient and placed directly within the sample container 104. Next, the sample container 104 can be delivered to the laboratory system 100. As will be described in more detail below, the sample container 104 can be loaded into the sample handler 106. The sample handler 106 can be a device of the laboratory system 100. From the sample handler 106, the sample container 104 can be transferred into sample carriers 112 (some of which are labeled), and the sample carriers 112 transport the sample container 104 to devices 102 etc. throughout the laboratory system 100 by means of a track 114.

[0020] The track 114 is configured to enable the sample carrier 112 to move throughout the laboratory system 100, including movement to and from the sample handler 106. For example, the track 114 can extend proximate to or around at least some of the devices 102 and the sample handler 106 as shown in FIG. 1. The devices 102 and the sample handler 106 can have devices such as robots (not shown in FIG. 1) that transfer the sample container 104 to and from the sample carrier 112. The track 114 can include a plurality of segments 120 (some of which are labeled) that can be interconnected. In some embodiments, some of the segments 120 can be integrated with one or more of the devices 102.

[0021] The components such as the sample handler 106 and the device 102 of the inspection room system 100 can include or be coupled to a computer 130 configured to execute one or more programs that control the inspection room system 100 including the components of the sample handler 106. The computer 130 can be configured to communicate with the device 102, the sample handler 106, and other components of the inspection room system 100. The computer 130 can include a processor 132 configured to execute a program including programs other than those described herein. The program can be implemented in computer code.

[0022] The computer 130 can include or have access to a memory 134 capable of storing one or more programs and / or data described herein. The memory 134 and / or the program stored therein can be referred to as a non-transitory computer-readable medium. The program can be computer code executable on or by the processor 132. The memory 134 can include a robot controller 136 (e.g., computer code executable by the processor 132) configured to generate instructions for controlling robots and / or similar devices in the device 102 and the sample handler 106. As described herein, the instructions generated by the robot controller 136 can be in response to data such as image data received from the sample handler 106.

[0023] Memory 134 can also store a sample property evaluation algorithm 138 (e.g., a classification algorithm or other suitable computer code) configured to identify and / or classify sample containers 104 and / or other items within sample handler 106. In some embodiments, property evaluation algorithm 138 classifies objects within the image data generated by the imaging devices described herein. Property evaluation algorithm 138 can include a trained model such as one or more neural networks. In some embodiments, property evaluation algorithm 138 can include an annotation generator (912 of FIG. 9) configured to annotate images captured by the imaging device. Property evaluation algorithm 138 can also include a convolutional neural network (CNN) trained to evaluate or identify objects within the image data. The trained model is implemented using artificial intelligence (AI). Thus, as described herein, the trained model can learn to classify sample container 104. Note that property evaluation algorithm 138 is a supervised or unsupervised model trained to evaluate and / or identify various types of sample containers 104, rather than a look-up table.

[0024] The feature evaluation algorithm 138 can also include one or more algorithms for training an AI (e.g., a neural network or other AI model) used to annotate, classify, and / or identify the sample container 104. The AI can be trained based on training images captured by at least one imaging device (not shown in FIG. 1, see, for example, cameras 636, 638 in FIG. 6). In some embodiments, the training images can be captured within the sample handler 106. There can be relative movement between the imaging device and the sample container. For example, a robot disposed on one or more of the instrument 102 and / or the sample handler 106 can be configured to move the imaging device relative to the sample container 104. Additionally, the robot can be configured to move the sample container 104 relative to the imaging device. During training, the feature evaluation algorithm 138 can instruct the robot controller 136 to generate commands to move the robot to a specific position to capture a specific image of the sample container 104.

[0025] The imaging controller 139 can be implemented on the computer 130. For example, the imaging controller 139 can be computer code stored in the memory 134 and executed by the processor 132. The imaging controller 139 can be configured to control the imaging device (e.g., imaging devices 226, 240 in FIG. 2) and the illumination source (e.g., illumination sources 642, 652 in FIG. 6) during image capture. For example, the imaging controller 139 can control a camera (e.g., cameras 636, 638 in FIG. 6) by setting a predetermined frame rate and exposure time during imaging. The imaging controller 139 can also set the illumination intensity and spectrum of the light used to illuminate the sample container 104 during imaging.

[0026] Computer 130 can be coupled to a workstation 140 configured to enable a user to interface with the examination room system 100. The workstation 140 can include a display 142, a keyboard 144, and other peripheral devices (not shown). Data generated by the computer 130 can be displayable on the display 142. In some embodiments, the data can include warnings of anomalies detected by the characteristic evaluation algorithm 138. The anomalies can include notifications that a particular one of the sample containers 104 cannot be characterized. Additionally, a user can input data into the computer 130 by the workstation 140. For example, data input by the user can be instructions to cause the robot controller 136, the characteristic evaluation algorithm 138, or the imaging controller 139 to perform a particular operation such as capturing and / or analyzing an image of the sample container 104. Other data input by the user can include annotations of training images used during the training of the characteristic evaluation algorithm 138.

[0027] Now, further refer to FIG. 2 showing a top view inside the sample handler 106 according to one or more embodiments. The sample handler 106 is configured to capture an image of the sample container 104 and transport the sample container 104 between the holding positions 210 (some of which are labeled) and the sample carrier 112. In the embodiment of FIG. 2, the holding positions 210 are located within a tray 212 that can be removable from the sample handler 106. The sample handler 106 can include a plurality of slides 214 configured to hold the tray 212. In some embodiments, the sample handler 106 can include four slides 214 individually referred to as the first slide 214A, the second slide 214B, the third slide 214C, and the fourth slide 214D. The third slide 214C is shown in a partially removed state from the sample handler 106, which may occur during the replacement of the tray 212. Other embodiments of the sample handler 106 may include fewer or more slides than shown in FIG. 2.

[0028] Each of the slides 214 can be configured to hold one or more trays 212. In the embodiment of FIG. 2, the slide 214 can include a receptacle 216 configured to receive the tray 212. Each of the trays 212 can include a plurality of holding positions 210, and each of the holding positions 210 can be configured to receive one of the sample containers 104. In the embodiment of FIG. 2, the trays can vary in size to include a large tray having 24 holding positions 210 and a small tray having 8 holding positions 210. Other configurations of the tray 212 can include various numbers of holding positions 210 and holding positions configured to hold more than one sample container.

[0029] In some embodiments, the sample handler 106 can include one or more slide sensors 220 configured to detect movement of one or more of the slides 214. The slide sensor 220 can generate a signal indicative of the movement of the slide, and the signal can be received and / or processed by the robot controller 136 as described herein. In the embodiment of FIG. 2, the sample handler 106 includes four slide sensors 220 arranged such that each of the slides 214 is associated with one of the slide sensors 220. The first slide sensor 220A detects movement of the first slide 214A, the second slide sensor 220B detects movement of the second slide 214B, the third slide sensor 220C detects movement of the third slide 214C, and the fourth slide sensor 220D detects movement of the fourth slide 214D. Various techniques can be used by the slide sensor 220 to detect movement of the slide 214. In some embodiments, the slide sensor 220 can include a mechanical switch that toggles when the slide 214 is moved, and the toggle generates an electrical signal indicating that the slide has moved. In other embodiments, the slide sensor 220 can include an optical sensor that generates an electrical signal in response to movement of the slide 214. In still other embodiments, the slide sensor 220 can be an imaging device that generates image data of the sample container 104 as the slide 214 moves.

[0030] The sample handler 106 can receive many different types of sample containers 104. The first type of sample container 104 is indicated by a triangle, the second type of sample container 104 is indicated by a square, and the third type of sample container 104 is indicated by a circle. The characteristic evaluation algorithm 138 is configured to classify the sample container 104 so that the sample container 104 can be easily identified by the computer 130 (FIG. 1). The characteristic evaluation algorithm 138 can also evaluate the characteristics of a new type of sample container (e.g., sample container 204) as described herein.

[0031] The sample handler 106 includes a sample container 204 (marked as a cross) of a new type or not classified by the characteristic evaluation algorithm 138. In the embodiment of FIG. 2, the sample container 204 is disposed within a tray 212A that can be designated to hold a new type of sample container. For example, when it is determined that the sample container 204 is within the tray 212A, the computer 130 can determine whether the sample container 204 is of a new type. If the sample container 204 is of a new type, as described herein, the computer 130 can cause the characteristic evaluation algorithm 138 to classify or evaluate the characteristics of the sample container 204.

[0032] In some embodiments, the tray 212A can have a mark 205 indicating that the tray 212A contains a new type of sample container 204. The user can load the sample container 204 into the tray 212A and insert the tray 212A into the sample handler 106. Next, the imaging device can capture an image of the mark 205. Next, the computer 130 can cause the characteristic evaluation algorithm 138 to classify the sample container 204 in response to the detection of the mark 205. In other embodiments, the user can indicate, via the workstation 140 (FIG. 1), that the sample container 204 has been received by the sample handler 106. In some embodiments, the user can indicate the position of the sample container 204 in the sample handler 106.

[0033] Reference is further made to FIGS. 3A-3C, which illustrate exemplary sample containers of different types that can be used within the inspection room system 100. Other types of sample containers may be used. In some embodiments, the sample container includes a tube with or without a cap attached to the tube. The sample container can also include a sample or other contents (e.g., a liquid) located within the sample container. Further reference is made to FIGS. 4A-4C, which illustrate sample containers without caps as shown in FIGS. 3A-3C. As shown in the figures, all sample containers can have different configurations or geometric shapes. For example, the caps and tubes of various sample container types can each have different geometric shapes and / or different features such as color of the tube and the cap. As described herein, the unique features of the sample container can be classified and identified by the characteristic evaluation algorithm 138 (FIG. 1). The characteristic evaluation algorithm 138 can also be trained using the features described herein (as described below).

[0034] The exemplary sample container 330 of FIG. 3A includes a cap 330A that is white with a red stripe and has an extended vertical portion. The cap 330A fits over the tube 330B. The sample container 330 has a height H31. FIG. 4A shows the tube 330B without the cap 330A. The tube 330B has a geometric shape of a tube that includes a height H41 and a width W41. The tube 330B can have a color of the tube, a material of the tube, and / or a surface characteristic of the tube (e.g., reflectivity). These dimensions, ratios of dimensions, and other characteristics can be referred to as a configuration and can be used to classify and / or identify the sample container 330 by the characteristic evaluation algorithm 138 during classification.

[0035] The exemplary sample container 332 of FIG. 3B has a dome-shaped top and is blue, and includes a cap 332A that fits over the tube 332B. The sample container 332 has a height H32. FIG. 4B shows the tube 332B without the cap 332A. The tube 332B can have a tube geometry including a height H42 and a width W42. The tube 332B can also have a tube color, a tube material, and / or a tube surface property. These dimensions, dimensional ratios, and other properties can be referred to as a configuration and can be used to classify and / or identify the sample container 332 by the property evaluation algorithm 138 during classification.

[0036] The exemplary sample container 334 of FIG. 3C has a flat top and is red and gray, and includes a cap 334A that fits over the tube 334B. The sample container 334 has a height H33. FIG. 4C shows the tube 334B without the cap 334A. The tube 334B can have a tube geometry including a height H43 and a width W43. The tube 334B can also have a tube color, a tube material, and / or a tube surface property. These dimensions, dimensional ratios, and other properties can be referred to as a configuration and can be used to classify and / or identify the sample container 332 by the property evaluation algorithm 138 during classification.

[0037] The tube 330B has an identification mark in the form of a barcode 330C, and the tube 334B has an identification mark in the form of a barcode 334C. As described herein, the images of the barcode 330C and the barcode 334C can be analyzed by the property evaluation algorithm 138 for classification purposes. (As described below) The barcode can be referred to as a feature and can be used to train the property evaluation algorithm 138.

[0038] Different types of sample containers can have different characteristics, such as different sizes, different surface properties, and different chemical additives inside, as shown in sample containers 330, 332, and 334 of FIGS. 3A - 3C. For example, some sample container types are chemically active, which means that the sample container contains one or more chemical additives used to change or maintain the state of the sample stored inside or otherwise assist in sample processing by the instrument 102. In some embodiments, the inner wall of the tube can be coated with one or more additives, or the additives can be provided elsewhere in the sample container. In some embodiments, the types of additives included in the tube can be serum separators, coagulants such as thrombin, anticoagulants such as EDTA or sodium citrate, antiglycolytic additives, or other additives for changing or maintaining the properties of the sample. For example, the sample container manufacturer can associate the color of the cap on the tube and / or the shape of the tube or cap with a particular type of chemical additive included in the sample container.

[0039] Different manufacturers can have their own criteria for associating attributes of a sample container, such as the color of the cap, the shape of the cap (e.g., the geometric shape of the cap), and the tube shape, with specific characteristics of the sample container. For example, the attributes can be related to the contents of the sample container or, in some cases, whether the sample container is provided with a vacuum capacity. In some embodiments, the manufacturer can associate all sample containers with a gray cap, and the tube contains potassium oxalate and sodium fluoride configured to test for glucose and lactic acid. A sample container with a green cap can contain heparin for stat electrolytes such as sodium, potassium, chloride, and bicarbonate. A sample container with a lavender cap can identify a tube containing EDTA (ethylenediaminetetraacetate - anticoagulant) configured to test for fractionated CBC (CBC with differential), HgBA1c, and parathyroid hormone. Other cap colors, such as red, yellow, light blue, dark blue, pink, orange, and black, can be used to indicate other additives or the absence of additives. In other embodiments, combinations of cap colors can be used, such as yellow and lavender indicating a combination of EDTA and a gel separator, or green and yellow indicating lithium heparin and a gel separator.

[0040] The laboratory system 100 can use the sample container attributes to further process the sample container 104 and / or the sample contained in the sample container 104. Since the sample container 104 can be chemically active and can affect the tests performed on the sample contained therein, it is important to associate the specific tests that can be performed on the sample with the specific sample container type. For this reason, the laboratory system 100 can confirm that the test being performed on the sample in the sample container 104 is correct by analyzing the color and / or shape of the cap and / or the tube. Other container attributes can also be analyzed.

[0041] Referring again to FIG. 2, the sample handler 106 can include an imaging device 226 that is movable throughout the sample handler 106. In the embodiment of FIG. 2, the imaging device 226 is attached to a robot 228 that is movable along an x-axis (e.g., in the x direction) and a y-axis (e.g., in the y direction) throughout the sample handler 106. In some embodiments, the imaging device 226 can be integral with the robot 228. In one or more embodiments, the robot 228 can further be movable along a z-axis (e.g., in the z direction) in and out of the page. In other embodiments, the robot 228 can include one or more components (not shown in FIG. 2) that move the imaging device 226 in the z direction.

[0042] In some embodiments, the robot 228 can receive movement instructions generated by a robot controller 136 (FIG. 1). The instructions can be data indicating the x, y, and z positions to which the robot 228 should move. In other embodiments, the instructions can be electrical signals that move the robot 228 in the x, y, and z directions. The robot controller 136 can generate instructions for moving the robot 228 in response to one or more of the slide sensors 220 detecting movement of one or more of the slides 214, for example. The instructions can move the robot 228 while the imaging device 226 is capturing an image of a newly added sample container 204.

[0043] The imaging device 226 includes one or more cameras (not shown in FIG. 2, see, for example, cameras 636, 638 in FIG. 6) that capture images, and the capture of the images generates image data representing the images. As described herein, the image data can be transmitted to the computer 130 and processed by the characteristic evaluation algorithm 138. The one or more cameras are configured to capture images of the sample containers 104, 204, and / or other positions or objects within the sample handler 106. The images can be, for example, the top and / or sides of the sample containers 104, 204. In some embodiments, the robot 228 can be a gripper robot that grips the sample containers 104, 204 and moves the sample containers 104, 204 between the holding position 210 and the sample carrier 112. In such embodiments, as described herein, the images can be captured while the robot 228 is gripping the sample containers 104, 204.

[0044] Further refer to FIG. 5, which is a perspective view of an embodiment of the robot 228 coupled to a gantry 530 configured to move the robot 228 in the x, y, and z directions. The gantry 530 can include two y slides 532 that allow the robot 228 to move in the y direction, an x slide 534 that allows the robot 228 to move in the x direction, and a z slide 536 that allows the robot 228 to move in the z direction. In some embodiments, the movements in the three directions can be simultaneous and can be controlled by commands generated by the robot controller 136 (FIG. 1). For example, the robot controller 136 can generate commands for a motor (not shown) coupled to the gantry 530 to move the slides to move the robot 228 and the imaging device 226 to a predetermined position or in a predetermined direction.

[0045] In some embodiments, the robot 228 can include a gripper 540 (e.g., an end effector) configured to grip the sample container 504. The sample container 504 can be an example of one of the sample containers 104 or one of the sample containers described in FIGS. 3A - 3C. The robot 228 is moved to a position above the holding position and then moved in the z - direction to remove the sample container 504 from the holding position. The gripper 540 opens, and the robot 228 moves downward in the z - direction such that the gripper 540 extends over the sample container 504. The gripper 540 closes to grip the sample container 504, and the robot 228 moves upward in the z - direction to extract the sample container 504 from the holding position. As shown in FIG. 5, since the imaging device 226 can be attached to the robot 228, the imaging device 226 moves with the robot 228 and can capture images of the sample container 504 located within the sample handler 106 and other sample containers 104, 204 (FIG. 2). The imaging device 226 includes at least one camera configured to capture images, and the captured images are converted into image data for processing by, for example, the characteristic evaluation algorithm 138. The characteristic evaluation algorithm 138 can be trained using the image data. In some embodiments, the image data can train or update the annotation generator 912 (FIG. 9).

[0046] Further refer to FIG. 6, which is a side view of an embodiment of a robot 228 that grips a sample container 504 using a gripper 540 while the sample container 504 is being imaged by an imaging device 226. The imaging device 226 shown in FIG. 6 can include a first camera 636 and a second camera 638. Other embodiments of the imaging device 226 may include a single camera or three or more cameras. The first camera 636 has a field of view 640 that at least partially extends in the y direction and can be configured to capture an image of the sample container 504 being gripped by the gripper 540. A first light source 642 can illuminate the sample container 504 within the field of view 640 by an illumination field 644. In some embodiments, the spectrum and / or intensity of the light emitted by the first light source 642 can be controlled by a characteristic evaluation algorithm 138 (FIG. 1) and / or an imaging controller 139 (FIG. 1). In other embodiments, the imaging controller 139 is configured to control at least one of the intensity of the first light source 642 and the spectrum of the light emitted by the first light source 642.

[0047] The second camera 638 can have a field of view 650 extending in the z direction and can capture images of the tray 212, the sample containers 104, 204 located within the tray 212, and other objects within the sample handler 106. The second light source 652 can illuminate the objects within the field of view 650 by means of an illumination field 654. In some embodiments, the spectrum and / or intensity of the light emitted by the second light source 652 can be controlled by the imaging controller 139. The field of view 650 and the illumination field 654 enable images of the upper portions (e.g., caps) of the sample containers 104, 204 to be captured as shown in FIG. 2. The captured images can be analyzed by a feature evaluation algorithm 138 (FIG. 1) to classify or identify the sample containers 104, 204 and / or to determine whether any abnormalities are present within the sample handler 106. In some embodiments, the imaging device 226 can have a single camera having a field of view capable of capturing one or more of at least a portion of the sample handler 106 and the holding position 210, with or without the sample containers 104, 204 disposed therein.

[0048] In some embodiments, the images can be captured as the robot 228 moves the imaging device 226 relative to the sample containers 104, 204. The robot controller 136 (FIG. 1) can set the speed and direction of the robot 228 relative to the sample containers 104, 204 during image capture.

[0049] The operations of the first camera 636, the second camera 638, the first light source 642, and / or the second light source 652 can be controlled by an imaging controller 139 (FIG. 1). As described herein, the imaging controller 139 can set imaging conditions for one or more of these devices during imaging. For example, the imaging controller 139 can set an exposure time, a frame rate, an illumination intensity, and / or an illumination spectrum during image capture. In some embodiments, a characteristic evaluation algorithm 138 can determine the imaging conditions. Additional images can be captured under a second imaging condition or a modified imaging condition.

[0050] As described herein, images captured by the imaging device 226 can be analyzed by a characteristic evaluation algorithm 138 to determine characteristics of the sample container 504, the robot 228, the sample containers 104, 204, and other components within the sample handler 106. For example, the characteristic evaluation algorithm 138 can characterize or identify the container types of the sample containers 104, 204, 504. When the image data generated by the first camera 636 is analyzed, the characteristic evaluation algorithm 138 can analyze a side view of the sample container 504. The characteristic evaluation algorithm 138 can also determine whether the sample container 504 is properly grasped by the gripper 540. When the image data generated by the second camera 638 is being analyzed, the tops or caps of the sample containers 104, 204 can be characterized. (As will be described below with reference to FIG. 9,) the annotation generator 912 can be trained or this training can be updated using images generated in different views.

[0051] Refer further to FIG. 7, which is a side view of another embodiment of robot 228 of FIG. 6, in which gripper 540 is pivotally coupled to the main structure 752 of robot 228. This embodiment of robot 228 includes a secondary arm 754 coupled to the main structure 752 by a pivoting mechanism 756, which allows the secondary arm 754 to rotate such that it traces an arc R relative to the main structure 752. In the embodiment of FIG. 7, since the gripper 540 is coupled to the secondary arm 754 and the imaging device 226 is coupled to the main structure 752, the sample container 504 can pivot relative to the imaging device 226, thereby enabling an image of the sample container 504 to be taken in different poses, such as tilted, during capture. In some embodiments, the pivoting mechanism 756 allows the secondary arm 754 to pivot in a direction other than arc R, such as in and out of the plane of the paper. The feature evaluation algorithm 138 can determine the pose of the sample container 504 relative to the imaging device 226, and the robot controller 136 can generate commands to move the robot 228 to the correct pose. The annotation generator 912 (FIG. 9) can be trained or this training can be updated using images generated during different poses.

[0052] Referring again to FIGS. 2 and 5, in some embodiments, the sample handler 106 includes a fixed imaging device 240 that can be in a fixed position. In such embodiments, the robot 228 can move the sample containers 104, 204 in proximity to the imaging device 240, where the imaging device 240 can then capture an image of the sample containers 104, 204. As described herein, the images generated by the imaging device 240 can be processed by, for example, the feature evaluation algorithm 138. The imaging device 240 can include a camera 242 and an illumination source 244, and the illumination source 244 can be configured to illuminate the object being imaged by the camera 242. In some embodiments, the spectrum and / or intensity of the light emitted by the illumination source 244 can be controlled by the feature evaluation algorithm 138 and / or the imaging controller 139.

[0053] Having described an exemplary embodiment of the examination room system 100, next, a method for processing image data generated by the examination room system 100 will be described. The examination room system 100 and method described herein generate data and annotations with real-world variations by using a combination of the imaging device 226 and the robot 228 to generate image data. Embodiments are applicable to sample container characteristic evaluation, and the characteristic evaluation can include evaluating sample containers 104, 204, and / or 504 with and without caps and / or samples contained within the sample containers 104, 204, and / or 504. As described herein, the characteristic evaluation can include annotations.

[0054] Further refer to FIG. 8, which is a flowchart of an exemplary sample container characteristic evaluation workflow 800 that can be implemented in the characteristic evaluation algorithm 138 and executed by the processor 132. In some embodiments, the robot 228 can grip the sample container 504 (FIG. 5), and the imaging device 226 can capture an image of the sample container 504. In other embodiments, the robot 228 can move the imaging device 226 relative to the sample containers 104, 204, whereby the imaging device 226 can capture an image of a particular one of the sample containers 104, 204. During image capture, each of the illumination sources 642 or 652 (FIGS. 6 and 7) can illuminate the sample containers 104, 204, 504 using a predetermined intensity and spectrum of light (e.g., full intensity, half intensity, etc., white light, red light, green light, blue light, etc.). In some embodiments, the predetermined intensity and spectrum of light can be determined by the characteristic evaluation algorithm 138. Next, the imaging device 226 can capture images of the sample containers 104, 204, 504 under these illumination conditions. The imaging controller 139 can set other illumination conditions during image capture. The images can be captured using one or both of the first camera 636 and the second camera 638 in the imaging device 226. Thus, the images can include the tops of the sample containers 104, 204 and / or the sample container 504 held by the gripper 540. The sample container 504 can have different orientations relative to the imaging device 226 (e.g., using a pivoting mechanism 756).

[0055] In operation block 802, image data can be received, and in operation block 802, preprocessing such as blur correction, gamma correction, and radial distortion correction can be performed before further processing. A data-driven machine learning approach such as an adversarial generation network (GAN) or another suitable AI network can be used for the preprocessing in operation block 802. The process can proceed to operation block 804, where an image of the sample container 504 can undergo sample container position identification and classification. (Although the process is described with respect to sample container 504, it is applicable to sample containers 104, 204.) The position identification can be an annotation of the image of the sample container 504 to specify the position of the sample container 504 within each image. For example, to isolate the sample container 504, it can include surrounding the image of the sample container 504 using a virtual box (e.g., a bounding box or pixel-wise mask). Classification can be performed using a data-driven machine learning-based approach such as a convolutional neural network (CNN).

[0056] The CNN can be enhanced using YOLOv4 or other image identification networks or models. YOLOv4 is a real-time object detection model that functions by dividing the object identification task into two operations. The first operation identifies the object position via a bounding box using regression, and the second operation determines the class of the object (e.g., sample container 104, 204, or 504) using classification. The position identification can provide a bounding box for the detected sample container. The classification determines high-level characteristics of the sample container, such as whether the sample container has a cap or not, or whether it is a tube top sample cup (TTSC). In some embodiments, the classification also determines the classification confidence.

[0057] The process can proceed to sample container tracking in operation block 806. In operation block 806, for each newly detected sample container, the computer 130 can assign a new tracklet identification (e.g., an identification of a portion of the path the sample container has traveled, such as a portion of track 114) to each sample container (e.g., via the robot controller 136 and / or the feature evaluation algorithm 138). Alternatively, the computer 130 can attempt to associate the detected sample container with an existing tracklet established in a previous image based on the overlap area between the detected bounding box and a predicted bounding box established on the motion trajectory, the classification confidence, and other features derived from the appearance of the image of the sample container. In situations where detections are potentially missing, this can prevent tracking, and more advanced data association algorithms such as the Hungarian algorithm can be utilized to ensure the robustness of tracking. In some embodiments, deep SORT or other machine learning algorithms can be used for sample container tracking.

[0058] When a tracklet contains sufficient observations collected over multiple images (e.g., frames), the feature evaluation algorithm 138 can start estimating more detailed features in operation block 808. Features can include, but are not limited to, sample container height and sample container diameter, cap color, cap shape, and barcode read values when a barcode or other sample container identification mark is within the field of view of the imaging device 226 or the imaging device 240.

[0059] Training data-driven machine learning algorithms, software models, and networks may require collecting image data under various controlled (e.g., predetermined) conditions. The varying conditions can include different sample container types, lighting conditions (e.g., lighting intensity, lighting spectrum, etc.), camera spectral characteristics, exposure time, sample container distance and / or orientation, relative motion between the imaging device 226 and the sample container, etc.

[0060] FIG. 9 is a diagram of an exemplary workflow 900 for generating image data under varying conditions according to one or more embodiments described herein. Referring to FIG. 9, a coordinator 902 is provided for directing workflow 900. In some embodiments, coordinator 902 can be implemented as computer program code stored in memory 134 (FIG. 1) and executed by processor 132. For example, coordinator 902 can be implemented in feature evaluation algorithm 138. Coordinator 902 can be coupled to a robot controller 136 for controlling robot 228, a lighting controller 906 for controlling the operation of lighting sources 642 and 652, an imaging controller 139 for controlling the operation of imaging devices 226 and 240, and an annotation generator 912 for annotating captured images as described below. In particular, coordinator 902 can (1) use robot controller 136 to position robot 228 and imaging device 226 coupled thereto; (2) use lighting controller 906 to illuminate a sample container with lighting sources 642 and / or 652 (e.g., using a desired illumination intensity, illumination spectrum, etc.); (3) use imaging controller 139 to instruct imaging devices 226 and / or 240 to capture an image of the sample container (e.g., image data 914) (e.g., using a desired exposure time or other imaging parameters); and (4) use annotation generator 912 to generate an annotated captured image (e.g., annotated image data 916), to control workflow 900 to generate image data of the sample container. As further described below, coordinator 902 can also store annotations used for one or more images in annotation generator 912 and instruct the stored annotations to be reused for one or more subsequent captured images. Further, in some embodiments, coordinator 902 can instruct retraining of annotation generator 912.

[0061] One or more positions of the sample containers 104, 204, or 504 to be characterized can be stored, for example, in at least one of the robot controller 136 or the characterization algorithm 138. In some embodiments, the characterization includes annotating or identifying an image of the sample container. For example, the sample container 204 to be characterized can be positioned within the tray 212A (FIG. 2). The robot controller 136 can generate a signal or command for the robot 228 to retrieve the sample container 204 and arrange an individual one of the sample containers in a specific position or orientation (e.g., pose) with respect to the imaging device 226 (or imaging device 240), whereby the imaging device can capture an image of the sample container (e.g., sample container 204). For example, the specific position can include a predetermined distance from the imaging device 226 (or 240), a predetermined orientation (e.g., angle) with respect to the imaging device 226 (or 240), and a relative movement between the sample container being imaged and the imaging device 226 (or 240) during imaging.

[0062] The illumination controller 906 manages the illumination intensity and / or spectrum of the illumination sources 642, 652. In some embodiments, the illumination controller 906 can be implemented in the imaging controller 139. The characterization algorithm 138 can generate commands or imaging requirements that can be converted by the illumination controller 906 to generate commands for controlling the illumination sources 642, 652 and / or the cameras 636, 638. The image controller 139 can instruct the cameras 636, 638 to generate image data 914. The image data 914 can be digital data representing a captured image of the sample container 204 under the illumination conditions established by the illumination controller 906.

[0063] The annotation generator 912 can identify objects within an image and label the objects. During some annotation processes, a bounding box can be generated within the image, and the bounding box will contain one or more objects to be identified. The objects can be identified or classified as classes or instances. For example, using the annotation generator 912, a sample container can be identified as a class of objects within the image. In other embodiments, segmentation can be used to identify specific instances such as the type of sample container within the image. The annotation generator 912 can use tools other than bounding boxes. For example, the annotation generator 912 can use polygon segmentation, semantic segmentation, 3D cuboids, keypoints and landmarks, or lines and splines. Next, the annotations can be used to create a training dataset for sample container identification. In some embodiments, the annotation generator 912 can include a deep learning network such as a general convolutional neural network (CNN). Exemplary networks include Inception, ResNet, ResNeXt, DenseNet, etc., although other CNNs and / or AI architectures may be used. The training of the annotation generator 912 will be further described below with reference to FIG. 10.

[0064] As further described herein, the annotation generator 912 can generate a predicted annotation of an image of a sample container represented by the image data 914 by leveraging previously annotated data. The annotation generator 912 generates annotated image data 916 from the image data 914. Next, the annotated image data 916 can be fed back to the annotation generator 912 for further annotation and / or further training of the annotation generator 912. In some embodiments, after performing a first annotation of one or more objects in a first image, the first annotation can be reused to perform a second annotation of one or more objects in a second image with respect to the second image. For example, the illumination intensity, illumination spectrum, pose, or another condition may vary between the capture of the first and second images, but as further described below with reference to FIG. 11, (e.g., if the sample container is expected to be in the same position within both images due to precise positioning of the imaging device 226 by the robot 228, etc.,) the second image can be annotated using the annotation of the first image. The annotation generator 912 can be trained by this process or the annotation generator 912 can have its training updated by this process.

[0065] The image annotation performed by the annotation generator 912 can include the task of annotating an image of a sample container using labels. In some embodiments, some of the annotations may involve additional human input tasks. The labels can be predefined during the programming of machine learning and are selected to provide information for a computer vision model (e.g., the feature evaluation algorithm 138) regarding the objects in the image. Exemplary considerations during annotation can include possible naming and classification issues, representation of occluded objects (e.g., an occluded tube or an occluded sample contained within a tube), labeling of unrecognizable portions of the image, and other considerations.

[0066] The annotation of an image of the sample container 104, 204, or 504 by the annotation generator 912 can include applying multiple labels to the objects in the image of the sample container by applying a bounding box to a particular one of the objects. For example, the cap, tube, and identification mark can be bounded by a bounding box. This process can be repeated and depending on the classification required, the amount of labels in each image can vary. Some classifications may require only one label (e.g., image classification) to represent the content of the entire image. Other classifications may require that multiple objects be annotated within a single image, each having a different label (e.g., different bounding boxes). For example, at least two of the cup, tube, and identification mark may need to be annotated to classify some types of sample containers.

[0067] The reproducibility of the positioning between the robot 228 and the sample containers 104, 204, or 504 enables the above-described method of annotating objects in the images of the sample containers. For example, the image sequence can be captured with a slow movement between the robot 228 and the sample container 204 under well-illuminated lighting conditions, whereby the annotation becomes relatively easy either in an automated manner or in a semi-supervised manner. Additionally, by capturing images of the same sample container at multiple known positions or sample container orientations, it is possible to extract high-resolution depth information of the sample container using stereo vision or multi-view stereo vision. The stereo images enable the reconstruction of a three-dimensional image (3-D image) of the sample containers 104, 204, or 504, providing detailed features for distinguishing sample categories, which can be useful for automating the annotation process. For example, some capped sample containers having a black / grey center and a white outer ring may appear almost identical to uncapped sample containers when viewed using a single top-down image (e.g., an image captured in the z-direction), whereby manual annotation may be required in conventional systems in some cases. In some embodiments, the ground truth can be automatically identified based on the 3-D image.

[0068] For sample container localization, the annotation can be a bounding box or a binary mask of each sample container in the image. For sample container tracking, the annotation can be a unique identifier for each sample container at the holding position over the image sequence. These annotations can then be propagated in an automated form to another image sequence acquired under different imaging conditions, such as different lighting conditions, motion profiles, viewing positions (e.g., poses), and other imaging conditions, based on the positions of the robot 228 and / or the imaging device 226 relative to the previously annotated image sequence. The annotation generator 912 can train on these images such that the training is an iterative process.

[0069] As described above, the laboratory system 100 can use different types of sample containers 104, 204, or 504 from different manufacturers, etc. The laboratory system 100 should know the type of the sample container 104, 204, or 504 in order to properly transport the sample container 104, 204, or 504 and process the sample. Robots such as robot 228, and the sample carrier 112 can have dedicated hardware and processes for transporting different types of sample containers 104, 204, or 504. For example, the robot 228 can grasp a first type of sample container in a different form from a second type of sample container. In addition, the laboratory system 100 can utilize different types of sample carriers 112 depending on the type of the sample container. For this reason, it is important for the laboratory system 100 to identify the sample container.

[0070] The laboratory system 100 described herein uses a vision system such as the imaging device 226 to capture images of the sample containers 104, 204, or 504. The characteristic evaluation algorithm 138 analyzes the image data generated by the imaging device 226 (or the imaging device 240) to identify and / or classify the sample container. Other imaging devices can capture images of the sample container, and the characteristic evaluation algorithm 138 can analyze the image data generated by these imaging devices.

[0071] The characteristic evaluation algorithm 138 can include an AI model configured to evaluate the variations of different types of sample containers and their respective tubes and / or caps. When a new type of sample container is introduced into the laboratory system 100, the AI model in the characteristic evaluation algorithm 138 should be updated so that it can classify the new type of sample container. As described above, retraining the AI model in a conventional laboratory system can be costly and time-consuming. The laboratory system 100 described herein overcomes the problems associated with new sample container classification by training the annotation generator 912 described herein.

[0072] In another example, the laboratory system 100 may receive a new sample container type from a manufacturer. In some embodiments, the new type of sample container can be loaded into the tray 212A (FIG. 2). Each attribute of the new sample container can be similar to the attributes of a particular sample container for which the characterization algorithm 138 (e.g., including the annotation generator 912) was trained. For example, the new sample container 204 may have the same tube material as the sample container for which the characterization algorithm 138 was trained, but may have a different cap shape. Another sample container type for which the characterization algorithm 138 was trained may have the same cap type as the new sample container type, but may have a different tube material. The characterization algorithm 138 (and the annotation generator 912) can be trained for the new sample container (e.g., by annotating an image of the new sample container type using one or more annotations from a previous image and then retraining the annotation generator 912).

[0073] In some embodiments, a user may receive a new type of sample container, or a sample container that has not yet been properly identified, and can load the sample container into one of the trays 212, such as the tray 212A (FIG. 2). For example, the user can slide the tray 212A into the sample handler 106 via the fourth slide 214D. When the fourth slide 214D is slid into the sample handler 106, the fourth slide sensor 220D can detect the movement and capture an image of the mark 205 that can indicate that the tray 212A contains a new sample container. In other embodiments, the user can input data via the workstation 140 (FIG. 1) indicating that the new sample container is located within the tray 212A. In some embodiments, the tray 212A can also include similar sample containers that can be used to image and train the annotation generator 912.

[0074] The feature evaluation algorithm 138 can send instructions to the robot controller 136 to move the robot 228 to a predetermined position through the use of the coordinator 902 (FIG. 9) so that the imaging device 226 can capture an image of the sample container 104, 204, or 504. The images can be captured under different imaging conditions using one or both of the first camera 636 and the second camera 638. For example, the images can be captured under different lighting and camera conditions determined by the feature evaluation algorithm 138 and the imaging controller 139.

[0075] In some embodiments, the robot 228 can grasp the sample container as shown in FIG. 5 and extract the sample container from the tray 212A. Next, the imaging device 226 can capture an image of the sample container. In some embodiments, the robot 228 can return the sample container to the tray 212A and grasp the sample container again so that the sample container is in a different orientation relative to the imaging device 226. Next, as described herein, the imaging device 226 can capture a new image for processing. Referring to FIG. 7, the robot 228 can rotate the sample container relative to the imaging device 226 via the secondary arm 754. Next, the imaging device 226 can capture an image of the sample container in a different orientation. In some embodiments, the imaging device 226 can capture an image of the sample container as the sample container is rotated relative to the imaging device 226. The rotation speed can be one of the imaging conditions described herein.

[0076] The second camera 638 (FIG. 6) can capture an image from above the sample container similar to the method described above with respect to the first camera 636. In some embodiments, the robot can move the imaging device 226 relative to the sample container when the second camera 638 captures an image of the sample container. The movement of the imaging device 226 relative to the sample container can be one of the imaging conditions described herein.

[0077] Using the image data generated by the imaging device 226, the characteristic evaluation algorithm 138 can be updated, trained, or retrained. For example, the AI model in the characteristic evaluation algorithm 138 can be updated using the image data. In some embodiments, the update or retraining includes training or updating the training of the annotation generator 912 as described below.

[0078] Referring now to FIG. 10, which shows a flowchart of a method 1000 for updating the training of an annotation generator (e.g., annotation generator 912) of a diagnostic examination room system (e.g., examination room system 100). The method includes, at block 1002, providing an imaging device (e.g., imaging device 226) to the diagnostic examination room system, the imaging device being controllably movable within the diagnostic examination room system. The method 1000 includes, at block 1004, capturing a first image within the diagnostic examination room system using the imaging device, the first image being captured under at least one imaging condition (e.g., a predetermined illumination intensity, illumination spectrum, sample container orientation, exposure rate, imaging device, and / or sample container speed, etc.). The method 1000 includes, at block 1006, performing an annotation of the first image using the annotation generator to generate a first annotated image. For example, in some embodiments, the annotation can include surrounding an image of the sample container with a virtual box or other shape (e.g., a bounding box or pixel-wise mask) to isolate the sample container.

[0079] Method 1000 includes, at block 1008, updating the training of an annotation generator using a first annotated image. The annotation generator training can be performed based on a specific algorithm to be trained. In the case of a sample container detection task, the annotation generator 912 can, for example, annotate the bounding box of the sample container based on a detection algorithm under training. In the case of a sample container classification / identification task, the annotation generator 912 can annotate the class / type of the sample container based on a classification algorithm under training. In the case of a semantic segmentation task, the annotation generator 912 can generate an annotation mask at the pixel level for each object region within the input image. Since the annotation is done at the pixel level, the annotation region can be of any irregular shape (e.g., pixel-level mask, polygon, contour, spline) instead of a predefined bounding box in the shape of a square, rectangle, circle, or oval. In some embodiments, the annotation generator 912 can train one or more tasks simultaneously. The annotations generated by the annotation generator 912 can be used together with the input image to perform the next iteration of algorithm / model training, and the updated algorithm / model can be used by the annotation generator 912 to annotate new input images under different imaging conditions (e.g., different illumination intensities, illumination spectra, sample container poses, exposure rates, etc.). Next, the next iteration of training can be performed using the annotations and these new input images. In some embodiments, the annotation generator 912 can be implemented through the use of machine learning techniques so that it can learn to handle increasingly difficult conditions through the iterations of training. For example, the annotation generator 912 can be implemented as a deep neural network algorithm.In some embodiments, the annotation generator 912 can include a deep learning network such as a general convolutional neural network (CNN). Exemplary networks include Inception, ResNet, ResNeXt, DenseNet, etc., although other CNNs and / or AI architectures may be used. Training can be performed continuously, periodically, or at any appropriate time. Training can be performed while the diagnostic examination room system 100 is online (e.g., in use) or offline.

[0080] Now, refer to FIG. 11 which shows a flowchart of another exemplary method 1100 for training an annotation generator (e.g., annotation generator 912) of a diagnostic examination room system (e.g., examination room system 100). Method 1100 includes, at block 1102, providing an imaging device (e.g., imaging device 226) to the diagnostic examination room system, the imaging device being controllably movable within the diagnostic examination room system. For example, the imaging device 226 can be attached to a robot 228 and moved with the robot 228. Method 1100 includes, at block 1104, capturing a first image of a sample container (e.g., sample container 104, 204, or 504) using the imaging device, the first image being captured using certain imaging conditions. In one exemplary embodiment, the imaging condition can be the illumination intensity. For example, the first image can be captured under well - illuminated conditions under which the annotation generator 912 has been previously trained. Other exemplary imaging conditions can include the relative position and angle between the illumination intensity, illumination spectrum, sample container and / or imaging device speed, the angle between the imaging device and one or more sample containers, imaging device exposure, imaging device lens characteristics such as focal length, aperture, depth of field, etc.

[0081] Method 1100 includes, at block 1106, performing an annotation of the first image to generate a first annotated image. For example, the annotation generator 912 can annotate an image using a bounding box, a pixel-wise mask, etc. If the imaging conditions used with the first image are the imaging conditions that the annotation generator 912 was previously trained on, the annotation provided by the annotation generator 912 should be very accurate.

[0082] Method 1100 includes, at block 1108, changing a first imaging condition to a second imaging condition. For example, in an embodiment of a well - illuminated first image, the imaging condition can be to reduce the illumination intensity before capturing the second image. Other imaging conditions can be changed. Method 1100 includes, at block 1110, using an imaging device to capture a second image of a sample container under the second imaging condition. Method 1100 includes, at block 1112, performing an annotation of the second image to generate a second annotated image. In some embodiments, the annotation generator 912 can use the same annotation as that used for the first image. For example, if the imaging condition being changed is the illumination intensity (or illumination spectrum), precise control provided by the robot 228 enables the imaging device 226 to be positioned at exactly the same viewing position so that the second image is acquired on exactly the same tray of the sample container. Since all (or most) conditions other than the illumination intensity (or spectrum) are the same during the capture of the first and second images, the annotation used for the first image can serve as the annotation for the second image. Using the annotation and the second image set, the annotation generator can be refined (e.g., retrained) to annotate the image acquired under that image condition, whether the reduced illumination condition (e.g., half intensity), different illumination spectrum, or changed image condition. Method 1100 includes, at block 1114, training an annotation generator using at least the first annotated image and the second annotated image. For example, both the first annotated image and the second annotated image can be included in the training image set used to train the annotation generator 912. The training can be performed continuously, periodically, or at any appropriate time.

[0083] Method 1100 includes, at block 1116, changing a second imaging condition to a third imaging condition. For example, the imaging conditions can include illumination intensity, illumination spectrum, illumination container and / or imaging device speed, sample container orientation, exposure rate, and the like. Method 1100 includes, at block 1118, using an imaging device to capture a third image of the sample container under the third imaging condition. With respect to the first and second images, in some embodiments, imaging device 226 can be utilized to capture the third image.

[0084] Method 1100 includes, at block 1120, using an annotation generator to perform an annotation of the third image and generate a third annotated image. In some embodiments, annotator generator 912 can use the same annotation that was used for the first image or the second image. For example, if the imaging condition being changed is illumination intensity (or illumination spectrum or exposure rate, etc.), precise control provided by robot 228 enables imaging device 226 to be positioned at exactly the same viewing position that was used for the first and second images in order to acquire the third image. Similarly, an exact change in the position of imaging device 226 with respect to the sample container can be provided between the images. In this way, the annotation used for the first or second image can serve as the annotation for the third image. Using the annotation and the third image, the annotation generator can be retrained to annotate images acquired under different imaging conditions such as reduced illumination, different illumination spectra, different sample container orientations, different exposure rates, and the like.

[0085] Method 1100 includes, at block 1122, further training of annotation generator 912 using at least the third annotated image. As described above, the training can be performed continuously, periodically, or at any suitable point in time.

[0086] Using the annotation and the first, second, and third images, the annotation generator 912 can be retrained to annotate images acquired under different conditions (e.g., reduced illumination, different illumination spectra, different speeds between the sample container and the imaging device, different sample container postures, different exposure rates, etc.). Thus, the annotation generator 912 itself can be made part of the sample characteristic evaluation algorithm 138 and can be repeatedly trained to handle even more variations. That is, with the controllable movement of the imaging device 226, using the annotation of the first image set acquired under the first set of conditions, it becomes possible to annotate the second image set acquired under the second set of conditions (e.g., different illumination intensities, different illumination spectra, different motion profiles, different sample container positions, etc.). This can be extended to the third, fourth, fifth, or other numbers of image sets and / or imaging conditions. Using the above process, the annotation generator 912 can be trained to annotate sample containers, images of sample container holders, or other image features using the various imaging conditions within an actual deployed diagnostic examination room system.

[0087] Figures 12A - 12I show exemplary images and image annotations according to the embodiments provided herein. Referring to Figure 12A, an image 1202 of a sample container 1204 is shown. The sample container 1204 can be similar to the sample containers 104, 204, or 504 described above and includes a tube 1205, a cap 1206, and a label 1208. The sample container 1204 is supported on a carrier 1210. Figure 12B shows an example of a bounding box annotation 1212 of the sample container 1204 of the image 1202, while Figure 12C shows an example of a pixel - wise mask annotation 1214 of the sample container 1204. Other annotation types may also be used.

[0088] FIG. 12D shows an example of a first image 1220a obtained under a first imaging condition (e.g., a first illumination intensity). FIG. 12E shows an example of a first annotated image 1220b based on the first image 1220a (e.g., using a bounding box 1212a or other suitable annotation). For example, the first imaging condition can be a condition under which the annotation generator 912 is trained such that the annotation 1212a of the first image 1220a is highly accurate. FIG. 12F shows an example of a second image 1222a obtained under a second imaging condition different from the first imaging condition used for the first image 1220a. For example, the second image 1222a can be obtained using a different illumination intensity, illumination spectrum, sample container and / or imaging device speed, relative position and angle between the imaging device 226 and the sample container 1204, imaging device exposure, imaging device lens characteristics, e.g., focal length, aperture, depth of field, etc. (represented by the light shading in FIGS. 12F and 12G). FIG. 12G shows an example of a second annotated image 1222b using the annotation 1212a of the first annotated image 1220b as described above based on the second image 1222a. FIG. 12H shows an example of a third image 1224a obtained under a third imaging condition different from the first or second imaging condition (as represented by the intermediate shading in FIGS. 12H and 12I). Finally, FIG. 12I shows a third annotated image 1224b using the annotation 1212a of either the first annotated image 1220b or the second annotated image 1222b based on the third image 1224a. Using the annotated second and / or third images 1222b, 1224b, the annotation generator 912 can be retrained to annotate images obtained under different imaging conditions such as different illumination intensities, illumination spectra, sample containers and / or imaging device speeds, relative position and angle between the imaging device and one or more sample containers, imaging device exposure, imaging device lens characteristics, e.g., focal length, aperture, depth of field, etc.

[0089] Figures 13A - 13F show further exemplary images and image annotations according to embodiments provided herein. Referring to Figure 13A, an image 1302 of a plurality of sample containers 1204 within a tray 1306 is shown. The sample containers 1204 can be similar to the sample containers 104, 204, or 504 described above. Figure 13B shows an example of a bounding box annotation 1212 of the sample container 1204 of the image 1302. Figure 13C shows examples of mask annotations 1312a and 1312b of the sample container 1204 that identify different characteristics of the sample container (e.g., having or not having a cap, different cap colors, different cap types, etc.). Other annotation types may be used.

[0090] FIG. 13D shows an example of a first annotated image 1320 obtained using the mask annotation at the top of each sample container 1204 under a first imaging condition (e.g., a first illumination intensity). For example, the first imaging condition can be a condition under which the annotation generator 912 is trained such that the annotation of the first annotated image 1320 is highly accurate. FIG. 13E shows an example of a second annotated image 1322 obtained under a second imaging condition different from the first imaging condition used for the first annotated image 1320. For example, the second annotated image 1322 can be obtained using different illumination intensities, illumination spectra, sample containers and / or imaging device speeds, relative positions and angles between the imaging device 226 and the sample container 1204, imaging device exposures, imaging device lens characteristics, e.g., focal length, aperture, depth of field, etc. (represented by the light shading in FIG. 13E). As described above, in some embodiments, the second annotated image 1322 can use the annotation of the first annotated image 1320. FIG. 13F shows an example of a third annotated image 1324 obtained under a third imaging condition different from the first or second imaging condition (as represented by the intermediate shading in FIG. 13F). In some embodiments, the third annotated image 1324 can use the annotation of either the first annotated image 1320 or the second annotated image 1322. As described above, using the annotated second and / or third images 1322, 1324, the annotation generator 912 can be retrained to annotate images obtained under different imaging conditions.

[0091] Although the image capture related to the imaging device 226 has been mainly described, it will be understood that the imaging device 240 or any other suitable imaging device can be used.

[0092] As described above, the annotation generator 912 can be trained to annotate images obtained under different imaging conditions. Such annotations can enable a more accurate characterization of the sample container and can enable improved substrate handling by the substrate handler 106 and / or the robot 228. In one or more embodiments, the sample container can be identified using the images annotated by the annotation generator 912, and the robot 228 or another robot can be arranged and / or used to transport the sample container based on the images annotated by the annotation generator 912.

[0093] Although the present disclosure is susceptible to various modifications and alternative forms, specific method and apparatus embodiments have been shown by way of example in the drawings and are described in detail herein. However, it should be understood that the specific methods and apparatus disclosed herein are not intended to limit the present disclosure.

Claims

1. A method for updating the training of a sample characteristic evaluation algorithm of a diagnostic examination room system, the method comprising: providing an imaging device in the diagnostic examination room system, wherein the imaging device is controllably movable within the diagnostic examination room system; capturing a first image within the diagnostic examination room system using the imaging device, wherein the first image is captured under certain imaging conditions; performing annotation of the first image using an annotation generator of the diagnostic examination room system to generate a first annotated image; updating the training of the annotation generator using the first annotated image; the method as described above.

2. changing the imaging conditions to changed imaging conditions; capturing a second image within the diagnostic examination room system under the changed imaging conditions using the imaging device; performing annotation of the second image using the annotation generator to generate a second annotated image; updating the training of the annotation generator using the second annotated image; the method according to claim 1, further comprising the above.

3. The method according to claim 2, wherein the first image and the second image include the holding position of the sample container.

4. The method according to claim 2, wherein the first image and the second image include the sample container.

5. The method according to claim 4, wherein the method further comprises providing a robot including a gripper, wherein providing the imaging device includes attaching the imaging device to the robot, and the method further comprises gripping the sample container during the capture of the first image.

6. The method according to claim 4, wherein the imaging condition is the speed of the imaging device with respect to the sample container during imaging.

7. The method according to claim 4, wherein the imaging condition is the posture of the imaging device with respect to the sample container.

8. The method according to claim 4, wherein the imaging condition is the position of the imaging device with respect to the sample container.

9. The method according to claim 1, wherein the imaging condition is the intensity of illumination within the diagnostic examination room system.

10. The method according to claim 1, wherein the annotation is a bounding box or a pixel-wise mask of an object in the first image.

11. The method according to claim 1, wherein the annotation is one or more characteristics of the sample container in the image.

12. The method according to claim 11, wherein one or more characteristics include the orientation of the sample container relative to the holding position for the sample handler.

13. The method according to claim 11, wherein one or more characteristics include at least one of the geometric shape of at least one part of the sample container, the sample container height, the sample container diameter, the characteristics of the liquid in the sample container, and the sample container identification mark.

14. A method for training a sample characteristic evaluation algorithm of a diagnostic laboratory system, the method comprising: providing an imaging device to the diagnostic laboratory system, wherein the imaging device is controllably movable within the diagnostic laboratory system; capturing a first image of the sample container using the imaging device, wherein the first image is captured under a first imaging condition; performing annotation of the first image to generate a first annotated image; changing the first imaging condition to a second imaging condition; capturing a second image of the sample container under the second imaging condition using the imaging device; performing annotation of the second image to generate a second annotated image; training an annotation generator of the diagnostic laboratory system using at least the first annotated image and the second annotated image; changing the second imaging condition to a third imaging condition; capturing a third image of the sample container under the third imaging condition using the imaging device; performing annotation of the third image using the annotation generator to generate a third annotated image; further training the annotation generator using at least the third annotated image, wherein the method includes the above steps.

15. The method further includes providing a robot including a gripper, wherein providing the imaging device includes providing an imaging device attached to the robot, and the method further includes gripping the sample container during the capture of the first image, the second image, or the third image. The method according to claim 14.

16. The first imaging condition is a first illumination intensity for illuminating the sample container during capture of the first image, the second imaging condition is a second illumination intensity for illuminating the sample container during capture of the second image, and the third imaging condition is a third illumination intensity for illuminating the sample container during capture of the third image, the method according to claim 14.

17. The first imaging condition is a first speed of the imaging device with respect to the sample container during capture of the first image, the second imaging condition is a second speed of the imaging device with respect to the sample container during capture of the second image, and the third imaging condition is a third speed of the imaging device with respect to the sample container during capture of the third image, the method according to claim 14.

18. The first imaging condition is a first pose of the imaging device with respect to the sample container during capture of the first image, the second imaging condition is a second pose of the imaging device with respect to the sample container during capture of the second image, and the third imaging condition is a third pose of the imaging device with respect to the sample container during capture of the third image, the method according to claim 14.

19. The annotation is a bounding box or pixel-wise mask of the sample container, the method according to claim 14.

20. A diagnostic examination room system, An imaging device that is controllably movable within the diagnostic examination room system, where the imaging device is configured to capture images within the diagnostic examination room system under various imaging conditions, said imaging device; A processor coupled to the imaging device; A memory coupled to the processor, where the memory includes an annotation generator trained to annotate images captured by the imaging device, and the processor, when executed by the processor, causes the processor to Receive first image data of a first image captured by the imaging device using at least one imaging condition; Cause the annotation generator to perform an annotation of the first image to generate a first annotated image; Update the training of the annotation generator using the first annotated image; The memory further including computer program code, The diagnostic examination room system including.

Images