Apparatus and method for training sample characterization algorithm in diagnostic laboratory system

Abstract

A method of updating training of an annotation generator of a diagnostic laboratory system includes providing an imaging device in the diagnostic laboratory system, where the imaging device is controllably movable within the diagnostic laboratory system; capturing a first image within the diagnostic laboratory system using the imaging device, the first image being captured under at least one imaging condition; performing annotation of the first image using the annotation generator to generate a first annotated image; and using the first annotation image to update training of the annotation generator. Other methods and systems are also disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 368,456, filed on July 14, 2022, entitled “DEVICES AND METHODS FOR TRAININGSAMPLE CHARACTERIZATION ALGORITHMS IN DIAGNOSTIC LABORATORY SYSTEMS,” the disclosure of which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] Embodiments of the present disclosure relate to apparatus and methods for training a sample characterization algorithm in a diagnostic laboratory system. Background Art

[0004] Diagnostic laboratory systems perform clinical chemistry reactions or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. These samples may be received in sample containers in the laboratory system and / or transported throughout the laboratory system. Many laboratory systems handle a large number of sample containers and samples contained in the sample containers.

[0005] Some laboratory systems use machine vision and machine learning to facilitate sample processing and sample container identification, which can be based on characterization and / or classification of sample containers. For example, vision-based machine learning models (e.g., artificial intelligence (AI) models) have been adapted to provide fast and non-invasive methods for sample container identification and characterization. However, the training cost of using machine learning models to support new types of sample containers or new imaging conditions may be prohibitive because a large amount of training data is required to retrain or adapt the machine learning model to characterize the new type of sample container or to adapt the machine learning model to work under new imaging conditions. Therefore, there is a need for laboratory systems and methods for improving the training of machine vision systems in laboratory systems. Summary of the invention

[0006] According to a first aspect, a method for updating training of a sample characterization algorithm of a diagnostic laboratory system is provided. The method comprises: providing an imaging device in the diagnostic laboratory system, wherein the imaging device is controllably movable within the diagnostic laboratory system; using the imaging device to capture a first image within the diagnostic laboratory system, the first image being captured under an imaging condition; using an annotation generator of the diagnostic laboratory system to perform annotation of the first image to generate a first annotated image; and using the first annotated image to update training of the annotation generator.

[0007] In another aspect, a method for training a sample characterization algorithm of a diagnostic laboratory system is provided. The method includes: providing an imaging device in the diagnostic laboratory system, wherein the imaging device is controllably movable within the diagnostic laboratory system; using the imaging device to capture a first image of a sample container, the first image being captured under a first imaging condition; performing annotation of the first image to generate a first annotated image; changing the first imaging condition to a second imaging condition; using the imaging device to capture a second image of the sample container under the second imaging condition; performing the annotation of the second image to generate a second annotated image; using at least the first annotated image and the second annotated image to train an annotation generator of the diagnostic laboratory system; changing the second imaging condition to a third imaging condition; using the imaging device to capture a third image of the sample container under the third imaging condition; performing the annotation of the third image using the annotation generator to generate a third annotated image; and using at least the third annotated image to further train the annotation generator.

[0008] In another aspect, a diagnostic laboratory system is provided, comprising: (1) an imaging device capable of controllably moving within the laboratory system, wherein the imaging device is configured to capture images within the laboratory system under different imaging conditions; (2) a processor coupled to the imaging device; and (3) a memory coupled to the processor, wherein the memory comprises an annotation generator, the annotation generator being trained to annotate images captured by the imaging device, and the processor further comprising computer program code, which, when executed by the processor, causes the processor to: (a) receive first image data of a first image captured by the imaging device using at least one imaging condition; (b) cause the annotation generator to perform annotation of the first image to generate a first annotated image; and (c) use the first annotated image to update the training of the annotation generator.

[0009] Other aspects, features and advantages of the present disclosure may be apparent from the following description and illustration of several exemplary embodiments, including the best mode contemplated for implementing the present disclosure. The present disclosure may also have other and different embodiments, and may modify its several details in various aspects, all of which do not depart from the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described below are provided for illustrative purposes and are not necessarily drawn to scale. Therefore, the drawings and descriptions are to be regarded as illustrative rather than restrictive in nature. The drawings are not intended to limit the scope of the present disclosure in any way.

[0011] Figure 1Illustrated is a block diagram of a diagnostic laboratory system including a sample handler according to one or more embodiments.

[0012] Figure 2 Illustrated is a top view of the interior of a sample processor of a diagnostic laboratory system in accordance with one or more embodiments.

[0013] Figures 3A-3C Different types of sample containers including caps secured to tubes that may be used within a diagnostic laboratory system are illustrated in accordance with one or more embodiments.

[0014] Figures 4A-4C Illustrated are tubes of different types of sample containers that may be used within a diagnostic laboratory system in accordance with one or more embodiments.

[0015] Figure 5 Illustrated is a perspective view of a robot in a sample handler of a diagnostic laboratory system coupled to a gantry configured to move the robot and attached imaging device along x, y, and z axes in accordance with one or more embodiments.

[0016] Figure 6 The present invention illustrates a method according to one or more embodiments. Figure 5 A side view of the robot wherein the imaging device operates to capture an image of a sample container.

[0017] Figure 7 The present invention illustrates a method according to one or more embodiments. Figure 6 A side view of a robot in which the gripper is pivotally coupled to the main structure of the robot.

[0018] Figure 8 A flow diagram is illustrated of a sample container characterization workflow that may be implemented in a characterization algorithm of a diagnostic laboratory system in accordance with one or more embodiments.

[0019] Fig. 9 Illustrated is a workflow for generating image data with variations for training a characterization algorithm of a diagnostic laboratory system in accordance with one or more embodiments.

[0020] Fig.10 A flow diagram is illustrated of another exemplary method of updating the training of an annotation generator of a diagnostic laboratory system in accordance with one or more embodiments.

[0021] Fig.11 A flow diagram for training an annotation generator of a diagnostic laboratory system is illustrated in accordance with one or more embodiments.

[0022] Figures 12A-12I Illustrative images and image annotations are illustrated in accordance with one or more embodiments.

[0023] Figures 13A-13F Additional exemplary images and image annotations are illustrated in accordance with one or more embodiments. DETAILED DESCRIPTION

[0024] As described, the diagnostic laboratory system performs clinical chemistry reactions and / or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. These samples are collected in sample containers and then sent to the diagnostic laboratory system. The sample containers are then loaded into the sample processor of the laboratory system. The sample containers are then transferred to sample carriers by robots, wherein the sample carriers transport the sample containers to the instruments and components of the laboratory system, where the samples are processed and analyzed.

[0025] The diagnostic laboratory system may use a vision system to capture images of sample containers and / or the contents of the sample containers (e.g., biological samples). The captured images are then used to identify the sample containers and / or the contents of the sample containers. For example, the diagnostic laboratory system may include a vision-based AI model configured to provide a fast and non-invasive method for characterizing or classifying sample containers. The AI ​​model may be trained to annotate images of different types of sample containers and variations of tube portions and / or caps of sample containers. The annotated images may then be used for sample identification purposes.

[0026] As new types of sample containers are introduced into the diagnostic laboratory system, the AI ​​model employed must be updated or "trained" to be able to annotate the new sample container types. Retraining AI models in conventional diagnostic laboratory systems is expensive and time consuming because multiple different types of sample containers need to be imaged and manually annotated in order to retrain the AI ​​model. The AI ​​models used in machine vision systems are typically trained using images of samples and / or sample containers captured under ideal conditions, such as in a studio-like environment with ideal imaging conditions. Capturing images under these ideal conditions is expensive and time consuming. In addition, these ideal conditions rarely exist within the confines of a "deployed" laboratory system. Therefore, due to the differences between the images used to train the machine vision system and the actual images captured during use of the machine vision system within the deployed diagnostic laboratory system, the machine vision system may not be accurately trained. Therefore, there is a need for systems and methods for improving the training of machine vision systems in diagnostic laboratory systems.

[0027] Embodiments of the systems and methods described herein overcome the problem of training sample container recognition and classification AI models by capturing training images of sample containers under actual conditions within a deployed laboratory system, and in some cases automatically annotating these training images. The training images can be used to train or retrain AI models (e.g., annotation generators) in diagnostic laboratory systems, etc.

[0028] In some embodiments, the diagnostic laboratory systems and methods disclosed herein use a robot for moving and / or placing sample containers. An imaging device is coupled to the robot and can be used to capture training images of sample containers within the diagnostic laboratory system. The use of the robot enables specific movement between the imaging device and the sample container, so that the image of the sample container can include predetermined changes between images. The changes between images can include different postures, lighting intensities, lighting spectra, exposure times, and other imaging conditions. Therefore, a large number and different training images can be obtained within the deployed diagnostic laboratory system.

[0029] In addition, in some embodiments, once an image is annotated, the annotated image can be used to retrain how to annotate future images. That is, a first set of images taken under a first set of conditions (e.g., lighting, posture, motion profile, etc.) can be annotated and then used to train the AI ​​model of the laboratory system on how to annotate a subsequent second set of captured images taken under a different second set of conditions (e.g., different lighting, posture, motion profile, etc.). For example, an AI for annotating images (referred to herein as a "annotation generator" for convenience) can be trained to annotate well-lit sample containers imaged using an imaging device capable of controllable movement (e.g., an imaging device fixed to a robot). Once the annotation generator is trained using images of well-lit sample containers, a first set of images of a tray of sample containers can be acquired under the same well-lit conditions, and the annotation generator can annotate the first set of images. Because the annotation generator is trained using images under well-lit conditions, the annotations of the first set of images should be accurate. Thereafter, the lighting conditions can be reduced (e.g., reduced to half intensity or another reduced intensity), and a second set of images can be captured by the imaging device. The precise control provided by the robot allows the imaging device to be positioned at exactly the same viewing position to capture a second set of images of the tray of exactly the same sample containers. Since all conditions except the illumination intensity are the same during the capture of the first and second image sets, the annotations used for the first set of images can be used as the annotations for the second set of images. Using the annotations and the second set of images, the annotation generator can be improved (e.g., retrained) to annotate images captured under reduced illumination conditions (e.g., half intensity). In this way, the annotation generator itself can be part of the sample characterization algorithm and can be iteratively trained to handle more and more variations. That is, the controllable movement of the imaging device allows the annotations of the first set of images captured under the first set of conditions to be used to annotate the second set of images captured under the second set of conditions (e.g., different illumination intensity, different illumination spectrum, different motion profile, different sample container position, etc.). The above process can be used to train the annotation generator to annotate images of sample containers, sample container holders, or other image features using imaging conditions that vary greatly within an actual deployed diagnostic laboratory system.

[0030] Reference below Figure 1-13F These and other systems and methods are described in further detail below.

[0031] Reference now Figure 1 , which illustrates a block diagram of an exemplary embodiment of a diagnostic laboratory system 100. The laboratory system 100 may include a plurality of instruments 102 configured to process sample containers 104 (some labeled) and perform assays or tests on samples located in the sample containers 104. The laboratory system 100 may have a first instrument 102A and a second instrument 102B. Other embodiments of the laboratory system 100 may include more or fewer instruments.

[0032] The samples located in the sample container 104 can be various biological specimens collected from individuals, such as patients being evaluated by medical professionals. These samples can be collected from patients and placed directly into the sample container 104. The sample container 104 can then be delivered to the laboratory system 100. As described in more detail below, the sample container 104 can be loaded into a sample processing machine 106, which can be an instrument of the laboratory system 100. The sample container 104 can be transferred from the sample processing machine 106 to a sample carrier 112 (some are labeled), which transports the sample container 104 throughout the laboratory system 100, such as to the instrument 102, by means of a track 114.

[0033] The track 114 is configured to enable the sample carrier 112 to move throughout the laboratory system 100, including to and from the sample processor 106. For example, the track 114 may extend near or around at least some of the instruments 102 and the sample processor 106, such as Figure 1 The instrument 102 and the sample processing machine 106 may have a robot ( Figure 1 104 to and from the sample carrier 112. The track 114 may include a plurality of segments 120 (some labeled) that may be interconnected. In some embodiments, some of the segments 120 may be integrated with one or more of the instruments 102.

[0034] Components of the laboratory system 100, such as the sample processor 106 and the instrument 102, may include or be coupled to a computer 130 configured to execute one or more programs that control the components of the laboratory system 100, including the sample processor 106. The computer 130 may be configured to communicate with the instrument 102, the sample processor 106, and other components of the laboratory system 100. The computer 130 may include a processor 132 configured to execute programs, including programs other than those described herein. The programs may be implemented in computer code.

[0035] The computer 130 may include or have access to a memory 134 that may store one or more programs and / or data described herein. The memory 134 and / or the programs stored therein may be referred to as non-transitory computer-readable media. These programs may be computer code that may be executed on or by the processor 132. The memory 134 may include a robot controller 136 (e.g., computer code that may be executed by the processor 132) that is configured to generate instructions to control the robot and / or similar devices in the instrument 102 and the sample processing machine 106. As described herein, the instructions generated by the robot controller 136 may be in response to data, such as image data received from the sample processing machine 106.

[0036] The memory 134 may also store a sample characterization algorithm 138 (e.g., a classification algorithm or other suitable computer code) configured to identify and / or classify other items in the sample container 104 and / or the sample processing machine 106. In some embodiments, the characterization algorithm 138 classifies objects in image data generated by an imaging device as described herein. The characterization algorithm 138 may include a training model, such as one or more neural networks. In some embodiments, the characterization algorithm 138 may include a label generator (912- Fig. 9 ), which is configured to annotate images captured by the imaging device. The characterization algorithm 138 may also include a convolutional neural network (CNN) trained to characterize or recognize objects in the image data. The training model is implemented using artificial intelligence (AI). Thus, the training model can learn to classify sample containers 104, as described herein. Note that the characterization algorithm 138 is not a lookup table, but rather a supervised or unsupervised model trained to characterize and / or recognize various types of sample containers 104.

[0037] The characterization algorithm 138 may also include one or more algorithms that train an AI (e.g., a neural network or other AI model) for labeling, classifying, and / or identifying the sample container 104. The AI ​​may be based on at least one imaging device ( Figure 1 Not shown, see for example Figure 6 The robot controller 136 may be trained using training images captured by the cameras 636, 638 of the apparatus 102. In some embodiments, the training images may be captured within the sample processor 106. There may be relative movement between the imaging device and the sample container. For example, a robot located in one or more of the instrument 102 and / or the sample processor 106 may be configured to move the imaging device relative to the sample container 104. Additionally, the robot may be configured to move the sample container 104 relative to the imaging device. During training, the characterization algorithm 138 may instruct the robot controller 136 to generate instructions to move the robot to a specific location to capture a specific image of the sample container 104.

[0038] The imaging controller 139 may be implemented in the computer 130. For example, the imaging controller 139 may be computer code stored in the memory 134 and executed by the processor 132. The imaging controller 139 may be configured to control the imaging devices (e.g., the imaging devices 226, 240- Figure 2 ) and illumination sources (e.g., illumination sources 642, 652- Figure 6 For example, the imaging controller 139 may control the cameras (eg, cameras 636, 638- Figure 6 ), for example, by setting a predetermined frame rate and exposure time during imaging. The imaging controller 139 may also set the intensity and spectrum of illumination used to illuminate the sample container 104 during imaging.

[0039] The computer 130 may be coupled to a workstation 140 configured to enable a user to interact with the laboratory system 100. The workstation 140 may include a display 142, a keyboard 144, and other peripheral devices (not shown). Data generated by the computer 130 may be displayed on the display 142. In some embodiments, the data may include an abnormality warning detected by the characterization algorithm 138. The abnormality may include a notification that some of the sample containers 104 cannot be characterized. In addition, the user may input data into the computer 130 with the aid of the workstation 140. For example, the data input by the user may be an instruction that causes the robot controller 136, the characterization algorithm 138, or the imaging controller 139 to perform certain operations, such as capturing and / or analyzing images of the sample container 104. Other data input by the user may include annotations of training images used during the training of the characterization algorithm 138.

[0040] Now refer to Figure 2 , which illustrates a top view of the interior of a sample processing machine 106 according to one or more embodiments. The sample processing machine 106 is configured to capture images of sample containers 104 and transport sample containers 104 between holding locations 210 (some are labeled) and sample carriers 112. Figure 2 In some embodiments, the holding position 210 is located within a tray 212 that is removable from the sample processing machine 106. The sample processing machine 106 may include a plurality of slides 214 configured to hold the tray 212. In some embodiments, the sample processing machine 106 may include four slides 214, which are referred to as a first slide 214A, a second slide 214B, a third slide 214C, and a fourth slide 214D. The third slide 214C is shown as being partially removed from the sample processing machine 106, which may occur during replacement of the tray 212. Other embodiments of the sample processing machine 106 may include more than Figure 2 Fewer or more sliding devices as shown in .

[0041] Each slide 214 can be configured to hold one or more trays 212. Figure 2 In an embodiment of the present invention, the slide 214 may include a receiver 216 configured to receive the tray 212. Each tray 212 may include a plurality of holding positions 210, wherein each holding position 210 may be configured to receive one of the sample containers 104. Figure 2 In the embodiment of the present invention, the tray can vary in size to include a large tray having twenty-four holding positions 210 and a small tray having eight holding positions 210. Other configurations of the tray 212 can include different numbers of holding positions 210 and holding positions configured to hold more than one sample container.

[0042] In some embodiments, the sample handler 106 may include one or more slide sensors 220 configured to sense movement of one or more of the slides 214. The slide sensors 220 may generate signals indicative of slide movement, wherein the signals may be received and / or processed by the robotic controller 136 as described herein. Figure 2 In an embodiment of the present invention, the sample processing machine 106 includes four slide sensors 220, which are arranged so that each slide 214 is associated with one of the slide sensors 220. The first slide sensor 220A senses the movement of the first slide 214A, the second slide sensor 220B senses the movement of the second slide 214B, the third slide sensor 220C senses the movement of the third slide 214C, and the fourth slide sensor 220D senses the movement of the fourth slide 214D. The slide sensors 220 can use various technologies to sense the movement of the slide 214. In some embodiments, the slide sensor 220 may include a mechanical switch that switches when the slide 214 moves, wherein the switching generates an electrical signal indicating that the slide has moved. In other embodiments, the slide sensor 220 may include an optical sensor that generates an electrical signal in response to the movement of the slide 214. In still other embodiments, the slide sensor 220 may be an imaging device that generates image data of the sample container 104 when the slide 214 moves.

[0043] The sample processing machine 106 can receive many different types of sample containers 104. A first type of sample container 104 is represented by a triangle, a second type of sample container 104 is represented by a square, and a third type of sample container 104 is represented by a circle. The characterization algorithm 138 is configured to classify the sample container 104 so that the sample container 104 can be identified by the computer 130 ( Figure 1 ) can be easily identified. The characterization algorithm 138 can also characterize new types of sample containers (e.g., sample container 204) as described herein.

[0044] The sample processing machine 106 includes a sample container 204 (marked with a cross) that is a new type or has not yet been classified by the characterization algorithm 138. Figure 2 In an embodiment of the present invention, the sample container 204 is placed into the tray 212A, which can be designated to hold sample containers of the new type. For example, upon determining that the sample container 204 is in the tray 212A, the computer 130 can determine whether the sample container 204 is of the new type. If the sample container 204 is of the new type, the computer 130 can cause the characterization algorithm 138 to classify or characterize the sample container 204, as described herein.

[0045] In some embodiments, tray 212A may have a label 205 indicating that tray 212A contains a new type of sample container 204. A user may load sample containers 204 into tray 212A and insert tray 212A into sample processing machine 106. An imaging device may then capture an image of label 205. Computer 130 may then cause characterization algorithm 138 to classify sample container 204 in response to detecting label 205. In other embodiments, a user may select a sample container 204 from a workstation 140 ( Figure 1 ) indicates that the sample container 204 has been received in the sample processing machine 106. In some embodiments, the user can indicate the position of the sample container 204 in the sample processing machine 106.

[0046] Now refer to Figures 3A-3C , which illustrates different types of exemplary sample containers that can be used within the laboratory system 100. Other types of sample containers may also be used. In some embodiments, the sample container includes a tube with or without a cap attached to the tube. The sample container may also include a sample or other contents (e.g., a liquid) located in the sample container. Also refer to Figures 4A-4C , which shows the uncapped Figures 3A-3C As shown in the figures, all sample containers may have different configurations or geometries. For example, the caps and tubes of different sample container types may each have different features, such as different tube and cap geometries and / or colors. The characterization algorithm 138 ( Figure 1 ) to classify and identify unique features of sample containers. The features described herein can also be used to train a characterization algorithm 138 (described below).

[0047] Figure 3A The exemplary sample container 330 of includes a cap 330A which is white with a red stripe and has an extended vertical portion. The cap 330A fits over a tube 330B. The sample container 330 has a height H31. Figure 4AThe tube 330B is illustrated without the cap 330A. The tube 330B has a tube geometry including a height H41 and a width W41. The tube 330B may have a tube color, a tube material, and / or tube surface properties (e.g., reflectivity). These dimensions, dimensional ratios, and other properties may be referred to as features and may be used during classification by the characterization algorithm 138 to classify and / or identify the sample container 330.

[0048] Figure 3B The exemplary sample container 332 of includes a cap 332A that is blue, has a dome-shaped top, and fits over a tube 332B. The sample container 332 has a height H32. Figure 4B The tube 332B is illustrated without the cap 332A. The tube 332B may have a tube geometry including a height H42 and a width W42. The tube 332B may also have tube color, tube material, and / or tube surface properties. These dimensions, dimensional ratios, and other properties may be referred to as features and may be used during classification by the characterization algorithm 138 to classify and / or identify the sample container 332.

[0049] Figure 3C The exemplary sample container 334 of includes a cap 334A, which is red and gray, has a flat top, and fits over a tube 334B. The sample container 334 has a height H33. Figure 4C The tube 334B is illustrated without the cap 334A. The tube 334B may also have a tube geometry including a height H43 and a width W43. The tube 334B may have tube color, tube material, and / or tube surface properties. These dimensions, dimensional ratios, and other properties may be referred to as features and may be used during classification by the characterization algorithm 138 to classify and / or identify the sample container 332.

[0050] Tube 330B has an identification mark in the form of barcode 330C, and tube 334B has an identification mark in the form of barcode 334C. Images of barcode 330C and barcode 334C may be analyzed by characterization algorithm 138 for classification purposes, as described herein. These barcodes may be referred to as features and may be used to train characterization algorithm 138 (described below).

[0051] Different types of sample containers may have different characteristics, such as different sizes, different surface properties and different chemical additives therein, e.g. Figures 3A-3C330, 332 and 334 of the sample container. For example, some sample container types are chemically active, which means that the sample container contains one or more additive chemicals that are used to change or maintain the state of the sample stored therein, or otherwise assist in sample processing by the instrument 102. In some embodiments, the inner wall of the tube may be coated with the one or more additives, or the additives may be provided elsewhere in the sample container. In some embodiments, the type of additive contained in the tube can be a serum separator, a coagulant (e.g., thrombin), an anticoagulant (e.g., EDTA or sodium citrate), an anti-sugar additive, or other additives for changing or retaining sample characteristics. For example, the sample container manufacturer may associate the color of the cap on the tube and / or the shape of the tube or cap with a specific type of chemical additive contained in the sample container.

[0052] Different manufacturers may have their own standards for associating the characteristics of sample containers, such as cap color, cap shape (e.g., cap geometry), and tube shape, with specific attributes of the sample container. For example, the characteristics may be related to the contents of the sample container, or may be related to whether the sample container is provided with vacuum capability. In some embodiments, the manufacturer may associate all sample containers with gray caps with tubes containing potassium oxalate and sodium fluoride configured to test glucose and lactate. Sample containers with green caps may contain heparin for static electrolytes (e.g., sodium, potassium, chloride, and bicarbonate). Sample containers with lavender caps may identify tubes containing EDTA (ethylenediaminetetraacetic acid-an anticoagulant) configured to test CBC with differential, HgBAlc, and parathyroid hormone. Other cap colors, such as red, yellow, light blue, royal blue, pink, orange, and black, may be used to indicate other additives or the lack of additives. In other embodiments, a combination of cap colors may be used, such as yellow and lavender to indicate a combination of EDTA and a gel separating agent, or green and yellow to indicate lithium heparin and a gel separating agent.

[0053] The laboratory system 100 can use sample container characteristics to further process the sample container 104 and / or the sample contained in the sample container 104. Because the sample container 104 may be chemically active and affect the testing of the sample stored therein, it is important to associate specific tests that can be performed on the sample with a specific sample container type. Thus, the laboratory system 100 can confirm that the test run on the sample in the sample container 104 is correct by analyzing the color and / or shape of the cap and / or tube. Other container characteristics may also be analyzed.

[0054] Reference again Figure 2, the sample processing machine 106 may include an imaging device 226 that is movable throughout the sample processing machine 106. Figure 2 In some embodiments, the imaging device 226 is secured to a robot 228 that can move along an x-axis (e.g., in the x-direction) and a y-axis (e.g., in the y-direction) throughout the sample handler 106. In some embodiments, the imaging device 226 can be integrated with the robot 228. In one or more embodiments, the robot 228 can additionally move along a z-axis (e.g., in the z-direction), i.e., into and out of the page. In other embodiments, the robot 228 can include one or more components that move the imaging device 226 in the z-direction (e.g., in the z-direction). Figure 2 not shown).

[0055] In some embodiments, the robot 228 may receive a signal from the robot controller 136 ( Figure 1 ) generated movement instructions. These instructions may be data indicating the x, y, and z positions to which the robot 228 should move. In other embodiments, these instructions may be electrical signals that cause the robot 228 to move in the x, y, and z directions. For example, the robot controller 136 may generate instructions to move the robot 228 in response to one or more of the slide sensors 220 detecting movement of one or more of the slides 214. These instructions may cause the robot 228 to move while the imaging device 226 captures an image of the newly added sample container 204.

[0056] Imaging device 226 includes one or more cameras ( Figure 2 Not shown; see for example Figure 6 The camera 636, 638 of the sample processing machine 106 may be configured to capture images of the sample container 104, 204 and / or other locations or objects in the sample processing machine 106. For example, the images may be of the top and / or side of the sample container 104, 204. In some embodiments, the robot 228 may be a gripper robot that grips the sample container 104, 204 and transfers the sample container 104, 204 between the holding position 210 and the sample carrier 112. In such an embodiment, the images may be captured while the robot 228 grips the sample container 104, 204, as described herein.

[0057] Additional references Figure 5, which is a perspective view of an embodiment of a robot 228 coupled to a rack 530 that is configured to move the robot 228 in the x-direction, the y-direction, and the z-direction. The rack 530 may include two y slides 532 that enable the robot 228 to move in the y-direction, an x ​​slide 534 that enables the robot 228 to move in the x-direction, and a z slide 536 that enables the robot 228 to move in the z-direction. In some embodiments, movement in these three directions may be simultaneous and may be controlled by the robot controller 136 ( Figure 1 ) generated by the robot controller 136. For example, the robot controller 136 may generate instructions that cause a motor (not shown) coupled to the frame 530 to move the slide so as to move the robot 228 and the imaging device 226 to a predetermined position or in a predetermined direction.

[0058] In some embodiments, the robot 228 may include a gripper 540 (eg, an end effector) configured to grip the sample container 504. The sample container 504 may be one of the sample containers 104 or Figures 3A-3C The robot 228 moves to a position above the holding position and then moves in the z direction to retrieve the sample container 504 from the holding position. The gripper 540 opens and the robot 228 moves downward in the z direction so that the gripper 540 extends above the sample container 504. The gripper 540 closes to grasp the sample container 504, and the robot 228 moves upward in the z direction to extract the sample container 504 from the holding position. Figure 5 As shown in FIG. 1 , the imaging device 226 can be fixed to the robot 228 so that the imaging device 226 can move with the robot 228 and capture the sample container 504 and other sample containers 104, 204 ( Figure 2 ). The imaging device 226 includes at least one camera configured to capture images, wherein the captured images are converted into image data for processing, for example, by the characterization algorithm 138. The image data can be used to train the characterization algorithm 138. In some embodiments, the image data can train the annotation generator 912 ( Fig. 9 ) or update the training of the annotation generator 912.

[0059] Additional references Figure 6 , which is a side view of an embodiment of the robot 228 grasping the sample container 504 using the gripper 540 while the sample container 504 is being imaged by the imaging device 226. Figure 6The imaging device 226 depicted in FIG. 2 may include a first camera 636 and a second camera 638. Other embodiments of the imaging device 226 may include a single camera or more than two cameras. The first camera 636 has a field of view 640 extending at least partially along the y direction and may be configured to capture an image of a sample container 504 being gripped by the gripper 540. The first illumination source 642 may illuminate the sample container 504 in the field of view 640 with the aid of an illumination field 644. In some embodiments, the spectrum and / or intensity of light emitted by the first illumination source 642 may be determined by the characterization algorithm 138 ( Figure 1 ) and / or imaging controller 139( Figure 1 In other embodiments, the imaging controller 139 is configured to control at least one of the intensity of the first illumination source 642 and the spectrum of light emitted by the first illumination source 642.

[0060] Second camera 638 may have a field of view 650 extending in the z-direction and may capture images of tray 212, sample containers 104, 204 located in tray 212, and other objects in sample handler 106. Second illumination source 652 may illuminate objects in field of view 650 via illumination field 654. In some embodiments, the spectrum and / or intensity of light emitted by second illumination source 652 may be controlled by imaging controller 139. Field of view 650 and illumination field 654 enable the capture of images of tops (e.g., caps) of sample containers 104, 204, such as Figure 2 The captured image can be represented by the characterization algorithm 138 ( Figure 1 ) analysis to classify or identify the sample containers 104, 204 and / or determine if there are any anomalies in the sample handler 106. In some embodiments, the imaging device 226 may have a single camera having a field of view that can capture at least a portion of the sample handler 106 and one or more of the holding locations 210 with or without a sample container 104, 204 positioned therein.

[0061] In some embodiments, images may be captured as the robot 228 moves the imaging device 226 relative to the sample container 104, 204. The robot controller 136 ( Figure 1 ) can set the speed and direction of the robot 228 relative to the sample container 104, 204 during image capture.

[0062] The operation of the first camera 636, the second camera 638, the first illumination source 642, and / or the second illumination source 652 may be controlled by the imaging controller 139 ( Figure 1) control. The imaging controller 139 may set one or more imaging conditions for these devices during imaging as described herein. For example, the imaging controller 139 may set the exposure time, frame rate, illumination intensity, and / or illumination spectrum during image capture. In some embodiments, the characterization algorithm 138 may determine the imaging conditions. Further images may be captured under a second imaging condition or an altered imaging condition.

[0063] The images captured by the imaging device 226 may be analyzed by the characterization algorithm 138 to determine characteristics of the sample container 504, the robot 228, the sample containers 104, 204, and other components in the sample processing machine 106, as described herein. For example, the characterization algorithm 138 may characterize or identify the container type of the sample container 104, 204, 504. When analyzing image data generated by the first camera 636, the characterization algorithm 138 may analyze a side view of the sample container 504. The characterization algorithm 138 may also determine whether the sample container 504 is properly gripped by the gripper 540. When analyzing image data generated by the second camera 638, the top or cap of the sample container 104, 204 may be characterized. The images generated during different views may also be used to train the annotation generator 912 or update the training of the annotation generator 912 (as described below with reference to FIG. 1 ). Fig. 9 described above).

[0064] Additional references Figure 7 , which is Figure 6 228, wherein the gripper 540 is pivotally coupled to the main structure 752 of the robot 228. This embodiment of the robot 228 includes a secondary arm 754 coupled to the main structure 752 by a pivot mechanism 756, which enables the secondary arm 754 to rotate along an arc R relative to the main structure 752. Figure 7 In an embodiment of the invention, the gripper 540 is coupled to the secondary arm 754 and the imaging device 226 is coupled to the main structure 752 so that the sample container 504 can pivot relative to the imaging device 226, which enables images of the sample container 504 in different postures (e.g., tilted) when captured. In some embodiments, the pivot mechanism 756 enables the secondary arm 754 to pivot in directions other than arc R, such as in directions into and out of the paper. The characterization algorithm 138 can determine the posture of the sample container 504 relative to the imaging device 226, and the robot controller 136 can generate instructions to move the robot 228 to the correct posture. The images generated during the different postures can be used to train the annotation generator 912 ( Fig. 9 ) or update the training of the annotation generator 912.

[0065] Reference again Figure 2 and Figure 5In some embodiments, the sample handler 106 includes a fixed imaging device 240 that can be in a fixed position. In such embodiments, the robot 228 can move the sample container 104, 204 to the vicinity of the imaging device 240, where the imaging device 240 can then capture images of the sample container 104, 204. The images generated by the imaging device 240 can be processed as described herein, such as by the characterization algorithm 138. The imaging device 240 can include a camera 242 and an illumination source 244, wherein the illumination source 244 can be configured to illuminate an object imaged by the camera 242. In some embodiments, the spectrum and / or intensity of light emitted by the illumination source 244 can be controlled by the characterization algorithm 138 and / or the imaging controller 139.

[0066] Having described an exemplary embodiment of a laboratory system 100, a method of processing image data generated by the laboratory system 100 will now be described. The laboratory system 100 and methods described herein generate data and annotations with real-world variations by generating image data using a combination of an imaging device 226 and a robot 228. Embodiments are applicable to sample container characterization, wherein the characterization may include characterizing sample containers 104, 204, and / or 504 with and without caps and samples contained in the sample containers 104, 204, and / or 504. The characterization may include annotations as described herein.

[0067] Additional references Figure 8 , which is a flow chart of an exemplary sample container characterization workflow 800 that may be implemented in the characterization algorithm 138 and executed by the processor 132. In some embodiments, the robot 228 may grasp the sample container 504 ( Figure 5 ), and the imaging device 226 may capture an image of the sample container 504. In other embodiments, the robot 228 may move the imaging device 226 relative to the sample containers 104, 204 so that the imaging device 226 may capture an image of a particular sample container in the sample containers 104, 204. During image capture, the illumination source 642 or 652 ( Figure 6 and Figure 7) can illuminate the sample container 104, 204, 504 using a predetermined intensity and spectrum of light (e.g., full intensity, half intensity, etc., white light, red light, green light, blue light, etc.). In some embodiments, the predetermined intensity and spectrum of light can be determined by the characterization algorithm 138. The imaging device 226 can then capture images of the sample container 104, 204, 504 under these lighting conditions. The imaging controller 139 can set other lighting conditions during image capture. Images can be captured using one or both of the first camera 636 and the second camera 638 in the imaging device 226. Thus, the image can include the sample container 104, 204 and / or the top of the sample container 504 gripped by the gripper 540. The sample container 504 can have different postures relative to the imaging device 226 (e.g., using the pivot mechanism 756).

[0068] Image data may be received at operation block 802, where preprocessing, such as deblurring, gamma correction, and radial distortion correction, may be performed prior to further processing. A data-driven machine learning approach, such as a generative adversarial network (GAN) or another suitable AI network, may be used for preprocessing at operation block 802. Processing may continue to operation block 804, where the image of the sample container 504 may undergo sample container localization and classification. (This process is described for sample container 504, but is also applicable to sample containers 104, 204.) Localization may be an annotation of the image of the sample container 504 to specify the location of the sample container 504 within each image, and, for example, may include surrounding the image of the sample container 504 with a virtual box (e.g., a bounding box or a pixel mask) to isolate the sample container 504. Classification may be performed using a data-driven machine learning-based approach, such as a convolutional neural network (CNN).

[0069] CNN can be enhanced using YOLOv4 or other image recognition networks or models. YOLOv4 is a real-time object detection model that works by splitting the object recognition task into two operations. The first operation uses regression to identify the object location through a bounding box, and the second operation uses classification to determine the class of the object (e.g., sample container 104, 204, or 504). The location can provide a bounding box for the detected sample container. The classification determines the high-level characteristics of the sample container, such as determining whether the sample container has a cap, no cap, or a tube top sample cup (TTSC). In some embodiments, the classification also determines the classification confidence.

[0070] Processing may continue to sample container tracking at operation block 806, where, for each newly detected sample container, the computer 130 (e.g., via the robot controller 136 and / or the characterization algorithm 138) may assign a new trajectory segment identification (e.g., an identification of a portion of the path traveled by the sample container, such as an identification of a portion of the track 114) to each sample container. Alternatively, the computer 130 may attempt to associate the detected sample container with an existing trajectory segment established in a previous image based on the overlap area between the detected bounding box and the predicted bounding box established on the motion trajectory, the classification confidence, and other features derived from the appearance of the image of the sample container. In situations where a detection may be missed and this prevents tracking, a more complex data association algorithm, such as the Hungarian algorithm, may be utilized to ensure the robustness of the tracking. In some embodiments, Deep SORT or other machine learning algorithms may be used for sample container tracking.

[0071] When a trajectory segment contains enough observations collected across multiple images (e.g., frames), the characterization algorithm 138 may begin estimating more detailed characteristics at operation block 808. The characteristics include, but are not limited to, sample container height and sample container diameter, cap color, cap shape, and bar code readings when a bar code or other sample container identification mark is in the field of view of the imaging device 226 or the imaging device 240.

[0072] Training data-driven machine learning algorithms, software models, and networks may require collecting image data under various controlled (e.g., predetermined) conditions. The varying conditions may include different sample container types, lighting conditions (e.g., lighting intensity, lighting spectrum, etc.), camera spectral properties, exposure time, sample container distance and / or posture, relative motion between imaging device 226 and sample container, etc.

[0073] Fig. 9 is a diagram of an exemplary workflow 900 for generating image data with different conditions according to one or more embodiments described herein. Fig. 9 , the coordinator 902 is configured to guide the workflow 900. In some embodiments, the coordinator 902 may be implemented as a memory stored in the memory 134 ( Figure 1) and executed by the processor 132. For example, the coordinator 902 can be implemented in the characterization algorithm 138. The coordinator 902 can be coupled to: the robot controller 136 for controlling the robot 228; the lighting controller 906 for controlling the operation of the lighting sources 642 and 652; the imaging controller 139 for controlling the operation of the imaging devices 226 and 240; and the annotation generator 912, which annotates the captured images as described below. Specifically, the coordinator 902 can control the workflow 900 to generate image data of the sample container by: (1) using the robot controller 136 to position the robot 228 and the imaging device 226 coupled thereto; (2) using the lighting controller 906 to illuminate the sample container using the illumination source 642 and / or 652 (e.g., using a desired illumination intensity, illumination spectrum, etc.); (3) using the imaging controller 139 to instruct the imaging device 226 and / or 240 to capture an image of the sample container (e.g., image data 914) (e.g., using a desired exposure time or other imaging parameters); and (4) using the annotation generator 912 to generate an annotated captured image (e.g., annotated image data 916). As will be further described below, the coordinator 902 can also instruct the annotation generator 912 to store annotations for one or more images and reuse the stored annotations on one or more subsequently captured images. In addition, in some embodiments, the coordinator 902 can direct the retraining of the annotation generator 912.

[0074] For example, the location of one or more of the sample containers 104, 204, or 504 to be characterized may be stored in at least one of the robot controller 136 or the characterization algorithm 138. In some embodiments, characterization includes annotating or identifying an image of the sample container. For example, the sample container 204 to be characterized may be located on the tray 212A ( Figure 2 ). The robot controller 136 may generate signals or instructions that cause the robot 228 to retrieve the sample containers 204 and position each of the sample containers in a specific position or orientation (e.g., posture) relative to the imaging device 226 (or the imaging device 240) so that the imaging device can capture an image of the sample container (e.g., the sample container 204). For example, the specific position may include a predetermined distance from the imaging device 226 (or 240), a predetermined posture (e.g., angle) relative to the imaging device 226 (or 240), and relative movement between the sample container and the imaging device 226 (or 240) during imaging.

[0075] The illumination controller 906 manages the illumination intensity and / or spectrum of the illumination sources 642, 652. In some embodiments, the illumination controller 906 may be implemented in the imaging controller 139. The characterization algorithm 138 may generate instructions or imaging requirements, which may be interpreted by the illumination controller 906 to generate instructions for controlling the illumination sources 642, 652 and / or the cameras 636, 638. The imaging controller 139 may instruct the cameras 636, 638 to generate image data 914, which may be digital data representing a captured image of the sample container 204 under the illumination conditions established by the illumination controller 906.

[0076] The annotation generator 912 can identify objects in the image and mark the objects. During some annotation processes, a bounding box may be generated within the image, wherein the bounding box includes one or more objects to be identified. These objects can be identified or classified into categories or instances. For example, the annotation generator 912 can be used to identify a sample container as a class of objects in an image. In other embodiments, segmentation can be used to identify specific instances, such as the type of sample container in an image. The annotation generator 912 can use tools other than bounding boxes. For example, the annotation generator 912 can use polygon segmentation, semantic segmentation, 3D cuboids, key points and landmarks, or lines and splines. Annotations can then be used to create a training data set for sample container identification. In some embodiments, the annotation generator 912 may include a deep learning network, such as a general convolutional neural network (CNN). Exemplary networks include Inception, ResNet, ResNeXt, DenseNet, etc., but other CNN and / or AI architectures may also be used. Reference below Fig.10 The training of annotation generator 912 is further described.

[0077] The annotation generator 912 may generate predicted annotations for an image of a sample container represented by the image data 914 by utilizing previously annotated data, as further described herein. The annotation generator 912 generates annotated image data 916 from the image data 914. The annotated image data 916 may then be fed back to the annotation generator 912 for further annotation and / or further training of the annotation generator 912. In some embodiments, a first annotation of one or more objects in a first image may be performed, followed by reusing the first annotation on a second image while performing a second annotation of the one or more objects in the second image. For example, the lighting intensity, lighting spectrum, pose, or another condition may change between the capture of the first and second images, but the annotations for the first image may be used to annotate the second image, as described below with reference to Fig.11As further described (eg, if the sample container is expected to be in the same position in both images due to precise positioning of the imaging device 226 by the robot 228), the annotation generator 912 may be trained by this process, or the annotation generator 912 may have its training updated by this process.

[0078] Image annotation, as performed by annotation generator 912, may include the task of annotating images of sample containers with labels. In some embodiments, some annotations may additionally involve human-driven tasks. Labels may be predetermined during programming of machine learning and selected to give a computer vision model (e.g., characterization algorithm 138) information about objects in the image. Exemplary considerations during annotation include possible naming and classification issues, representing occluded objects (e.g., occluded tubes or occluded samples stored in tubes), marking unrecognizable portions of the image, and other considerations.

[0079] Labeling an image of a sample container 104, 204, or 504 by the label generator 912 may include applying multiple labels to objects in the image of the sample container by applying bounding boxes to certain of the objects. For example, a cap, a tube, and an identification mark may be bounded by a bounding box. This process may be repeated, and the amount of labels in each image may vary depending on the desired classification. Certain classifications may require only one label to represent the contents of the entire image (e.g., image classification). Other classifications may require labeling multiple objects within a single image, each with a different label (e.g., a different bounding box). For example, it may be necessary to label at least two of a cap, a tube, and an identification mark to classify certain types of sample containers.

[0080] The repeatability of positioning between robot 228 and sample container 104, 204 or 504 enables the method of annotating objects in the image of sample container described above. For example, a sequence of images can be captured using slow movement between robot 228 and sample container 204 under well-lit lighting conditions, which makes it relatively easy to annotate in an automatic or semi-supervised manner. In addition, by capturing images of the same sample container at multiple known positions or sample container orientations, stereoscopic vision or multi-view stereoscopic vision can be used to extract high-resolution depth information of the sample container. Stereoscopic images enable reconstruction of a three-dimensional image (3-D image) of sample container 104, 204 or 504, as well as providing detailed features for distinguishing sample categories, which may help to automate the annotation process. For example, some capped sample containers with a black / grey center and a white outer ring may look almost the same as uncapped sample containers when observed using a single top-down image (e.g., an image captured along the z direction), making it possible to use manual annotation of a conventional system. In some embodiments, ground truth can be automatically identified based on a 3-D image.

[0081] For sample container localization, the annotations may be bounding boxes or binary masks for each sample container in the image. For sample container tracking, the annotations may be unique identifiers for each sample container in a held position across a sequence of images. These annotations may then be propagated in an automated manner to another sequence of images acquired under different imaging conditions, such as different lighting conditions, motion profiles, viewing positions (e.g., poses), and other imaging conditions, based on the position of the robot 228 and / or imaging device 226 relative to the previously annotated sequence of images. The annotation generator 912 may be trained on these images, such that training is an iterative process.

[0082] As described above, the laboratory system 100 can use different types of sample containers 104, 204, or 504, such as from different manufacturers. The laboratory system 100 should know the type of sample container 104, 204, or 504 in order to properly transport the sample container 104, 204, or 504 and process the sample. The robot (e.g., robot 228) and the sample carrier 112 may have specific hardware and processes for transporting different types of sample containers 104, 204, or 504. For example, the robot 228 may grip a first type of sample container that is different from a second type of sample container. In addition, the laboratory system 100 may utilize different types of sample carriers 112 depending on the type of sample container. Therefore, it is important for the laboratory system 100 to identify the sample container.

[0083] The laboratory system 100 described herein uses a vision system, such as the imaging device 226, to capture images of the sample containers 104, 204, or 504. The characterization algorithm 138 analyzes the image data generated by the imaging device 226 (or the imaging device 240) to identify and / or classify the sample container. Other imaging devices may capture images of the sample containers, and the characterization algorithm 138 may analyze the image data generated by these imaging devices.

[0084] The characterization algorithm 138 may include an AI model that is configured to characterize different types of sample containers and their corresponding tube and / or cap variations. As new types of sample containers are introduced into the laboratory system 100, the AI ​​model in the characterization algorithm 138 should be updated to be able to classify the new types of sample containers. As previously described, retraining the AI ​​model in conventional laboratory systems can be expensive and time consuming. The laboratory system 100 described herein overcomes the problem of new sample container classification by training the annotation generator 912 as described herein.

[0085] In another example, the laboratory system 100 may receive a new sample container type from a manufacturer. In some embodiments, the new type of sample container may be loaded into the tray 212A ( Figure 2). Each trait of the new sample container may be similar to the trait of a particular sample container on which the characterization algorithm 138 (e.g., including the annotation generator 912) has been trained. For example, the new sample container 204 may have the same tube material as the sample container on which the characterization algorithm 138 has been trained, but with a different cap shape. Another sample container type on which the characterization algorithm 138 has been trained may have the same cap type as the new sample container type, but with a different tube material. The characterization algorithm 138 (and the annotation generator 912) may be trained on the new sample container (e.g., by annotating images of the new sample container type with one or more annotations from previous images, and then retraining the annotation generator 912).

[0086] In some embodiments, a user may receive a new type of sample container or a sample container that has not yet been properly identified and load the sample container into one of the trays 212, such as tray 212A ( Figure 2 ). For example, a user may slide tray 212A into sample processing machine 106 via fourth slide 214D. When fourth slide 214D is slid into sample processing machine 106, fourth slide sensor 220D may detect the movement and may capture an image of indicia 205, which may indicate that tray 212A contains a new sample container. In other embodiments, a user may slide tray 212A into sample processing machine 106 via workstation 140( Figure 1 ) input data indicating that a new sample container is located in tray 212A. In some embodiments, tray 212A may also contain similar sample containers that can be imaged and used to train annotation generator 912.

[0087] Characterization algorithm 138 is implemented by using coordinator 902 ( Fig. 9 ) can transmit instructions to the robot controller 136, which causes the robot 228 to move to a predetermined position so that the imaging device 226 can capture images of the sample container 104, 204, or 504. These images can be captured by one or both of the first camera 636 and the second camera 638 and under different imaging conditions. For example, the images can be captured under different lighting and camera conditions determined by the characterization algorithm 138 and the imaging controller 139.

[0088] In some embodiments, the robot 228 can grasp the sample container and extract the sample container from the tray 212A, such as Figure 5 . The imaging device 226 can then capture an image of the sample container. In some embodiments, the robot 228 can return the sample container to the tray 212A and re-grip the sample container so that the sample container is in a different orientation relative to the imaging device 226. The imaging device 226 can then capture a new image for processing as described herein. Figure 7, the robot 228 can rotate the sample container relative to the imaging device 226 via the auxiliary arm 754. The imaging device 226 can then capture images of the sample container at different orientations. In some embodiments, the imaging device 226 can capture images of the sample container while the sample container is rotating relative to the imaging device 226. The rotation rate can be one of the imaging conditions described herein.

[0089] The second camera 638 ( Figure 6 ) can capture images from the top of the sample container, similar to the method described above with respect to the first camera 636. In some embodiments, the robot can move the imaging device 226 relative to the sample container while the second camera 638 captures an image of the sample container. The movement of the imaging device 226 relative to the sample container can be one of the imaging conditions described herein.

[0090] The image data generated by the imaging device 226 may be used to update, train, or retrain the characterization algorithm 138. For example, the image data may be used to update an AI model in the characterization algorithm 138. In some embodiments, updating or retraining includes training the annotation generator 912 or updating the training of the annotation generator 912, as described below.

[0091] Reference now Fig.10 , which illustrates a flow chart of a method 1000 for updating the training of an annotation generator (e.g., annotation generator 912) of a diagnostic laboratory system (e.g., laboratory system 100). The method includes providing an imaging device (e.g., imaging device 226) in the diagnostic laboratory system at block 1002, wherein the imaging device is controllably movable within the diagnostic laboratory system. The method 1000 includes, at block 1004, capturing a first image within the diagnostic laboratory system using the imaging device, the first image being captured with at least one imaging condition (e.g., a predetermined illumination intensity, illumination spectrum, sample container pose, exposure, image device and / or sample container velocity, etc.). The method 1000 includes, at block 1006, performing annotation of the first image using the annotation generator to generate a first annotated image. For example, in some embodiments, the annotation may include surrounding the image of the sample container with a virtual box or other shape (e.g., a bounding box or pixel mask) to isolate the sample container.

[0092] Method 1000 includes, at block 1008, updating the training of the annotation generator using the first annotated image. The annotation generator training may be implemented based on a specific algorithm to be trained. For a sample container detection task, the annotation generator 912 may, for example, annotate the bounding box of the sample container based on the detection algorithm in training. For a sample container classification / recognition task, the annotation generator 912 may annotate the category / type of the sample container based on the classification algorithm in training. For a semantic segmentation task, the annotation generator 912 may generate an annotation mask for each object region within the input image at the pixel level. Since the annotation occurs at the pixel level, the annotated region may be any irregular shape (e.g., a pixel-level mask, a polygon, a contour, a spline), rather than a predetermined bounding box in a square, rectangular, circular, or elliptical shape. In some embodiments, the annotation generator 912 may train one or more tasks simultaneously. The annotations generated by the annotation generator 912 can be used together with the input images for the next iteration of algorithm / model training, and the updated algorithm / model can be used by the annotation generator 912 to annotate new input images under different imaging conditions (e.g., different illumination intensities, illumination spectra, sample container postures, exposure rates, etc.). Then, the annotations and these new input images can be used for the next training iteration. In some embodiments, the annotation generator 912 can be implemented by using machine learning methods, so it can learn to handle increasingly challenging conditions through training iterations. For example, the annotation generator 912 can be implemented as a deep neural network algorithm. In some embodiments, the annotation generator 912 may include a deep learning network, such as a general convolutional neural network (CNN). Exemplary networks include Inception, ResNet, ResNeXt, DenseNet, etc., but other CNN and / or AI architectures may also be used. Training can be performed continuously, periodically, or at any appropriate time. Training can be performed when the diagnostic laboratory system 100 is online (e.g., in use) or offline.

[0093] Reference now Fig.11, which illustrates a flow chart of another exemplary method 1100 for training an annotation generator (e.g., annotation generator 912) of a diagnostic laboratory system (e.g., laboratory system 100). The method 1100 includes providing an imaging device (e.g., imaging device 226) in the diagnostic laboratory system at block 1102, wherein the imaging device can be controllably moved within the diagnostic laboratory system. For example, the imaging device 226 can be fixed to the robot 228 and moved therewith. The method 1100 includes, at block 1104, using the imaging device to capture a first image of a sample container (e.g., sample container 104, 204, or 504), the first image being captured under an imaging condition. In an exemplary embodiment, the imaging condition can be an illumination intensity. For example, the first image can be captured under a well-lit condition, under which the annotation generator 912 has been previously trained. Other exemplary imaging conditions may include illumination intensity, illumination spectrum, sample container and / or imaging device speed, relative position and angle between the imaging device and one or more sample containers, imaging device exposure, imaging device lens properties (e.g., focal length, aperture, and depth of field), and the like.

[0094] The method 1100 includes performing annotation of the first image to generate a first annotated image at block 1106. For example, the annotation generator 912 may annotate the image using a bounding box, a pixel mask, etc. If the imaging conditions used with the first image are imaging conditions that the annotation generator 912 was previously trained on, the annotations provided by the annotation generator 912 should be very accurate.

[0095] Method 1100 includes changing the first imaging condition to the second imaging condition at block 1108. For example, in an embodiment of a well-illuminated first image, the imaging condition may be to reduce the illumination intensity before capturing the second image. Other imaging conditions may also be changed. Method 1100 includes capturing a second image of a sample container at a second imaging condition using an imaging device at block 1110. Method 1100 includes performing annotation of the second image at block 1112 to generate a second annotation image. In some embodiments, annotation generator 912 may use the same annotation used with the first image. For example, if the imaging condition that is changed is the illumination intensity (or illumination spectrum), the precise control provided by robot 228 allows imaging device 226 to be positioned at exactly the same viewing position to capture the second image on a tray of exactly the same sample container. Since all (or most) conditions except the illumination intensity (or spectrum) are the same during the capture of the first and second images, the annotation used for the first image may be used as the annotation of the second image. Using the annotations and the second set of images, the annotation generator can be improved (e.g., retrained) to annotate images taken under reduced lighting conditions (e.g., half intensity), different lighting spectra, or any altered image conditions. The method 1100 includes training the annotation generator using at least the first annotated image and the second annotated image at block 1114. For example, both the first annotated image and the second annotated image can be included in the training image set used to train the annotation generator 912. The training can be performed continuously, periodically, or at any suitable time.

[0096] The method 1100 includes changing the second imaging condition to a third imaging condition at block 1116. For example, the imaging condition may include illumination intensity, illumination spectrum, sample container and / or imaging device speed, sample container posture, exposure rate, etc. The method 1100 includes capturing a third image of the sample container using the imaging device under the third imaging condition at block 1118. As with the first and second images, in some embodiments, the third image may be captured using the imaging device 226.

[0097] Method 1100 includes, at block 1120, performing annotation of a third image using an annotation generator to generate a third annotated image. In some embodiments, the annotation generator 912 may use the same annotation used with the first image or the second image. For example, if the imaging condition that is changed is the illumination intensity (or illumination spectrum or exposure rate, etc.), the precise control provided by the robot 228 allows the imaging device 226 to be positioned at exactly the same observation position as used for the first and second images to capture the third image. Similarly, precise changes in the position of the imaging device 226 relative to the sample container may be provided between images. In this way, the annotation for the first or second image may be used as the annotation for the third image. Using the annotation and the third image, the annotation generator may be retrained to annotate images captured under different imaging conditions, such as reduced illumination, different illumination spectra, different sample container postures, different exposure rates, etc.

[0098] The method 1100 includes further training the annotation generator 912 using at least a third annotated image at block 1122. As previously described, the training may be performed continuously, periodically, or at any suitable time.

[0099] Using the annotations and the first, second, and third images, the annotation generator 912 can be retrained to annotate images taken under different conditions (e.g., reduced lighting, different illumination spectra, different speeds between the sample container and the imaging device, different sample container postures, different exposure rates, etc.). In this way, the annotation generator 912 itself can be part of the sample characterization algorithm 138 and can be iteratively trained to handle increasing variations. That is, the controllable movement of the imaging device 226 allows the annotations of the first set of images taken under the first set of conditions to be used to annotate the second set of images taken under the second set of conditions (e.g., different illumination intensity, different illumination spectra, different motion profiles, different sample container positions, etc.). This can be extended to a third, fourth, fifth, or other number of image sets and / or imaging conditions. The above process can be used to train the annotation generator 912 to annotate images of sample containers, sample container holders, or other image features using widely varying imaging conditions within an actual deployed diagnostic laboratory system.

[0100] Figures 12A-12I FIG. 1 illustrates exemplary images and image annotations according to embodiments provided herein. Fig. 12A , an image 1202 of a sample container 1204 is shown. The sample container 1204 may be similar to the previously described sample containers 104, 204, or 504, and include a tube 1205, a cap 1206, and a label 1208. The sample container 1204 is supported on a carrier 1210. Fig. 12B An example of a bounding box annotation 1212 for a sample container 1204 of an image 1202 is shown. Fig. 12CAn example of a pixel mask annotation 1214 is shown for a sample container 1204. Other annotation types may also be used.

[0101] Fig.12D An example of a first image 1220a captured under a first imaging condition (eg, a first illumination intensity) is illustrated. Fig.12E An example of a first annotated image 1220b based on a first image 1220a is illustrated (eg, using a bounding box 1212a or other suitable annotation). For example, the first imaging condition may be a condition under which the annotation generator 912 is trained such that the annotation 1212a of the first image 1220a is highly accurate. Fig.12F An example of a second image 1222a captured under a second imaging condition that is different from the first imaging condition used for the first image 1220a is illustrated. For example, the second image 1222a may be captured using different lighting intensities, lighting spectra, sample container and / or imaging device speeds, relative positions and angles between the imaging device 226 and the sample container 1204, imaging device exposure, imaging device lens properties (e.g., focal length, aperture, depth of field), etc. (e.g., Fig.12F and 12G ). Figure 12G An example of a second annotated image 1222b based on the second image 1222a and using the annotations 1212a of the first annotated image 1220b as previously described is illustrated. Fig.12H An example of a third image 1224a captured under a third imaging condition different from the first or second imaging condition is shown. Fig.12H and 12I Finally, Fig.12I A third annotated image 1224b is illustrated that is based on the third image 1224a and uses the annotations 1212a of the first annotated image 1220b or the second annotated image 1222b. Using the annotated second and / or third images 1222b, 1224b, the annotation generator 912 can be retrained to annotate images taken under different imaging conditions, such as different lighting intensities, lighting spectra, sample containers and / or imaging device speeds, relative positions and angles between the imaging device and one or more sample containers, imaging device exposure, imaging device lens properties (e.g., focal length, aperture, depth of field), etc.

[0102] Figures 13A-13F Additional exemplary images and image annotations according to embodiments provided herein are illustrated. Fig.13A , an image 1302 of a plurality of sample containers 1204 in a tray 1306 is shown. The sample containers 1204 may be similar to the sample containers 104, 204, or 504 previously described. Fig. 13BAn example of bounding box annotation 1212 for sample container 1204 of image 1302 is illustrated. Fig. 13C Examples of mask annotations 1312a and 1312b are shown for sample container 1204 that identify different characteristics of the sample container (eg, capped or uncapped, different cap colors, different cap types, etc.) Other annotation types may also be used.

[0103] Fig.13D An example of a first annotated image 1320 captured under a first imaging condition (e.g., a first illumination intensity) and annotated using a mask for the top of each sample container 1204 is illustrated. For example, the first imaging condition may be a condition for training the annotation generator 912 so that the annotations of the first annotated image 1320 are highly accurate. Fig.13E An example of a second annotated image 1322 captured under a second imaging condition that is different from the first imaging condition used for the first annotated image 1320 is shown. For example, the second annotated image 1322 may be captured using a different lighting intensity, lighting spectrum, sample container and / or imaging device speed, relative position and angle between the imaging device 226 and the sample container 1204, imaging device exposure, imaging device lens properties (e.g., focal length, aperture, depth of field), etc. (e.g., Fig.13E In some embodiments, the second annotated image 1322 may use the annotations of the first annotated image 1320 as described above. Fig.13F An example of a third annotated image 1324 captured under a third imaging condition different from the first or second imaging condition is shown. Fig.13F In some embodiments, the third annotated image 1324 may use annotations of the first annotated image 1320 or the second annotated image 1322. Using the annotated second and / or third images 1322, 1324, the annotation generator 912 may be retrained to annotate images taken under different imaging conditions as described above.

[0104] Although image capture is primarily described with respect to imaging device 226, it will be appreciated that imaging device 240 or any other suitable imaging device may be used.

[0105] As described above, the annotation generator 912 may be trained to annotate images taken under different image conditions. Such annotation may allow for more accurate characterization of sample containers and allow for improved substrate processing by the substrate handler 106 and / or the robot 228. In one or more embodiments, the images annotated by the annotation generator 912 may be used to identify sample containers, and the robot 228 or another robot may be located and / or used based on the images annotated by the annotation generator 912 to transport the sample containers.

[0106] While the present disclosure is susceptible to various modifications and alternative forms, specific method and apparatus embodiments have been shown by way of example in the drawings and described in detail herein. However, it should be understood that the specific methods and apparatus disclosed herein are not intended to limit the present disclosure.

Claims

1. A method for updating the training of a sample characterization algorithm of a diagnostic laboratory system, the method comprising: providing an imaging device in the diagnostic laboratory system, wherein the imaging device is controllably movable within the diagnostic laboratory system; capturing a first image within the diagnostic laboratory system using the imaging device, the first image being captured under an imaging condition; performing annotation of the first image using an annotation generator of the diagnostic laboratory system to generate a first annotated image; and The first annotated image is used to update training of the annotation generator.

2. The method according to claim 1, further comprising: changing the imaging condition to a changed imaging condition; capturing a second image within the diagnostic laboratory system using the imaging device with the altered imaging conditions; Using the annotation generator to perform annotation of the second image to generate a second annotated image; as well as The training of the annotation generator is updated using the second annotated image.

3. The method according to claim 2, wherein: The first image and the second image include a holding position of a sample container.

4. The method according to claim 2, wherein: The first image and the second image include a sample container.

5. The method of claim 4, further comprising providing a robot including a gripper, wherein: Providing the imaging device includes securing the imaging device to the robot, and further includes grasping the sample container during capture of the first image.

6. The method according to claim 4, wherein: The imaging condition is the velocity of the imaging device relative to the sample container during imaging.

7. The method according to claim 4, wherein: The imaging condition is the posture of the imaging device relative to the sample container.

8. The method according to claim 4, wherein: The imaging condition is the position of the imaging device relative to the sample container.

9. The method according to claim 1, wherein: The imaging condition is the illumination intensity within the diagnostic laboratory system.

10. The method according to claim 1, wherein: The annotation is a bounding box or a pixel mask of an object in the first image.

11. The method according to claim 1, wherein: The annotation is one or more attributes of the sample container in the image.

12. The method according to claim 11, wherein: The one or more attributes include an orientation of the sample container relative to a holding position of the sample processing machine.

13. The method according to claim 11, wherein: The one or more attributes include at least one of a geometry of at least a portion of the sample container, a sample container height, a sample container diameter, a property of a liquid in the sample container, and a sample container identification mark.

14. A method for training a sample characterization algorithm of a diagnostic laboratory system, the method comprising: providing an imaging device in the diagnostic laboratory system, wherein the imaging device is controllably movable within the diagnostic laboratory system; capturing a first image of a sample container using the imaging device, the first image being captured under first imaging conditions; performing annotation of the first image to generate a first annotated image; changing the first imaging condition to a second imaging condition; capturing a second image of the sample container using the imaging device under the second imaging condition; performing the annotation of the second image to generate a second annotated image; training an annotation generator of the diagnostic laboratory system using at least the first annotated image and the second annotated image; changing the second imaging condition to a third imaging condition; capturing a third image of the sample container using the imaging device under the third imaging condition; performing the annotation of the third image using the annotation generator to generate a third annotated image; and The annotation generator is further trained using at least the third annotated image.

15. The method of claim 14, further comprising providing a robot including a gripper, wherein: Providing the imaging device includes providing the imaging device secured to the robot, and also includes grasping the sample container during capture of the first image, the second image, or the third image.

16. The method according to claim 14, wherein: The first imaging condition is a first illumination intensity for illuminating the sample container during capturing the first image, the second imaging condition is a second illumination intensity for illuminating the sample container during capturing the second image, and the third imaging condition is a third illumination intensity for illuminating the sample container during capturing the third image.

17. The method according to claim 14, wherein: The first imaging condition is a first speed of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second speed of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third speed of the imaging device relative to the sample container during capturing the third image.

18. The method according to claim 14, wherein: The first imaging condition is a first posture of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second posture of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third posture of the imaging device relative to the sample container during capturing the third image.

19. The method according to claim 14, wherein: The annotation is a bounding box or a pixel mask of the sample container.

20. A diagnostic laboratory system comprising: an imaging device controllably movable within the diagnostic laboratory system, wherein the imaging device is configured to capture images within the diagnostic laboratory system under different imaging conditions; a processor coupled to the imaging device; a memory coupled to the processor, wherein the memory includes an annotation generator trained to annotate images captured by the imaging device, the processor further comprising computer program code that, when executed by the processor, causes the processor to: receiving first image data of a first image captured by the imaging device using at least one imaging condition; causing the annotation generator to perform annotation of the first image to generate a first annotated image; and The first annotated image is used to update training of the annotation generator.

Images