Diagnostic laboratory system and method for imaging a tube assembly - Patent Application 20070122997

Abstract

A method for integrating images of a tube assembly includes capturing images of a tube assembly to generate a captured image. The captured image is decomposed into a plurality of features in a latent space using a trained image decomposition model. One or more of the features in the latent space are manipulated into one or more manipulated features. An integrated tube assembly image is generated having at least one of the manipulated features using a trained image synthesis model. Other methods and systems are disclosed.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 268,846, filed March 3, 2022, entitled "DIAGNOSTIC LABORATORY SYSTEMS AND METHODS OF IMAGING TUBE ASSEMBLIES," the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] SUMMARY OF THE INVENTION Embodiments of the present disclosure relate to diagnostic laboratory systems and methods for imaging tube assemblies in diagnostic laboratory systems. [Background technology]

[0003] Diagnostic laboratory systems perform clinical chemistry or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. Samples can be received and / or transported throughout the system in a sample tube assembly.

[0004] Many laboratory systems process large volumes of tube assemblies and samples in the tube assemblies. Some laboratory systems use machine vision and machine learning to facilitate sample processing and tube assembly identification (e.g., characterization). For example, vision-based machine learning models (e.g., AI models) have been adopted to provide fast and non-invasive methods for tube assembly identification and fluid characterization. However, retraining or applying a machine learning model to characterize a new tube assembly may require a large amount of training data, and therefore the training cost for adding a new tube assembly may be excessive. Summary of the Invention [Problem to be solved by the invention]

[0005] Tube assembly manufacturers continue to produce new tube assembly configurations to introduce new configurations or reduce production costs. Additionally, due to availability and / or economic reasons, some laboratories may utilize third-party, low-cost tube assembly alternatives. With so many tube assembly configurations, it is difficult to collect all possible tube assembly variations worldwide, meaning that a machine learning model likely cannot be trained on all tube assembly configurations. As such, there is a limit to the number of tube assembly configurations that a machine learning model can practically handle. Therefore, there is a need for a laboratory system and method that facilitates the introduction of new tube assembly configurations into a laboratory system. [Means for solving the problem]

[0006] According to a first aspect, there is provided a method for synthesizing images of a tube assembly, the method including: capturing images of a tube assembly to generate a captured image; decomposing the captured image into a plurality of features in a latent space using a trained image decomposition model; manipulating one or more of the features in the latent space into manipulated features; and generating a synthesized tube assembly image having at least one of the manipulated features using a trained image synthesis model.

[0007] In a further aspect, a method for synthesizing images of tube assemblies is provided, the method including receiving an input image of a tube assembly and constructing an image decomposition model configured to decompose the input image into a plurality of features in a latent space; constructing an image synthesis model configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space; and training the image decomposition model and the image synthesis model using at least an image of a first tube assembly and an image of a second tube assembly, wherein there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

[0008] In another aspect, a diagnostic laboratory system is provided, including an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; and an image synthesis model configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space, wherein the image decomposition model and the image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, wherein there are one or more known differences between the image of the first tube assembly and the second tube assembly.

[0009] Further aspects, configurations, and advantages of the present disclosure will be readily apparent from the following description and illustration of several exemplary embodiments, including the best mode contemplated for carrying out the disclosure. The present disclosure is capable of other and different embodiments, and its several details can be modified in various respects without departing from the scope of the present disclosure. The present disclosure includes all modifications, equivalents, and alternatives that come within the scope of the claims and their equivalents.

[0010] The drawings described below are for illustrative purposes and are not necessarily drawn to scale. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive. The drawings are not intended to limit the scope of the present disclosure in any way. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 shows a block diagram of a diagnostic laboratory system including multiple instruments, according to one or more embodiments. [Figure 2] 1 illustrates a top view of imaging equipment in a diagnostic laboratory system, according to one or more embodiments. [Figure 3] 3A-3C illustrate various types of tube assemblies, including caps attached to tubes, that can be used in diagnostic laboratory systems according to one or more embodiments. [Figure 4] 4A-4C illustrate various types of tubing in a tubing assembly that can be used in a diagnostic laboratory system, according to one or more embodiments. [Figure 5] 1 illustrates a block diagram of a process for implementing a method for integrating images of a tube assembly, according to one or more embodiments. [Figure 6] 1 illustrates a block diagram of a process for implementing a method for integrating external and internal images of a tube assembly, according to one or more embodiments. [Figure 7] 1 illustrates a process that can be used to train an image decomposition model and an image synthesis model, according to one or more embodiments. [Figure 8] Figures 8A and 8B show images of a similar tube assembly, with the image of Figure 8A having a blue cap and the image of Figure 8B having a green cap. Figure 8C shows an example of an exploded image of the tube assembly of Figure 8A, according to one or more embodiments. [Figure 9]9A-9B show images of tube assemblies having different internal and / or fluidic features, according to one or more embodiments. [Figure 10] 1 is a flowchart illustrating a method for integrating images of a tube assembly according to one or more embodiments. [Figure 11] 10 is a flowchart illustrating another method for integrating images of a tube assembly according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] Diagnostic laboratory systems perform clinical chemistry and / or assays that identify analytes or other components in biological samples, such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. Samples are collected in tube assemblies (e.g., sample tubes) and transported via the tube assemblies to the laboratory system, where the samples are analyzed. In some embodiments, the tube assemblies are transported throughout the laboratory system so that various instruments can perform the analysis.

[0013] Laboratory systems may use tube assemblies from various manufacturers to collect samples and transport the samples to and throughout the laboratory system. Tube assemblies can include sample tubes, such as closed-bottom tubes. Some tube assemblies can include a cap to seal the tube. Tube assemblies can also contain the contents of the tube. Different tube assembly types can have different characteristics, such as different sizes and different chemical additives therein. For example, many tube assembly types are chemically active, meaning that the tube assembly contains one or more additive chemicals used to alter or maintain the condition of the sample or otherwise assist in processing the sample. In some embodiments, the inner wall of the tube can be coated with one or more additives, or the additives can be applied elsewhere in the tube. The type of additive contained in the tube can be a serum separator, a clotting agent such as thrombin, an anticoagulant such as EDTA or sodium citrate, an antiglycolytic additive, or other additives to alter or maintain the characteristics of the sample. A tube assembly manufacturer may associate the color of the cap on the tube and / or the shape of the tube or cap with a particular type of chemical additive contained in the tube.

[0014] Various manufacturers may have their own standards for associating attributes of tube assemblies, such as cap color, cap shape, and tube shape, with specific characteristics of the tube assembly. For example, the attributes may relate to the contents of the tube or, in some cases, whether the tube has vacuum capability. In some embodiments, a manufacturer may associate all tube assemblies with gray caps as tubes containing potassium oxalate and sodium fluorate configured to test glucose and lactate. Tube assemblies with green caps may contain heparin for stat electrolytes such as sodium, potassium, chloride, and bicarbonate. Tube assemblies with lavender caps may identify tubes containing EDTA (ethylenediaminetetraacetic acid—anticoagulant) configured to test CBC with diff., HgBA1c, and parathyroid hormone. Other cap colors, such as red, yellow, light blue, dark blue, pink, orange, and black, may be used to indicate other additives or the absence of additives. In other embodiments, a cap color combination, such as yellow and lavender, can be used to indicate a combination of EDTA and a gel separator, or green and yellow can be used to indicate lithium heparin and a gel separator.

[0015] Laboratory systems can use tube assembly attributes for further processing of the tube and / or sample. Because tubes can be chemically active, associating specific tests that can be performed on a sample with a specific tube assembly type is important because the tests may vary depending on the contents of the tube. Thus, laboratories can verify that the tests being performed on the sample in the tube are correct by analyzing the color and / or shape of the cap and / or tube. Other attributes can also be analyzed.

[0016] Transport mechanisms, such as robots and tube assembly carriers within a laboratory system, may have specific hardware and processing for moving different types of tube assemblies. For example, a robot may grip a first type of tube assembly differently than a second type of tube assembly. Additionally, the laboratory system may utilize different carriers depending on the type of tube assembly to be transported throughout the laboratory system. Therefore, the laboratory system needs to identify the tube assemblies.

[0017] The laboratory systems described herein can use a vision system to capture images of tube assemblies and identify the tube assemblies and / or the contents of the tube assemblies. For example, the laboratory system can include a vision-based artificial intelligence (AI) model configured to provide a fast and non-invasive method for characterizing tube assemblies. Embodiments of the AI model can be trained to characterize different types of tube assemblies and subtle variations in tubes and / or caps. When new types of tube assemblies are introduced into the laboratory system, the AI model must be updated to be able to classify the new types of tube assemblies. Retraining the AI model in conventional laboratory systems can be costly and time-consuming because multiple tube assembly types must be imaged and manually classified to retrain the AI model.

[0018] The systems and methods described herein overcome challenges with tube assembly classification by integrating images of the tube assemblies. The integrated data and / or paired tube assembly images, along with controlled differences in the images, can be used to train an image decomposition model and an image synthesis model. The trained decomposition model is used to decompose the tube assembly images into features (e.g., decomposed features) in a latent space. One or more of the features in the latent space can then be manipulated. The trained synthesis model can reassemble the features, including the manipulated features, to generate an integrated tube assembly image.

[0019] In some embodiments, image-to-image transformation is used with decomposition and synthesis models to integrate images of the tube assembly, where the new image, the integrated image, is a controlled modification of the reference image. Image-to-image transformation is a type of vision and graphics processing whose goal is to learn a mapping between an input image of the tube assembly and an output image of the tube assembly using a training set of aligned image pairs. In some embodiments, the processing can include using a generative adversarial network (GAN) to perform the image-to-image transformation. These and other systems and methods are described in further detail below with reference to FIGS. 1-11.

[0020] Reference is now made to FIG. 1 . FIG. 1 illustrates an exemplary embodiment of an automated diagnostic system 100 configured to process and / or analyze biological samples contained in tube assemblies 102 (e.g., sample containers). Tube assemblies 102 can be any suitable container (e.g., tubes), including transparent or translucent containers, such as blood collection tubes, test tubes, sample cups, cuvettes, or other containers that can contain samples and / or allow for imaging of samples contained therein. Tube assemblies may or may not have caps or lids attached thereto. As described herein, tube assemblies 102 can have different sizes and can have different cap colors and / or cap types. The system 100 and methods described herein enable the system 100 or devices coupled to the system 100 to generate images integrating different tube assemblies, as described herein.

[0021] System 100 can include multiple instruments 104, which are configured to process tube assemblies 102 and / or samples in tube assemblies 102. Tube assemblies 102 can be received in system 100 by input / output instrument 104A, which can include one or more racks 106 and a robot 108. Robot 108 can transport tube assemblies 102 between racks 106 and carriers 114 that are located on tracks 112. Carriers 114 are configured to transport tube assemblies 102 on tracks 112.

[0022] The instruments 104 can process the sample and / or the tube assembly 102. Processing can include pre-processing or pre-screening the sample and / or the tube assembly 102 prior to analysis by one or more of the instruments 104. For example, pre-processing can prepare the sample for a particular assay or other analysis. Pre-processing can also identify the type of tube assembly 102 and prepare the sample for analysis in response to that identification. Others of the instruments 104 can perform analysis of the sample.

[0023] One or more of the equipment 104 may include an imaging device that captures images of the tube assemblies 102 and / or carriers 114. In the embodiment of FIG. 1, imaging equipment 104B may include an imaging device as described below and may be configured to capture images of the tube assemblies 102 and / or carriers 114. In some embodiments, imaging device 116 may be located in or near input / output equipment 104A and may be configured to capture images of the tube assemblies 102 before they are transferred to truck 112 or loaded onto carriers 114.

[0024] Referring further to FIG. 2, FIG. 2 shows a top view of imaging device 104B. Other embodiments of imaging devices can be implemented in system 100. In the embodiment of FIG. 2, imaging device 104B is configured to capture images of tube assembly 102 while it is on track 112. Tube assembly 202 is shown in imaging position 214 on track 112 of imaging device 104B. Components of imaging device 104B can be controlled by computer 220. In some embodiments, computer 220 can be incorporated into computer 120 (FIG. 1).

[0025] Imaging device 104B may include three imaging devices 216, which are configured to capture images of tube assembly 202 at imaging position 214. Imaging devices 216 are individually referred to as first imaging device 216A, second imaging device 216B, and third imaging device 216C. In other embodiments, imaging device 104B may include one or more imaging devices. Imaging devices 216 may be configured to generate image data representative of tube assembly 202. Computer 220 and / or computer 120 (FIG. 1) may be configured to process the image data generated by imaging devices 216 as described herein.

[0026] Imaging device 104B may also include one or more illumination devices 217 configured to illuminate imaging location 214. Accordingly, illumination device 217 may be configured to illuminate tube assembly 202. In the embodiment of Figure 2, imaging device 104B may include three illumination devices 217, individually referred to as first illumination device 217A, second illumination device 217B, and third illumination device 217C.

[0027] 1 , system 100 may also include or be coupled to computer 120, which may be configured to operate and / or communicate with device 104. In some embodiments, computer 120 may analyze data received by system 100 and / or data generated by device 104. Computer 120 and programs executed by computer 120 may also analyze image data generated by imaging device 104B and other imaging devices, as described herein. Computer 120 may include processor 122, which is configured to execute computer-readable instructions (e.g., program code).

[0028] 1, computer 120 may be coupled to each of devices 104. For example, computer 120 may be coupled to computer 220 in imaging device 104B. In some embodiments, computer 120 may be coupled to one or more workstations (not shown) coupled to one or more of devices 104. In some embodiments, computer 120 may be configured to transmit computer instructions and / or computer code to each of devices 104. In some embodiments, computer 120 may be configured to receive and / or process data generated by devices 104.

[0029] The computer 120 can include or be coupled to a memory 124, which can store or access one or more modules and / or programs 126, which are described herein as being stored in the memory 124. The modules or programs 126 can be executable by the processor 122. The programs 126 can be configured to operate one or more of the devices 104. Although the programs are described as individual programs, in some embodiments, the programs 126 can be implemented as a single program. One or more of the programs 126 can execute artificial intelligence (AI) algorithms. The programs 126 can include a data processing program 126A, which can be configured to process data in at least one of the devices 104, including the imaging device 104B. For example, the data processing program 126A can process data input to the system 100 and / or one or more of the devices 104. The data processing program 126A may also process data generated by one or more of the instruments 104.

[0030] In some embodiments, memory 124 may include processing algorithm 126B. In some embodiments, memory 124 may include two or more processing algorithms. In some embodiments, two or more processing algorithms may be implemented as a single processing algorithm, such as processing algorithm 126B. Processing algorithm 126B may include computer code configured to analyze image data and other data generated by imaging devices in or coupled to system 100. In some embodiments, processing algorithm 126B may process the data using AI, which may be implemented as tube characterization model 126C that analyzes the image data as described herein. Tube characterization model 126C may be updated as described herein. For example, image decomposition model 502 ( FIG. 5 ) and / or image synthesis model 504 ( FIG. 5 ) may be updated as described herein.

[0031] The imaging device 216 and the computer (e.g., computer 120) used to process images captured by the imaging device 216 can be part of the machine vision and machine learning used by the system 100 to facilitate processing of the tube assembly 102. For example, as described herein, the machine vision and AI models that can be run in the tube characterization model 126C can identify the tube assembly 102 and / or a particular type of sample present in the tube assembly 102 to facilitate processing and analysis of the sample as described herein.

[0032] The computer 120 may be coupled to or implemented with a workstation 128. The workstation 128 may enable user interaction with the system 100 and / or one or more of the devices 104. The workstation 128 may include a display 128A and / or a keyboard 128B, which allow a user to interact with individual ones of the system 100 and / or devices 104.

[0033] In some embodiments, system 100 can be coupled to a laboratory information system (LIS) 130, which can determine how a sample should be tested by system 100. In some embodiments, LIS 130 can be implemented on computer 120. LIS 130 can be coupled to a hospital information system (HIS) 132, which can accept specific assay instructions for a particular sample. HIS 132 can also receive assay results after an assay is performed on a particular sample by system 100.

[0034] In one example of the operation of system 100, a physician may prescribe a particular test to be performed on a sample from a patient. The physician may enter the test order into HIS 132. HIS 132 may then transmit the test order to LIS 130. The sample may be drawn from the patient and placed into a tube assembly, such as tube assembly 202 (FIG. 2). The configuration of tube assembly 202, such as color, size, and shape, may correspond to the type of test in the test order. Tube assembly 202 may then be sent to system 100. LIS 130 may determine the manner in which the sample should be tested and may transmit that information to computer 120, which may generate instructions to operate instrument 104 to perform the test. For example, data processing program 126A may generate the instructions. Imaging device 216 (FIG. 2) may generate an image of tube assembly 202, and based on the image data, tube characterization model 126C may identify the type of tube assembly 202. The testing of the tube assembly 202 and its movement through the system 100 may be based at least in part on the type of tube assembly 202. After the testing is completed, the LIS 130 may transmit the test results to the HIS 132.

[0035] Reference is now made additionally to Figures 3A-3C, which illustrate various types of tube assemblies that may be used within system 100. Tube assemblies refer to tubes with or without caps attached. Tube assemblies may also contain samples or other contents of the tube assembly. Reference is also made additionally to Figures 4A-4C, which illustrate the tube assemblies of Figures 3A-3C without caps. As shown in each figure, all tube assemblies have various configurations or geometries. Each cap may have a variety of cap geometries.

[0036] Tube assembly 340 in Figure 3A includes cap 340A, which is white with a red stripe and has a vertically extending portion. Cap 340A fits onto tube 340B. Tube assembly 340 has a height H31. Figure 4A shows tube 340B without cap 340A. Tube 340B has a height H41 and a width W41.

[0037] Tube assembly 342 in Figure 3B includes cap 342A, which is blue and has a dome-shaped top. Cap 342A can fit over tube 342B. Tube assembly 342 has a height H32. Figure 4B shows tube 342B without cap 342A. Tube 342B has a height H42 and a width W42.

[0038] Tube assembly 344 in Figure 3C includes cap 344A, which is red and gray and has a flat top. Cap 344A can fit over tube 344B. Tube assembly 344 has a height H33. Figure 4C shows tube 344B without cap 344A. Tube 344B has a height H43 and a width W43.

[0039] 3A-4C illustrate various types of tubes, caps, and tube assemblies that can be used with system 100. Tube characterization model 126C is trained to characterize or identify the tubes, caps, and tube assemblies of FIGS. 3A-4C. In some embodiments, tube characterization model 126C can also determine whether a tube assembly cannot be characterized. In some embodiments, tubes can have shapes other than the cylindrical shape of FIGS. 3A-4C and can be characterized by tube characterization model 126C. Additionally, tubes and caps can be made from various materials and have various shapes, structures, albedos, and textures, which can be characterized by tube characterization model 126C.

[0040] Over time, laboratories may utilize tube assemblies from sources that provide tube assemblies that deviate from the manufacturer-supplied tube assemblies on which the tube characterization model 126C was trained. For example, the laboratory may use tube assemblies supplied by a local manufacturer or new tube assembly designs that are not stored in the tube characterization model 126C. Training data may be limited, limiting the number of tube variations that the tube characterization model 126C can identify or characterize. Retraining the tube characterization model 126C to handle different tube types typically requires training a large number of images showing various configurations of tube assemblies under different viewpoints and lighting conditions. In some conventional embodiments, training is performed manually, which makes retraining costs prohibitive.

[0041] Embodiments disclosed herein integrate tube assembly images to generate an integrated tube assembly image. In one example, tube characterization model 126C is trained to characterize a particular type of tube assembly that can be used for a particular inspection. Tube assemblies can have specific tube material, tube color, tube shape, cap shape, cap color, and other unique attributes. Over time, one or more manufacturers may change those attributes of tube assemblies. Embodiments disclosed herein enable system 100, such as processing algorithm 126B, to integrate images of tube assemblies with new attributes; these integrated images can then be used by tube characterization model 126C to characterize new tube assemblies.

[0042] In another example, system 100 may receive a new tube assembly type from a new manufacturer. Attributes of the new tube assembly type may be similar to attributes of the particular tube assembly on which tube characterization model 126C was trained. For example, the new tube assembly may have the same tube material as the tube assembly on which tube characterization model 126C was trained, but a different cap shape. Meanwhile, another tube assembly type on which tube characterization model 126C was trained may have the same cap type as the new tube assembly type, but a different tube material. System 100 may use tube characterization model 126C to integrate images of the new tube assembly with a different cap shape. In some embodiments, system 100 may integrate images using similar cap shapes of another type of tube assembly. In other embodiments, system 100 may integrate the cap shape of the new tube assembly.

[0043] Some embodiments disclosed herein can use image-to-image transformation of unpaired or paired images to apply existing data (e.g., image data) from existing tube assembly types to new tube assembly types. The existing data can be images used to train the tube characterization model 126C. The image-to-image transformation can be performed using machine learning, such as deep learning (DL) methods. An example of machine learning includes generative adversarial networks (GANs). An example of a GAN is cycleGAN, which performs image-to-image transformation of unpaired images using a cycle-consistent adversarial network. Some embodiments decompose a captured image of a tube assembly into decomposed features. One or more of the decomposed features can be transformed into an integrated tube assembly image or a portion of the integrated tube assembly. Thus, the systems and methods disclosed herein enable the collection of data (e.g., images) from existing tube assembly configurations to create an integrated image of a different tube assembly.

[0044] Further reference is made to FIG. 5 . FIG. 5 shows a block diagram of a process 500 for implementing a method for integrating images of tube assemblies as described herein. The images referred to in FIG. 5 may be image data, such as image data generated by imaging device 104B. Process 500 uses integrated data or images of pairs of tube assemblies with controlled differences to train an image decomposition model 502 and an image synthesis model 504. The trained image decomposition model 502 can then receive input images of tube assemblies and decompose the input images into a latent space 506 having multiple features. A latent space manipulation module 508 can manipulate one or more features of the images in the latent space 506. For example, the latent space manipulation module 508 can manipulate one or more features in the latent space related to attributes, such as the color of the caps, of the images of the tube assemblies. The manipulated features can then be used by image synthesis model 504 to generate an integrated image of the tube assemblies.

[0045] 6 shows a block diagram of a process 600 that implements a method for integrating images of the tube assembly exterior and the tube assembly interior, according to one or more embodiments. Integrating the tube assembly interior can integrate the fluid in the tube assembly. In some embodiments, process 600 can be considered two separate processes.

[0046] The process 600 may include decomposing the image using the image decomposition model 502 described in FIG. 5 , which decomposes the input image into various features. The process 600 may use a tube exterior latent space 610 configured to characterize the exterior attributes of the image of the tube assembly. The tube exterior latent space 610 may include individual variables or features 612 configured to separate different attributes of the image of the exterior of the tube assembly that are decomposed by the image decomposition model 502. The features 612 may separate the image based on attributes such as the geometry and material of the tube assembly, the surface characteristics of the tube assembly, the lighting conditions when the image was captured, the camera characteristics of the imaging device (e.g., imaging device 216 of FIG. 2 ), etc. Other attributes of the image may be separated or characterized.

[0047] The tube-external latent space 610 can be manipulated by a latent space manipulation module 614. The latent space manipulation module 614 can be the same as or substantially similar to the latent space manipulation module 508 of Figure 5. The latent space manipulation module 614 can manipulate the features 612 that are disjointed in the tube-external latent space 610. The image synthesis model 504 can then synthesize an image using the manipulated features manipulated by the latent space manipulation module 614.

[0048] Process 600 can include decomposing an image of a tube interior or fluid using image decomposition model 502 described in FIG. 5 . Process 600 can use a tube interior latent space 618 configured to integrate interior attributes of an image of a tube assembly. In some embodiments, tube interior latent space 618 can characterize the contents of a tube assembly, such as a sample contained in the tube assembly, into features 620. Tube interior latent space 618 can include individual variables or features 620, which can include the geometry and material of the tube assembly, fluid properties, lighting conditions when the image was captured, camera properties of the imaging device (e.g., imaging device 216 of FIG. 2 ), and label and / or barcode properties. Other attributes of the image can be isolated or characterized into features 620.

[0049] The features 620 in the tube interior latent space 618 can be manipulated by a latent space manipulation module 624. The latent space manipulation module 624 can be the same as or substantially similar to the latent space manipulation module 614 or the latent space manipulation module 508 of FIG. 5. The latent space manipulation module 624 can manipulate the disjointed features 620 in the tube interior latent space 618. The image synthesis model 504 can then use the manipulated features to synthesize a unified image of the tube assembly.

[0050] Further reference is made to Figure 7, which illustrates one embodiment of a training process 700 that can be used to train the image decomposition model 502 and the image synthesis model 504. A first image A of a first tube assembly is input to an encoder network. The encoder network calculates the latent variables L in a multidimensional latent space. AThe decomposition model 730 may be similar to or identical to the decomposition model 530. In some embodiments, the decomposition model 730 generates latent features in a low-dimensional space. The latent variables L A The features can be divided into multiple groups of features, such as W, X, Y, and Z, as shown in FIG. 7. Each feature can correspond to a specific attribute of the tube assembly. In some embodiments, at least one of the features (e.g., feature Z) can be reserved for a unique characteristic of the image of the tube assembly whose differences cannot be controlled by the setup. In some examples, all tubes in a study can all have a common cylindrical structure, and thus are expected to all share the same value of feature Z.

[0051] The synthesis model 732 (sometimes called an image synthesis model), which can be implemented as a decoder network, is A can be used to reconstruct an image A' of the tube assembly that is as similar as possible to the input image A. The synthesis model 732 can be similar or identical to the synthesis model 532. The reconstruction loss module 734 can compare the input image A with the output image A' to determine whether the images are close to each other. For example, the reconstruction loss module 734 analyzes the attributes of the image A and the image A' and calculates a latent variable L A is correct and whether the decomposition model 730 and the composition model 732 are properly trained.

[0052] The training process 700 includes receiving a second input image B of a second tube assembly at the decomposition model 730 and calculating a latent variable L in the multidimensional space. B The images A and B may be a pair of tube assembly images. BThe features of can be divided into multiple groups of features, such as W', X', Y', and Z', as shown in FIG. 7. Each feature can correspond to a specific one of the attributes of the tube assembly. At least one of the features (e.g., feature Z') can be reserved for a unique characteristic of the tube image that cannot be captured by one of the features. The synthesis model 732 can use the features to reconstruct an image B' of the tube assembly that is as similar as possible to the input image B. The reconstruction loss module 736 can compare the input image B with the output image B' to determine whether the images are close to each other. For example, the reconstruction loss module 736 analyzes the attributes of image B and image B' and calculates a latent variable L B is correct and whether the decomposition model 730 and the composition model 732 are properly trained.

[0053] Training a disentangled latent space, such as that used for the decomposition model 730 and the synthesis model 732, can involve making input image B a specific difference to input image A. For example, when integrating the caps of a tube, the latent variable L A and the latent variable L B The first feature (e.g., W and W') of can be albedo, such as the color of the tube cap. In this example, input image B is of the same tube assembly type as input image A, differing only in the color of the tube cap. Input image B can be synthetically generated, such as generated using computer-generated imagery (CGI), or captured from the same type of tube assembly and the same imaging setup used to acquire input image A. The caps of the tubes in input image B can be very similar to the caps in input image A, with the only difference, or the only significant difference, being the different color of the caps. The latent space L A and L BIn (x, y, z) the remaining features (e.g., X and X', Y and Y', Z and Z') should be as similar as possible, while allowing the first features W and W' to be different as explained above.

[0054] The above process is performed on the latent space L A and L B , and so each latent feature captures a particular difference. In some embodiments, it is possible to have several differences between input image A and input image B simultaneously. In such embodiments, corresponding latent features can be allowed to differ while remaining features are kept consistent. As an example, input image A and input image B may be captured with the same camera but contain tube assemblies with different tube geometries, surface characteristics, and lighting conditions. In this embodiment, only the latent features corresponding to camera characteristics are constrained to be similar, while other features may differ.

[0055] The architecture of the training process 700 includes a decomposition model 730 and a composition model 732. Thus, training can be augmented by training on unpaired images to minimize the reconstruction loss alone. Such images can also be edited in the latent domain by replacing some of their latent features with latent features borrowed from another image. The decomposition model 730 and the composition model 732 can then be trained by an adversarial network that attempts to classify the edited and unedited images. For example, there can be a set of tube assembly images S1 captured by a first imaging device C1 (e.g., the first imaging device 216A in FIG. 2 ) using an illumination condition L1 provided by the first illumination device 217A. The latent features of the images S1 corresponding to the camera characteristics of the first imaging device 216A and the illumination conditions provided by the first illumination device 217A can be similar to each other. The two latent features can be expressed as a mean vector (m C1,m L1 ) can be normalized to approximate a unit Gaussian distribution with a mean vector (m C2 ,m L2 ) is constrained to approximate a unit Gaussian distribution with

[0056] The latent features of the tube image in S1 are expressed as a vector (m C2 ,m L2 ) and reconstructing an image of the tube assembly using the synthesis model 732. In some embodiments, an adversarial classifier can be further trained to enforce the rule that the synthesized image should be indistinguishable from the image of S2 in terms of camera and lighting. Similarly, the latent features of the tube assembly image of S2 can be calculated by (m C1 ,m L1 ) and integrate the captured images using C1 and L1. These processes ensure that the image differences are disentangled by the latent space (e.g., latent space 610 and tube interior latent space 618). After training, the trained image decomposition model 730 learns how to decompose the image into the disentangled latent space, and the trained image synthesis model 732 synthesizes the latent variables L in the latent space. A or L B We learn how to perform image synthesis from latent variables such as one of the latent vectors. Because each latent space is disentangled using known disparities, we can perturb the latent vector of a given tube assembly image to generate multiple tube assembly images with desired disparities.

[0057] In some embodiments, the latent space representation can encode data across different scales (e.g., decompose an image). Examples of generative deep learning include, for example, normalized flow, autoregressive models, variational autoencoders (VAEs), and deep energy-based models. Furthermore, the latent space can be restricted to a known parameter distribution (e.g., a Gaussian or Gaussian mixture distribution) or a non-parametric distribution, such as with vector quantization variational autoencoders (VQ-VAEs).

[0058] Additionally, images other than the generated images can be used as image data to characterize attributes such as tube cap classification and tube fluid characterization. The disentangled features in latent space can also provide useful information (used as ground truth annotations or treated as discriminative features) for downstream tasks that can be performed by system 100. Furthermore, the disentangled features in latent space can be used to create tools that assess novel tube assembly types and tube assembly differences to determine the ability of system 100 (FIG. 1) to properly characterize tube assemblies, which can be used by laboratory personnel such as suppliers, research and development staff, field service technicians, and / or end users (e.g., lab technicians and managers).

[0059] In some embodiments, system 100 can use unpaired image-to-image transformation, such as cycleGAN, to apply existing data from previous tube assembly images to images of new tube assemblies. Such a procedure allows system 100 and / or tube assembly developers to collect field data from existing tube assemblies as they create new types of tube assemblies.

[0060] In some embodiments, image data can be collected in groups that can include a high-density collection of typical tube assembly types and a low-density collection of other similar tube assembly types. Augmented data can be used to fill in missing attributes in the low-density collected images of tube assembly types. For example, color augmentation can be used to fill in missing colors in the low-density collected images of tube assembly types.

[0061] An example of a system 100 for integrating tube assemblies is described below. Reference is made to FIGS. 8A and 8B, which show images of similar tube assemblies. Tube assembly 840 in FIG. 8A is the same as or similar to tube assembly 842 in FIG. 8B, except that cap 840A of tube assembly 840 is blue in color and cap 842A is green in color. For example, tube 840B can be similar to tube 842B, and both tube assembly 840 and tube assembly 842 can have the same height H81. Decomposition model 730 can receive an image of tube assembly 840 as input image A. Decomposition model 730 can also receive an image of tube assembly 842 as input image B. Decomposition model 730 can combine input image A with variables L A Then, we decompose the image B into a set of features (W, X, Y, Z) in the latent space represented as B The decomposition is performed into a set of features (W', X', Y', Z') in the same latent space, represented as

[0062] Further, refer to FIG. 8C, which provides an example of a decomposed image 844 of a tube assembly 840. It is noted that other decompositions of the image of the tube assembly 840 can be performed. In the embodiment of FIG. 8C, feature W can be the color of the cap, which is blue. Feature X can be the height H 82 of the cap, feature Y can be the height H 83 of the tube, and feature Z can be the width W 81 of the tube. Other features can include the geometry of the tube or other parts of the geometry of the tube assembly. The synthesis model 732 can reconstruct the decomposed image to generate image A'. The input image A can be compared to the reconstructed image A' by the reconstruction loss module 734. If the loss or difference between the input image A and the reconstructed image A' is within a predetermined amount, the decomposition model 730 and the synthesis model 732 can be considered properly trained, and the latent variable L A is suitable for processing the tube assembly 840.

[0063] Tube assembly 842 can be selected as a pair image because the only significant difference between tube assembly 840 and tube assembly 842 is the color of the cap. The decomposition model 730 then maps image B of tube assembly 842 to a latent space L B The latent space L B The features of are stored in the latent space L A The feature W' may be the color of the cap, which is green, the feature X' may be the height H82 of the cap, the feature Y' may be the height H83 of the tube, and the feature Z' may be the width W81 of the tube. The synthesis model 732 may synthesize the disassembled images to generate image B'. The input image B may be compared to the reconstructed image B' by the reconstruction loss module 736. If the loss or difference between the input image B and the reconstructed image B' is within a predetermined amount, the disassembly model 730 and synthesis model 732 may be considered properly trained, and the latent variable L B is suitable for processing the tube assembly 842.

[0064] The green cap 842A can be a new cap color. To reconstruct the tube assembly 842 with the blue cap, feature W is used instead of feature W', which is the green cap. Image-to-image reconstruction and other methods can be used to reconstruct the tube assembly 842 with the blue cap. The resulting reconstructed or merged image using features W, X', Y', and Z' is tube assembly 842A with the blue cap of tube assembly 840. In some embodiments, the entire cap 840A can be used in place of cap 842A, and in other embodiments, the color of cap 840A can be used in place of the color of cap 842A. In other embodiments, the color of cap 840A can be stored and used for cap 842A without imaging tube assembly 840.

[0065] Reference is now made to Figures 9A-9B, which illustrate tube assemblies having different interior and / or fluid features that can be used in tube interior potential space 618 (Figure 6). Tube assembly 940 can include one or more items (e.g., attributes) that are inside tube 940B and can be sealed by cap 940A. One or more fluids 946 can be inside tube 940B and can include serum 946A, gel separator 946B, and blood 946C. A void 946D can be above fluid 946.

[0066] Tube assembly 942 can include one or more items (e.g., attributes) inside tube 942B and can be sealed by cap 942A. One or more fluids 948 can be inside tube 942B and can include serum 948A, gel separator 948B, and blood 948C. A void 948D can be above fluid 948. The material of tube 940B and the material of tube 942B can be similar and can be identified by generating image data through voids 946D and 948D. Thus, attributes of tube assembly 940 and tube assembly 942 include, but are not limited to, the geometry of the tube, the material of the tube, the color of the tube, the surface characteristics of the tube, the sample fluid in the tube, the label affixed to the tube, and the lighting conditions of the tube during image capture.

[0067] All of the above attributes of tube assembly 940 and tube assembly 942 may be the same or substantially similar. The difference between tube assembly 940 and tube assembly 942 may be the labels attached to tube 940B and tube 942B. Tube assembly 940 has label 950 affixed to tube 940B, and tube assembly 942 has label 952 affixed to tube 942B. As shown in FIGS. 9A-9B , label 950 differs from label 952. For example, the size of label 950 differs from the size of label 952, and the location of barcode 954 differs from the location of barcode 956.

[0068] Decomposition model 730 and composition model 732 can be configured to identify and / or characterize attributes of the images of tube assembly 940 and tube assembly 942 into features in a latent space. Accordingly, system 100 can be configured to substitute these features into an integrated image (e.g., image A' or B'). In the example of FIGS. 9A-9B, system 100 can use the labels of the original images in place of either label 950 or label 952 to generate the integrated image. In some embodiments, system 100 can replace the labels and maintain the original barcode or other label information in the integrated image.

[0069] In some embodiments, the integrated tube assembly image is used by the system 100 during the tube assembly identification process. For example, an image of the tube assembly 102 can be captured by the imaging device 116 or another imaging device when the tube assembly 102 is loaded into the input / output instrument 104A. The integrated tube assembly image can be used to identify tube assemblies not initially recognized by the tube characterization model 126C. The tube assemblies can then be characterized and transported to the appropriate instrument 104, where the sample is analyzed as described herein. In some embodiments, the instrument 104 can perform processing, such as aspiration, that is customized for the identified tube assembly. In other embodiments, the robot 108 (FIG. 1) can move sample containers to and from the carrier 114 using specific techniques, such as specific gripping techniques, depending on the type of identified tube assembly.

[0070] Reference is now made to FIG. 10 . FIG. 10 is a flowchart illustrating a method 1000 for synthesizing an image of a tube assembly (e.g., tube assembly 202). Method 1000 includes, at 1002, capturing an image of the tube assembly to generate a captured image. Method 1000 includes, at 1004, decomposing the captured image into a plurality of features (e.g., features 612) in a latent space (e.g., latent space 610) using a trained image decomposition model (e.g., image decomposition model 502). Method 1000 includes, at 1006, manipulating one or more of the features in the latent space to result in manipulated features. For example, the color of the cap can be changed from red to blue, and / or the shape of the cap can be changed from round to oval. Other features can additionally or alternatively be modified (i.e., manipulated). The method 1000 includes, at 1008, using the trained image synthesis model (e.g., image synthesis model 504) to generate an integrated tube assembly image having at least one of the manipulated features.

[0071] Reference is now made to FIG. 11 , which is a flowchart illustrating a method 1100 for integrating images of a tube assembly (e.g., tube assembly 202). Method 1100 includes, at 1102, receiving an input image of a tube assembly and constructing an image decomposition model (e.g., image decomposition model 502) configured to decompose the input image into a plurality of features (e.g., feature 612) in a latent space (e.g., latent space 610). Method 1100 includes, at 1104, constructing an image synthesis model (e.g., image synthesis model 504) configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space. Method 1100 also includes, at 1106, training the image decomposition model and the image synthesis model using at least an image of a first tube assembly (e.g., tube assembly 840) and an image of a second tube assembly (e.g., tube assembly 842) where there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

[0072] While the disclosure is susceptible to various modifications and alternative forms, specific method and apparatus embodiments have been shown by way of example in the drawings and are herein described in detail. It is to be understood, however, that the specific methods and apparatus disclosed herein are not intended to limit the disclosure, but on the contrary include all modifications, equivalents, and alternatives falling within the scope of the appended claims.

Claims

1. 1. A method for integrating an image of a tube assembly, comprising: capturing an image of the tube assembly to generate a captured image; Decomposing the captured image into a plurality of features in a latent space using the trained image decomposition model; Manipulating one or more of the features in the latent space into manipulated features; generating an integrated tube assembly image having at least one of the manipulated features using the trained image synthesis model; training an image decomposition model and an image synthesis model using at least an image of the first tube assembly and an image of the second tube assembly, where there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly, and the training produces a trained image decomposition model and a trained image synthesis model; The method.

2. The method of claim 1 , wherein training the decomposition model is based at least in part on a reconstruction loss between the integrated image and the input image.

3. The method of claim 1 , wherein the tube assembly includes a tube and a cap, and at least one of the plurality of features is a geometry of the tube, a color of the tube, a surface characteristic of the tube, a geometry of the cap, a color of the cap, or a lighting condition.

4. 2. The method of claim 1, wherein the tube assembly includes a tube, and at least one of the plurality of features is a geometry of the tube, a material of the tube, a sample fluid in the tube, a label attached to the tube, or an illumination condition of the tube.

5. 1. A method for integrating an image of a tube assembly, comprising: receiving an input image of a tube assembly, and converting the input image into a plurality of feature quantities in a latent space; constructing an image decomposition model configured to decompose into constructing an image synthesis model configured to synthesize an integrated tube assembly image based on a plurality of features in a latent space; The method includes training an image decomposition model and an image synthesis model using at least an image of a first tube assembly and an image of a second tube assembly, wherein there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

6. capturing an image of the tube assembly to generate a captured image; Decomposing the captured image into a plurality of decomposed features in a latent space using the trained image decomposition model; Manipulating one or more of the decomposed features in the latent space into manipulated features; generating an integrated tube assembly image having at least one of the manipulated features using the trained image synthesis model; and The method of claim 5 further comprising:

7. The method of claim 5 , wherein training the decomposition model is based at least in part on a reconstruction loss between the integrated image and the input image.

8. The method of claim 5 , wherein the image synthesis model is trained based on a reconstruction loss between the integrated image and the input image.

9. The method of claim 5 , wherein at least one of the features in the latent space is generated using computer-generated imagery.

10. The method of claim 5 , wherein at least two images of the tube assembly differ from each other with respect to one or more controlled attributes.

11. The method of claim 5 , wherein the images of at least two tube assemblies share one or more attributes.

12. 6. The method of claim 5, wherein the tube assembly includes a tube and a cap, and at least one of the plurality of features in the latent space is a tube geometry, a tube color, a surface property of the tube, a cap color, a cap geometry, or a lighting condition.

13. 6. The method of claim 5, wherein the tube assembly includes a tube, and at least one of the plurality of features in the latent space is a geometry of the tube, a material of the tube, a sample fluid in the tube, a label attached to the tube, or a lighting condition of the tube.

14. The method of claim 5 , wherein the one or more features in the latent space include fluid properties in the tube assembly.

15. 1. A diagnostic laboratory system comprising: an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; and an image synthesis model configured to synthesize an integrated tube assembly image based on multiple features in a latent space. Including, The diagnostic laboratory system, wherein the image decomposition model and the image synthesis model are trained using at least images of a first tube assembly and images of a second tube assembly where there are one or more known differences between the images of the first tube assembly and the images of the second tube assembly.

16. 16. The diagnostic laboratory system of claim 15, wherein the image decomposition model is trained based at least in part on a reconstruction loss between the integrated image and the input image.

17. 16. The diagnostic laboratory system of claim 15, wherein the image synthesis model is trained based on a reconstruction loss between the integrated image and the input image.

18. 16. The diagnostic laboratory system of claim 15, wherein the tube assembly includes a tube and a cap, and at least one of the plurality of features in the latent space is a tube geometry, a tube color, a surface property of the tube, a cap color, a cap geometry, or a lighting condition.

Images