Diagnostic laboratory system and method for imaging a tube assembly - Patents.com

Abstract

A method for integrating images of a tube assembly includes capturing images of a tube assembly to generate a captured image. The captured image is decomposed into a plurality of features in a latent space using a trained image decomposition model. One or more of the features in the latent space are manipulated into one or more manipulated features. An integrated tube assembly image is generated having at least one of the manipulated features using a trained image synthesis model. Other methods and systems are disclosed.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 268,846, filed March 3, 2022, entitled "DIAGNOSTIC LABORATORY SYSTEMS AND METHODS OF IMAGING TUBE ASSEMBLIES," the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to diagnostic laboratory systems and methods for imaging tube assemblies in diagnostic laboratory systems. [Background technology]

[0003] Diagnostic laboratory systems perform clinical chemistry or assays to identify analytes or other components in biological samples such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. Samples can be received and / or transported throughout the system in a sample tube assembly.

[0004] Many laboratory systems process a large number of tube assemblies and samples in the tube assemblies. Some laboratory systems use machine vision and machine learning to facilitate sample processing and tube assembly identification (e.g., characterization). For example, vision-based machine learning models (e.g., AI models) have been employed to provide fast and non-invasive methods for tube assembly identification and fluid characterization. However, retraining or applying machine learning models to characterize new tube assemblies may require a large amount of training data, so the training cost for adding new tube assemblies may be excessive. Summary of the Invention [Problem to be solved by the invention]

[0005] Tube assembly manufacturers continue to produce new tube assembly configurations to bring new configurations or reduce production costs. Additionally, some laboratories may utilize third-party low-cost tube assembly alternatives due to availability and / or economic reasons. With so many tube assembly configurations, it is difficult to collect all possible tube assembly variations in the world, meaning that machine learning models cannot possibly be trained on all tube assembly configurations. Thus, there is a limit to the number of tube assembly configurations that a machine learning model can actually handle. Thus, there is a need for a laboratory system and method that facilitates the introduction of new tube assembly configurations into the laboratory system. [Means for solving the problem]

[0006] According to a first aspect, a method for integrating images of a tube assembly is provided, the method including: capturing images of a tube assembly to generate a captured image; decomposing the captured image into a plurality of features in a latent space using a trained image decomposition model; manipulating one or more of the features in the latent space into manipulated features; and generating an integrated tube assembly image having at least one of the manipulated features using a trained image synthesis model.

[0007] In a further aspect, a method of integrating images of tube assemblies is provided, the method including: receiving an input image of a tube assembly, constructing an image decomposition model configured to decompose the input image into a plurality of features in a latent space; constructing an image synthesis model configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space; and training the image decomposition model and the image synthesis model using at least an image of a first tube assembly and an image of a second tube assembly, where the image of the first tube assembly and the image of the second tube assembly have one or more known differences between the image of the first tube assembly and the image of the second tube assembly, the training producing a trained image decomposition model and a trained image synthesis model.

[0008] In another aspect, a diagnostic laboratory system is provided, the diagnostic laboratory system including an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; and an image synthesis model configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space, where the image decomposition model and the image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, where there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly.

[0009] Further aspects, configurations, and advantages of the present disclosure will become readily apparent from the following description and illustration of several exemplary embodiments, including the best mode contemplated for carrying out the disclosure. The present disclosure is also capable of other and different embodiments, and its several details can be modified in various respects without departing from the scope of the present disclosure. The present disclosure includes all modifications, equivalents, and alternatives that come within the scope of the claims and their equivalents.

[0010] The drawings described below are for illustrative purposes and are not necessarily drawn to scale. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive. The drawings are not intended to limit the scope of the present disclosure in any way. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 shows a block diagram of a diagnostic laboratory system including multiple instruments, according to one or more embodiments. [Diagram 2] 1 illustrates a top view of imaging equipment provided in a diagnostic laboratory system in accordance with one or more embodiments. [Diagram 3] 3A-3C show various types of tube assemblies, including caps attached to tubes, that can be used in a diagnostic laboratory system according to one or more embodiments. [Figure 4] 4A-4C show various types of tubes of a tube assembly that can be used in a diagnostic laboratory system according to one or more embodiments. [Diagram 5] 1 illustrates a block diagram of a process for implementing a method for integrating images of a tube assembly, according to one or more embodiments. [Figure 6] 1 illustrates a block diagram of a process for implementing a method for integrating external and internal images of a tube assembly, according to one or more embodiments. [Figure 7] 1 illustrates a process that can be used to train an image decomposition model and an image synthesis model, according to one or more embodiments. [Figure 8] Figures 8A and 8B show images of a similar tube assembly, with the image of Figure 8A having a blue cap and the image of Figure 8B having a green cap. Figure 8C shows an example of an exploded image of the tube assembly of Figure 8A according to one or more embodiments. [Figure 9]9A-9B show images of tube assemblies having different internal and / or fluidic features according to one or more embodiments. [Figure 10] 1 is a flowchart illustrating a method for integrating images of a tube assembly according to one or more embodiments. [Figure 11] 10 is a flowchart illustrating another method of integrating images of a tube assembly according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Diagnostic laboratory systems perform clinical chemistry and / or assays that identify analytes or other components in biological samples, such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. Samples are collected in tube assemblies (e.g., sample tubes) and transported via the tube assemblies to the laboratory system where the samples are analyzed. In some embodiments, the tube assemblies are transported throughout the laboratory system so that various instruments can perform the analysis.

[0013] Laboratory systems may use tube assemblies from various manufacturers to collect samples and transport the samples to and through the laboratory system. The tube assemblies may include a sample tube, such as a closed-bottom tube. Some tube assemblies may include a cap to seal the tube. The tube assemblies may also include the contents of the tube. Different tube assembly types may have different characteristics, such as different sizes and different chemical additives therein. For example, many tube assembly types are chemically active, meaning that the tube assemblies contain one or more additive chemicals that are used to change or maintain the condition of the sample or otherwise aid in the processing of the sample. In some embodiments, the inner walls of the tube may be coated with one or more additives, or additives may be applied elsewhere in the tube. The types of additives contained in the tube may be serum separators, coagulants such as thrombin, anticoagulants such as EDTA or sodium citrate, antiglycolytic additives, or other additives to change or maintain characteristics of the sample. A tube assembly manufacturer may associate the color of the cap on the tube and / or the shape of the tube or cap with a particular type of chemical additive contained in the tube.

[0014] Various manufacturers may have their own standards for associating attributes of a tube assembly, such as cap color, cap shape, and tube shape, with specific characteristics of the tube assembly. For example, the attributes may relate to the contents of the tube, or in some cases, whether the tube has vacuum capabilities. In some embodiments, a manufacturer may associate all tube assemblies with gray caps with tubes containing potassium oxalate and sodium fluorate, configured to test glucose and lactate. Tube assemblies with green caps may contain heparin for stat electrolytes such as sodium, potassium, chloride, and bicarbonate. Tube assemblies with lavender caps may identify tubes containing EDTA (ethylenediaminetetraacetic acid - an anticoagulant), configured to test CBC w / diff., HgBA1c, and parathyroid hormone. Other cap colors, such as red, yellow, light blue, dark blue, pink, orange, and black, may be used to indicate other additives or the absence of additives. In other embodiments, a cap color combination can be used, such as yellow and lavender, to indicate a combination of EDTA and a gel separator, or green and yellow to indicate lithium heparin and a gel separator.

[0015] The laboratory system can use the tube assembly attributes for further processing of the tube and / or sample. Because tubes may be chemically active, it is important to associate a particular test that can be performed on a sample with a particular tube assembly type, since the test may vary depending on the contents of the tube. Thus, the laboratory can verify that the test being performed on the sample in the tube is correct by analyzing the color and / or shape of the cap and / or tube. Other attributes can also be analyzed.

[0016] Transport mechanisms, such as robots and tube assembly carriers within a lab system, may have specific hardware and processing for moving different types of tube assemblies. For example, a robot may grip a first type of tube assembly differently than a second type of tube assembly. Additionally, the lab system may utilize different carriers depending on the type of tube assembly to be transported throughout the lab system. Thus, the lab system needs to identify the tube assemblies.

[0017] The laboratory systems described herein can use a vision system to capture images of tube assemblies and identify the tube assemblies and / or contents of the tube assemblies. For example, the laboratory system can include a vision-based artificial intelligence (AI) model configured to provide a fast and non-invasive tube assembly characterization method. An embodiment of the AI ​​model can be trained to characterize different types of tube assemblies and slight variations in tubes and / or caps. When a new type of tube assembly is introduced into the laboratory system, the AI ​​model needs to be updated to be able to classify the new type of tube assembly. Retraining the AI ​​model in a conventional laboratory system can be costly and time-consuming due to the need to image and manually classify multiple tube assembly types to retrain the AI ​​model.

[0018] The systems and methods described herein overcome challenges with tube assembly classification by integrating images of the tube assemblies. The integrated data and / or paired tube assembly images, along with controlled differences in the images, can be used to train an image decomposition model and an image synthesis model. The trained decomposition model is used to decompose the tube assembly images into features (e.g., decomposed features) in a latent space. One or more of the features in the latent space can then be manipulated. The trained synthesis model can reassemble the features, including the manipulated features, to generate an integrated tube assembly image.

[0019] In some embodiments, image-to-image transformation is used with decomposition and synthesis models to integrate images of the tube assembly, where the new image, the integrated image, is a controlled modification of the reference image. Image-to-image transformation is a type of vision and graphics processing whose goal is to learn a mapping between an input image of the tube assembly and an output image of the tube assembly using a training set of aligned image pairs. In some embodiments, the processing can include using a generative adversarial network (GAN) to perform the image-to-image transformation. These and other systems and methods are described in further detail below with reference to FIGS. 1-11.

[0020] Reference is now made to FIG. 1. FIG. 1 illustrates an exemplary embodiment of an automated diagnostic system 100 configured to process and / or analyze a biological sample contained in a tube assembly 102 (e.g., a sample container). The tube assembly 102 can be any suitable container (e.g., a tube), including a transparent or translucent container, such as a blood collection tube, test tube, sample cup, cuvette, or other container that can contain a sample and / or allow imaging of a sample contained therein. The tube assembly may or may not have a cap or lid attached thereto. As described herein, the tube assembly 102 can have different sizes and can have different cap colors and / or cap types. The system 100 and methods described herein enable the system 100 or a device coupled to the system 100 to generate images integrating different tube assemblies, as described herein.

[0021] The system 100 can include multiple instruments 104, which are configured to process the tube assemblies 102 and / or samples in the tube assemblies 102. The tube assemblies 102 can be received in the system 100 at an input / output instrument 104A, which can include one or more racks 106 and a robot 108. The robot 108 can transport the tube assemblies 102 between the racks 106 and a carrier 114 that is on a track 112. The carrier 114 is configured to transport the tube assemblies 102 on the track 112.

[0022] The instruments 104 may process the sample and / or the tube assembly 102. Processing may include pre-processing or pre-screening the sample and / or the tube assembly 102 prior to analysis by one or more of the instruments 104. For example, pre-processing may prepare the sample for a particular assay or other analysis. Pre-processing may also identify the type of tube assembly 102 and prepare the sample for analysis in response to the identification. Others of the instruments 104 may perform analysis of the sample.

[0023] One or more of the equipment 104 may include an imaging device that captures images of the tube assemblies 102 and / or carriers 114. In the embodiment of FIG. 1, imaging equipment 104B may include an imaging device as described below and may be configured to capture images of the tube assemblies 102 and / or carriers 114. In some embodiments, imaging device 116 may be in or near input / output equipment 104A and may be configured to capture images of the tube assemblies 102 before they are transferred to the truck 112 or loaded onto the carriers 114.

[0024] Reference is further made to FIG. 2, which illustrates a top view of imaging device 104B. Other embodiments of imaging devices can be implemented in system 100. In the embodiment of FIG. 2, imaging device 104B is configured to capture an image of tube assembly 102 while it is on track 112. Tube assembly 202 is shown in imaging position 214 on track 112 of imaging device 104B. Components of imaging device 104B can be controlled by computer 220. In some embodiments, computer 220 can be incorporated into computer 120 (FIG. 1).

[0025] The imaging device 104B can include three imaging devices 216 configured to capture images of the tube assembly 202 at the imaging position 214. The imaging devices 216 are individually referred to as a first imaging device 216A, a second imaging device 216B, and a third imaging device 216C. In other embodiments, the imaging device 104B can include one or more imaging devices. The imaging devices 216 can be configured to generate image data representative of the tube assembly 202. The computer 220 and / or the computer 120 (FIG. 1) can be configured to process the image data generated by the imaging devices 216 as described herein.

[0026] The imaging device 104B may also include one or more illumination devices 217 configured to illuminate the imaging location 214. The illumination devices 217 may thus be configured to illuminate the tube assembly 202. In the embodiment of Figure 2, the imaging device 104B may include three illumination devices 217, individually referred to as a first illumination device 217A, a second illumination device 217B, and a third illumination device 217C.

[0027] 1 , system 100 may also include or be coupled to computer 120, which may be configured to operate and / or communicate with device 104. In some embodiments, computer 120 may analyze data received by system 100 and / or generated by device 104. Computer 120 and programs executed by computer 120 may also analyze image data generated by imaging device 104B and other imaging devices, as described herein. Computer 120 may include processor 122, which may be configured to execute computer-readable instructions (e.g., program code).

[0028] 1 embodiment, computer 120 may be coupled to each of the devices 104. For example, computer 120 may be coupled to computer 220 in imaging device 104B. In some embodiments, computer 120 may be coupled to one or more workstations (not shown) coupled to one or more of the devices 104. In some embodiments, computer 120 may be configured to transmit computer instructions and / or computer code to each of the devices 104. In some embodiments, computer 120 may be configured to receive and / or process data generated by devices 104.

[0029] The computer 120 can include or be coupled to a memory 124 that can store or access one or more modules and / or programs 126, which are described herein as being stored in the memory 124. The modules or programs 126 can be executable by the processor 122. The programs 126 can be configured to operate one or more of the devices 104. Although the programs are described as individual programs, in some embodiments, the programs 126 can be implemented as a single program. One or more of the programs 126 can execute an artificial intelligence (AI) algorithm. The programs 126 can include a data processing program 126A, which can be configured to process data in at least one of the devices 104, including the imaging device 104B. For example, the data processing program 126A can process data input to the system 100 and / or one or more of the devices 104. The data processor 126A may also process data generated by one or more of the instruments 104.

[0030] In some embodiments, the memory 124 can include a processing algorithm 126B. In some embodiments, the memory 124 can include two or more processing algorithms. In some embodiments, the two or more processing algorithms can be implemented as a single processing algorithm, such as the processing algorithm 126B. The processing algorithm 126B can include computer code configured to analyze image data and other data generated by an imaging device in or coupled to the system 100. In some embodiments, the processing algorithm 126B can process the data using AI, which can be implemented as a tube characterization model 126C that analyzes the image data as described herein. The tube characterization model 126C can be updated as described herein. For example, the image decomposition model 502 (FIG. 5) and / or the image synthesis model 504 (FIG. 5) can be updated as described herein.

[0031] The imaging device 216 and a computer (e.g., computer 120) used to process images captured by the imaging device 216 can be part of the machine vision and machine learning used by the system 100 to facilitate processing of the tube assembly 102. For example, as described herein, the machine vision and AI models that can be run in the tube characterization model 126C can identify the tube assembly 102 and / or a particular type of sample present in the tube assembly 102 to facilitate processing and analysis of the sample as described herein.

[0032] The computer 120 may be coupled to or implemented with a workstation 128. The workstation 128 may enable user interaction with the system 100 and / or one or more of the devices 104. The workstation 128 may include a display 128A and / or a keyboard 128B that enable a user to interact with individual ones of the system 100 and / or the devices 104.

[0033] In some embodiments, the system 100 can be coupled to a laboratory information system (LIS) 130 that can determine how a sample should be tested by the system 100. In some embodiments, the LIS 130 can be implemented on the computer 120. The LIS 130 can be coupled to a hospital information system (HIS) 132 that can accept specific assay instructions for a particular sample. The HIS 132 can also receive assay results after an assay has been performed on a particular sample by the system 100.

[0034] In one example of the operation of the system 100, a physician may prescribe a particular test to be performed on a sample from a patient. The physician may input the test order into the HIS 132. The HIS 132 may then transmit the test order to the LIS 130. The sample may be drawn from the patient and placed into a tube assembly, such as the tube assembly 202 (FIG. 2). The configuration of the tube assembly 202, such as color, size, and shape, may correspond to the type of test in the test order. The tube assembly 202 may then be sent to the system 100. The LIS 130 may determine how the sample is to be tested and may transmit that information to the computer 120, where instructions may be generated to operate the instrument 104 to perform the test. For example, the data processing program 126A may generate the instructions. The imaging device 216 (FIG. 2) may generate an image of the tube assembly 202, and based on the image data, the tube characterization model 126C may identify the type of tube assembly 202. The testing of the tube assembly 202 and its movement through the system 100 may be based at least in part on the type of tube assembly 202. After the testing is completed, the LIS 130 may transmit the test results to the HIS 132.

[0035] Reference is now made further to Figures 3A-3C, which show various types of tube assemblies that may be used within the system 100. The tube assemblies refer to tubes with or without caps attached. The tube assemblies may also include samples or other contents of the tube assembly. Reference is also made further to Figures 4A-4C, which show the tube assemblies of Figures 3A-3C without caps. As shown in each figure, all tube assemblies have various configurations or geometries. Each of the caps may have various cap geometries.

[0036] Tube assembly 340 in Fig. 3A includes cap 340A, which is white with red stripes and has a vertically extending portion. Cap 340A can fit over tube 340B. Tube assembly 340 has a height H31. Fig. 4A shows tube 340B without cap 340A. Tube 340B has a height H41 and a width W41.

[0037] Tube assembly 342 in Fig. 3B includes cap 342A, which is blue and has a dome-shaped top. Cap 342A can fit over tube 342B. Tube assembly 342 has a height H32. Fig. 4B shows tube 342B without cap 342A. Tube 342B has a height H42 and a width W42.

[0038] Tube assembly 344 in Fig. 3C includes cap 344A, which is red and gray and has a flat top. Cap 344A can fit onto tube 344B. Tube assembly 344 has a height H33. Fig. 4C shows tube 344B without cap 344A. Tube 344B has a height H43 and a width W43.

[0039] 3A-4C show various types of tubes, caps, and tube assemblies that can be used in the system 100. The tube characterization model 126C is trained to characterize or identify the tubes, caps, and tube assemblies of FIGS. 3A-4C. In some embodiments, the tube characterization model 126C can also determine if a tube assembly cannot be characterized. In some embodiments, the tubes can have shapes other than the cylindrical shapes of FIGS. 3A-4C and can be characterized by the tube characterization model 126C. Additionally, the tubes and caps can be made from various materials and have various shapes, structures, albedo, and textures, which can be characterized by the tube characterization model 126C.

[0040] Over time, a laboratory may utilize tube assemblies from a source that provides tube assemblies that deviate from the manufacturer-supplied tube assemblies that the tube characterization model 126C was trained on. For example, a laboratory may use tube assemblies supplied by a local manufacturer or new tube assembly designs that are not stored in the tube characterization model 126C. Training data may be limited, limiting the number of tube differences that the tube characterization model 126C can identify or characterize. Retraining the tube characterization model 126C to handle different tube types typically requires training a large number of images showing various configurations of tube assemblies under different viewpoints and lighting conditions. In some conventional embodiments, training is performed manually, which makes the cost of retraining prohibitively high.

[0041] The embodiments disclosed herein integrate tube assembly images to generate an integrated tube assembly image. In one example, the tube characterization model 126C is trained to characterize a particular type of tube assembly that can be used for a particular inspection. The tube assembly can have a particular tube material, tube color, tube shape, cap shape, cap color, and other unique attributes. Over time, one or more manufacturers may change those attributes of the tube assembly. The embodiments disclosed herein enable the system 100, such as the processing algorithm 126B, to integrate the images of the tube assembly with new attributes, and those integrated images can then be used by the tube characterization model 126C to characterize new tube assemblies.

[0042] In another example, the system 100 may receive a new tube assembly type from a new manufacturer. Each attribute of the new tube assembly type may be similar to the attributes of the particular tube assembly on which the tube characterization model 126C was trained. For example, the new tube assembly may have the same tube material as the tube assembly on which the tube characterization model 126C was trained, but a different cap shape. Meanwhile, another tube assembly type on which the tube characterization model 126C was trained may have the same cap type as the new tube assembly type, but a different tube material. The system 100 may use the tube characterization model 126C to integrate images of the new tube assembly with a different cap shape. In some embodiments, the system 100 may integrate images using a similar cap shape of another type of tube assembly. In other embodiments, the system 100 may integrate the cap shape of the new tube assembly.

[0043] Some embodiments disclosed herein can use image-to-image transformation of unpaired or paired images to apply existing data (e.g., image data) from existing tube assembly types to a new tube assembly type. The existing data can be images used to train the tube characterization model 126C. The image-to-image transformation can be performed using machine learning, such as deep learning (DL) methods. An example of machine learning includes generative adversarial networks (GANs). An example of a GAN is cycleGAN, which performs image-to-image transformation of unpaired images using a cycle-consistent adversarial network. Some embodiments decompose a captured image of a tube assembly into decomposed features. One or more of the decomposed features can be transformed into an integrated tube assembly image or a portion of an integrated tube assembly. Thus, the systems and methods disclosed herein enable the collection of data (e.g., images) from existing tube assembly configurations to create integrated images of different tube assemblies.

[0044] Further reference is made to FIG. 5. FIG. 5 illustrates a block diagram of a process 500 for implementing a method for integrating images of a tube assembly as described herein. The images referred to in FIG. 5 may be image data, such as image data generated by the imaging device 104B. The process 500 uses integrated data or images of pairs of tube assemblies with controlled differences to train an image decomposition model 502 and an image synthesis model 504. The trained image decomposition model 502 can then receive an input image of a tube assembly and decompose the input image into a latent space 506 having a plurality of features. The latent space manipulation module 508 can manipulate one or more features of the image in the latent space 506. For example, the latent space manipulation module 508 can manipulate one or more features in the latent space related to attributes, such as the color of the cap, of the image of the tube assembly. The manipulated features can then be used by the image synthesis model 504 to generate an integrated image of the tube assembly.

[0045] 6 shows a block diagram of a process 600 that implements a method of integrating images of a tube assembly exterior and an interior of the tube assembly, according to one or more embodiments. Integrating the interior of the tube assembly can integrate the fluid in the tube assembly. In some embodiments, the process 600 can be considered to be two separate processes.

[0046] The process 600 may include decomposing the image using the image decomposition model 502 described in FIG. 5, which decomposes the input image into various features. The process 600 may use a tube exterior latent space 610 configured to characterize the exterior attributes of the image of the tube assembly. The tube exterior latent space 610 may include individual variables or features 612 that may be configured to separate different attributes of the image of the exterior of the tube assembly that are decomposed by the image decomposition model 502. The features 612 may separate the image based on attributes such as the geometry and material of the tube assembly, the surface characteristics of the tube assembly, the lighting conditions when the image was captured, the camera characteristics of the imaging device (e.g., imaging device 216 of FIG. 2), etc. Other attributes of the image may be separated or characterized.

[0047] The tube exterior latent space 610 can be manipulated by a latent space manipulation module 614. The latent space manipulation module 614 can be the same as or substantially similar to the latent space manipulation module 508 of Figure 5. The latent space manipulation module 614 can manipulate the features 612 that are disjointed in the tube exterior latent space 610. The image synthesis model 504 can then synthesize an image using the manipulated features manipulated by the latent space manipulation module 614.

[0048] The process 600 can include decomposing the image of the tube interior or fluid using the image decomposition model 502 described in FIG. 5. The process 600 can use a tube interior latent space 618 configured to integrate interior attributes of the image of the tube assembly. In some embodiments, the tube interior latent space 618 can characterize the contents of the tube assembly, such as a sample contained in the tube assembly, into features 620. The tube interior latent space 618 can include individual variables or features 620, which can include the geometry and material of the tube assembly, fluid properties, lighting conditions when the image was captured, camera properties of the imaging device (e.g., imaging device 216 of FIG. 2), label and / or barcode properties. Other attributes of the image can be isolated or characterized into features 620.

[0049] The features 620 in the tube interior latent space 618 can be manipulated by a latent space manipulation module 624. The latent space manipulation module 624 can be the same as or substantially similar to the latent space manipulation module 614 or the latent space manipulation module 508 of FIG. 5. The latent space manipulation module 624 can manipulate the disjointed features 620 in the tube interior latent space 618. The image synthesis model 504 can then use the manipulated features to synthesize a unified image of the tube assembly.

[0050] Further reference is made to Figure 7, which is an embodiment of a training process 700 that can be used to train the image decomposition model 502 and the image synthesis model 504. A first image A of a first tube assembly is input to an encoder network. The encoder network calculates the latent variables L AThe decomposition model 730 may be similar to or identical to the decomposition model 530. In some embodiments, the decomposition model 730 generates latent features in a low-dimensional space. The latent variables L A The features of can be divided into multiple groups of features, such as W, X, Y, and Z, as shown in FIG. 7. Each feature can correspond to a particular one of the attributes of the tube assembly. In some embodiments, at least one of the features (e.g., feature Z) can be reserved for an intrinsic property of the image of the tube assembly whose differences cannot be controlled by the setup. In some examples, all the tubes in a study can all have a common tubular structure, and thus are expected to all share the same value of feature Z.

[0051] The synthesis model 732 (sometimes referred to as an image synthesis model), which may be implemented as a decoder network, is A The features of the image A can be used to reconstruct an image A' of the tube assembly that is as similar as possible to the input image A. The synthesis model 732 can be similar or identical to the synthesis model 532. The reconstruction loss module 734 can compare the input image A to the output image A' to determine whether the images are close to each other. For example, the reconstruction loss module 734 analyzes the attributes of the image A and the image A' and calculates a latent variable L A is correct, and whether the decomposition model 730 and the composition model 732 are properly trained.

[0052] The training process 700 includes receiving a second input image B of a second tube assembly at the decomposition model 730 and calculating a latent variable L in the multi-dimensional space. B The images A and B may be a pair of tube assembly images. BThe features of can be divided into multiple groups of features, such as W', X', Y', and Z', as shown in FIG. 7. Each feature can correspond to a particular one of the attributes of the tube assembly. At least one of the features (e.g., feature Z') can be reserved for a unique characteristic of the tube image that cannot be captured by one of the features. The synthesis model 732 can use the features to reconstruct an image B' of the tube assembly that is as similar as possible to the input image B. The reconstruction loss module 736 can compare the input image B with the output image B' to determine whether the images are close to each other. For example, the reconstruction loss module 736 can analyze the attributes of image B and image B' and calculate a latent variable L B is correct, and whether the decomposition model 730 and the composition model 732 are properly trained.

[0053] Training a disentangled latent space, such as that used for the decomposition model 730 and the synthesis model 732, can involve making input image B a particular difference to input image A. For example, when synthesizing the cap of a tube, the latent variable L A and the latent variable L B The first feature quantity (e.g., W and W') of can be an albedo, such as the color of the cap of the tube. In this example, input image B is of the same tube assembly type as input image A, with only the difference being the color of the tube cap. Input image B can be synthetically generated, such as generated using computer-generated imagery (CGI), or captured from the same type of tube assembly and the same imaging setup used to obtain input image A. The caps of the tubes in input image B can be very similar to the caps in input image A, with the only difference, or the only significant difference, being the difference in the color of the cap. The latent space L A and L BIn, the remaining features (e.g., X and X', Y and Y', Z and Z') should be as similar as possible while allowing the first features W and W' to be different as explained above.

[0054] The above process is performed on the latent space L A and L B , so that each of the latent feature variables captures a particular difference. In some embodiments, it is possible to have several differences between input image A and input image B at the same time. In such embodiments, corresponding latent feature variables can be allowed to differ while remaining features are kept consistent. As an example, input image A and input image B may be captured with the same camera but contain tube assemblies with different tube geometries, surface properties, and lighting conditions. In this embodiment, only the latent feature variables corresponding to the camera properties are forced to be similar, while other features may differ.

[0055] The architecture of the training process 700 includes a decomposition model 730 and a composition model 732. Thus, training can be augmented by training on unpaired images to minimize the reconstruction loss alone. Such images can also be edited in the latent domain by replacing some of their latent features with latent features borrowed from another image. The decomposition model 730 and composition model 732 can then be trained by an adversarial network that tries to classify edited and unedited images. For example, under lighting conditions L provided by the first lighting device 217A, 1 A first imaging device C using 1 A set of tube assembly images S captured by (e.g., the first imaging device 216A in FIG. 2 ). 1 There can be an image S corresponding to the camera characteristics of the first imaging device 216A and the lighting conditions provided by the first lighting device 217A. 1The latent features of can be similar to each other. The two latent features are expressed by a mean vector (m C1 ,m L1 ) can be normalized to approximate a unit Gaussian distribution with the illumination condition L 2 A second imaging device C using 2 Another set of tube assembly images S are captured by (e.g., a second imaging device 216B). 2 In this case, the latent features related to the camera and lighting conditions are expressed as the mean vector (m C2 ,m L2 )

[0056] S 1 The latent features of the tube image are expressed as a vector (m C2 ,m L2 ) and reconstructing the image of the tube assembly using the synthetic model 732, 2 (second imaging device 216B) and L 2 The same tube captured with (second illumination device 217B) is merged. In some embodiments, the adversarial classifier is further trained to recognize that the merged image in terms of camera and illumination is S 2 Similarly, we can enforce the rule that images of S should be indistinguishable from images of S. 2 The latent features of the tube assembly image are calculated by (m C1 ,m L1 ) and replace it with C 1 and L 1 These processes ensure that image differences are disentangled by latent space (e.g., latent space 610 and tube interior latent space 618). After training, the trained image decomposition model 730 learns how to decompose images into a disentangled latent space, and the trained image synthesis model 732 synthesizes the latent variables L A or L BWe learn how to perform image synthesis from latent variables such as one of the latent vectors, i.e., θ = θ ...

[0057] In some embodiments, the latent space representation can encode data across different scales (e.g., decompose an image). Examples of generative deep learning include, for example, normalized flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models. Additionally, the latent space can be restricted to non-parametric distributions with known parameter distributions (e.g., Gaussian or Gaussian mixtures) or vector quantization variational autoencoders (VQ-VAEs), etc.

[0058] Additionally, images other than the generated images can be used as image data to characterize attributes such as tube cap classification and tube fluid characterization. The disentangled features in the latent space can also provide useful information (used as ground truth annotations or treated as discriminatory features) for downstream tasks that can be performed by the system 100. Furthermore, the disentangled features in the latent space can be used to create tools that assess novel tube assembly types and tube assembly differences to determine the ability of the system 100 (FIG. 1) to properly characterize tube assemblies, which can be used by laboratory personnel such as suppliers, research and development staff, field service technicians, and / or end users (e.g., lab technicians and managers).

[0059] In some embodiments, system 100 can use unpaired image-to-image transformation, such as cycleGAN, to apply existing data from previous tube assembly images to images of new tube assemblies. Such a procedure allows system 100 and / or developers of tube assemblies to collect field data from existing tube assemblies as they create new types of tube assemblies.

[0060] In some embodiments, image data can be collected in groups that can include a high density collection of typical tube assembly types and a low density collection of other similar tube assembly types. Augmented data can be used to fill in missing attributes in the low density collected tube assembly type images. For example, color augmentation can be used to fill in missing colors in the low density collected tube assembly type images.

[0061] An example of the system 100 for integrating tube assemblies is described below. Please refer to Figures 8A and 8B, which show images of similar tube assemblies. Tube assembly 840 in Figure 8A is the same as or similar to tube assembly 842 in Figure 8B, except that cap 840A of tube assembly 840 is blue in color and cap 842A is green in color. For example, tube 840B can be similar to tube 842B, and both tube assembly 840 and tube assembly 842 can have the same height H81. Decomposition model 730 can receive an image of tube assembly 840 as input image A. Decomposition model 730 can also receive an image of tube assembly 842 as input image B. Decomposition model 730 can convert input image A into a variable L A We decompose image B into a set of features (W, X, Y, Z) in the latent space represented as B We decompose the feature vectors into a set of features (W', X', Y', Z') in the same latent space, represented as

[0062] Further, reference is made to FIG. 8C, which provides an example of a decomposition image 844 of a tube assembly 840. It is noted that other decompositions of the image of the tube assembly 840 can be performed. In the embodiment of FIG. 8C, feature W can be the color of the cap, which is blue in color. Feature X can be the height H82 of the cap, feature Y can be the height H83 of the tube, and feature Z can be the width W81 of the tube. Other features can include other parts of the geometry of the tube or the geometry of the tube assembly. The synthesis model 732 can reconstruct the disassembled image to generate an image A'. The input image A can be compared to the reconstructed image A' by the reconstruction loss module 734. If the loss or difference between the input image A and the reconstructed image A' is within a predetermined amount, the decomposition model 730 and the synthesis model 732 can be considered to be properly trained, and the latent variable L A is suitable for processing the tube assembly 840.

[0063] Tube assembly 842 can be selected as a pair image because the only significant difference between tube assembly 840 and tube assembly 842 is the color of the cap. The decomposition model 730 maps image B of tube assembly 842 to a latent space L B The latent space L B The features of are stored in the latent space L A The feature W' may be the color of the cap, which is green, the feature X' may be the height H82 of the cap, the feature Y' may be the height H83 of the tube, and the feature Z' may be the width W81 of the tube. The synthesis model 732 may synthesize the disassembled images to generate an image B'. The input image B may be compared to the reconstructed image B' by the reconstruction loss module 736. If the loss or difference between the input image B and the reconstructed image B' is within a predetermined amount, the disassembly model 730 and synthesis model 732 may be considered properly trained and the latent variable L B is suitable for processing the tube assembly 842.

[0064] The green cap 842A can be the new cap color. To reconstruct the tube assembly 842 with the blue cap, feature W is used instead of feature W', which is the green cap. Image-to-image reconstruction and other methods can be used to reconstruct the tube assembly 842 with the blue cap. The resulting reconstructed or merged image using features W, X', Y', and Z' is the tube assembly 842A with the blue cap of the tube assembly 840. In some embodiments, the entire cap 840A can be used in place of the cap 842A, and in other embodiments, the color of the cap 840A can be used in place of the color of the cap 842A. In other embodiments, the color of the cap 840A can be stored and used for the cap 842A without imaging the tube assembly 840.

[0065] Reference is now made to Figures 9A-9B, which show tube assemblies having different interior and / or fluid features that may be used in the tube interior potential space 618 (Figure 6). The tube assembly 940 may include one or more items (e.g., attributes) that are inside the tube 940B and may be sealed by a cap 940A. One or more fluids 946 may be inside the tube 940B and may include serum 946A, a gel separator 946B, and blood 946C. A void 946D may be above the fluid 946.

[0066] Tube assembly 942 may include one or more items (e.g., attributes) that are inside tube 942B and may be sealed by cap 942A. One or more fluids 948 may be inside tube 942B and may include serum 948A, gel separator 948B, and blood 948C. A void 948D may be above fluid 948. The material of tube 940B and the material of tube 942B may be similar and may be identified by generating image data through void 946D and void 948D. Thus, attributes of tube assembly 940 and tube assembly 942 include, but are not limited to, tube geometry, tube material, tube color, tube surface characteristics, sample fluid within the tube, label affixed to the tube, and lighting conditions of the tube during image capture.

[0067] All of the above attributes of tube assembly 940 and tube assembly 942 may be the same or substantially similar. The difference between tube assembly 940 and tube assembly 942 may be the labels attached to tubes 940B and 942B. Tube assembly 940 has label 950 affixed to tube 940B, and tube assembly 942 has label 952 affixed to tube 942B. As shown in FIGS. 9A-9B, label 950 is different from label 952. For example, the size of label 950 is different from the size of label 952, and the location of barcode 954 is different from the location of barcode 956.

[0068] The decomposition model 730 and the composition model 732 can be configured to identify and / or characterize attributes of the images of the tube assemblies 940 and 942 into features in a latent space. Accordingly, the system 100 can be configured to substitute the features into the integrated image (e.g., image A' or B'). In the example of FIGS. 9A-9B, the system 100 can use the labels of the original images in place of either the labels 950 or the labels 952 to generate the integrated image. In some embodiments, the system 100 can replace the labels and maintain the original barcode or other label information of the integrated image.

[0069] In some embodiments, the integrated tube assembly image is used by the system 100 during the tube assembly identification process. For example, an image of the tube assembly 102 can be captured by the imaging device 116 or another imaging device when the tube assembly 102 is loaded into the input / output device 104A. The integrated tube assembly image can be used to identify tube assemblies that were not initially recognized by the tube characterization model 126C. The tube assemblies can then be characterized and transported to an appropriate instrument 104 where the samples are analyzed as described herein. In some embodiments, the instrument 104 can perform processing, such as suction, that is customized for the identified tube assembly. In other embodiments, the robot 108 (FIG. 1) can move sample containers to and from the carrier 114 using specific techniques, such as specific gripping techniques, depending on the type of tube assembly identified.

[0070] Reference is now made to FIG. 10. FIG. 10 is a flow chart illustrating a method 1000 of synthesizing an image of a tube assembly (e.g., tube assembly 202). Method 1000 includes, at 1002, capturing an image of the tube assembly to generate a captured image. Method 1000 includes, at 1004, decomposing the captured image into a plurality of features (e.g., features 612) in a latent space (e.g., latent space 610) using a trained image decomposition model (e.g., image decomposition model 502). Method 1000 includes, at 1006, manipulating one or more of the features in the latent space into manipulated features. For example, the color of the cap can be changed from red to blue, and / or the shape of the cap can be changed from round to oval. Other features can additionally or alternatively be modified (i.e., manipulated). The method 1000 includes, at 1008, generating an integrated tube assembly image having at least one of the manipulated features using a trained image synthesis model (e.g., image synthesis model 504).

[0071] Reference is now made to FIG. 11. FIG. 11 is a flow chart illustrating a method 1100 of integrating an image of a tube assembly (e.g., tube assembly 202). The method 1100 includes, at 1102, receiving an input image of a tube assembly and constructing an image decomposition model (e.g., image decomposition model 502) configured to decompose the input image into a plurality of features (e.g., feature 612) in a latent space (e.g., latent space 610). The method 1100 includes, at 1104, constructing an image synthesis model (e.g., image synthesis model 504) configured to synthesize an integrated tube assembly image based on the plurality of features in the latent space. The method 1100 includes, at 1106, training the image decomposition model and the image synthesis model using at least an image of a first tube assembly (e.g., tube assembly 840) and an image of a second tube assembly (e.g., tube assembly 842) having one or more known differences between the image of the first tube assembly and the image of the second tube assembly, the training producing a trained image decomposition model and a trained image synthesis model.

[0072] While the disclosure is susceptible to various modifications and alternative forms, specific method and apparatus embodiments have been shown by way of example in the drawings and are herein described in detail. It is to be understood, however, that the specific methods and apparatus disclosed herein are not intended to limit the disclosure, but on the contrary include all modifications, equivalents, and alternatives falling within the scope of the appended claims.

Claims

1. 1. A method for integrating an image of a tube assembly, comprising: capturing an image of the tube assembly to generate a captured image; Decomposing the captured image into a plurality of features in a latent space using the trained image decomposition model; Manipulating one or more of the features in the latent space into manipulated features; generating an integrated tube assembly image having at least one of the manipulated features using the trained image synthesis model; The method comprising:

2. 2. The method of claim 1, further comprising training an image decomposition model and an image synthesis model using at least an image of the first tube assembly and an image of the second tube assembly, where there are one or more known differences between the image of the first tube assembly and the image of the second tube assembly, and the training produces a trained image decomposition model and a trained image synthesis model.

3. The method of claim 2 , wherein training the decomposition model is based at least in part on a reconstruction loss between the integrated image and the input image.

4. The method of claim 1 , wherein the tube assembly includes a tube and a cap, and at least one of the plurality of features is a tube geometry, a tube color, a surface characteristic of the tube, a cap geometry, a cap color, or a lighting condition.

5. 2. The method of claim 1, wherein the tube assembly includes a tube, and at least one of the plurality of features is a geometry of the tube, a material of the tube, a sample fluid in the tube, a label affixed to the tube, or an illumination condition of the tube.

6. 1. A method for integrating an image of a tube assembly, comprising: receiving an input image of a tube assembly and constructing an image decomposition model configured to decompose the input image into a plurality of features in a latent space; constructing an image synthesis model configured to synthesize an integrated tube assembly image based on a plurality of features in a latent space; The method includes training an image decomposition model and an image synthesis model using at least an image of a first tube assembly and an image of a second tube assembly, where the image of the first tube assembly and the image of the second tube assembly have one or more known differences between the images of the first tube assembly and the images of the second tube assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

7. capturing an image of the tube assembly to generate a captured image; Decomposing the captured image into a plurality of decomposed features in a latent space using the trained image decomposition model; Manipulating one or more of the decomposed features in the latent space into manipulated features; generating an integrated tube assembly image having at least one of the manipulated features using the trained image synthesis model; The method of claim 6 further comprising:

8. The method of claim 6 , wherein training the decomposition model is based at least in part on a reconstruction loss between the integrated image and the input image.

9. The method of claim 6 , wherein the image synthesis model is trained based on a reconstruction loss between the integrated image and the input image.

10. The method of claim 6 , wherein at least one of the features in the latent space is generated using computer-generated imagery.

11. The method of claim 6 , wherein at least two images of the tube assembly differ from each other with respect to one or more controlled attributes.

12. The method of claim 6 , wherein the images of at least two tube assemblies share one or more attributes.

13. The method of claim 6 , wherein at least one of the image decomposition model or the image synthesis model is trained based on the reconstruction loss without images of the tube assembly.

14. 7. The method of claim 6, wherein the tube assembly includes a tube and a cap, and at least one of the features in the latent space is a tube geometry, a tube color, a surface property of the tube, a cap color, a cap geometry, or a lighting condition.

15. 7. The method of claim 6, wherein the tube assembly includes a tube, and at least one of the features in the latent space is a geometry of the tube, a material of the tube, a sample fluid in the tube, a label affixed to the tube, or a lighting condition of the tube.

16. The method of claim 6 , wherein the one or more features in the latent space include fluid properties in the tube assembly.

17. 1. A diagnostic laboratory system comprising: an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; and an image synthesis model configured to synthesize an integrated tube assembly image based on a plurality of features in a latent space. Including, The diagnostic laboratory system, wherein the image decomposition model and the image synthesis model are trained using at least images of a first tube assembly and images of a second tube assembly, where there are one or more known differences between the images of the first tube assembly and the images of the second tube assembly.

18. 20. The diagnostic laboratory system of claim 17, wherein the image decomposition model is trained based at least in part on a reconstruction loss between the integrated image and the input image.

19. 20. The diagnostic laboratory system of claim 17, wherein the image synthesis model is trained based on a reconstruction loss between the integrated image and the input image.

20. 20. The diagnostic laboratory system of claim 17, wherein the tube assembly includes a tube and a cap, and at least one of the plurality of features in the latent space is a tube geometry, a tube color, a surface property of the tube, a cap color, a cap geometry, or a lighting condition.

Images