Diagnostic laboratory system and method of imaging tube assembly

Through image decomposition and synthesis model technology, the problem of high recognition cost of new tube components in the diagnostic laboratory system is solved, and the rapid identification and processing of tube components is achieved, reducing training costs and improving adaptability.

CN118922867APending Publication Date: 2024-11-08SIEMENS HEALTHCARE DIAGNOSTICS INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202380025022.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-03
Filing Date
2023-03-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing diagnostic laboratory systems require frequent retraining of machine learning models to identify and process different types of tube components when facing new tube components, resulting in high training costs and difficulty in adapting to all possible tube component configurations worldwide. .

Method used

Through image decomposition and synthesis model, the image of the tube assembly is decomposed into features in hidden space, and these features are used to generate comprehensive tube assembly images, thereby training the image decomposition and synthesis model to adapt to different tube assembly types and achieving rapid tube assembly Identify and process.

Benefits of technology

Reduces the need for retraining machine learning models, improves the recognition of new tube components, reduces the operating costs of laboratory systems, and improves global adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922867A_ABST
    Figure CN118922867A_ABST
Patent Text Reader

Abstract

A method of synthesizing an image of a tube assembly includes capturing an image of the tube assembly, wherein the capturing generates a captured image. The captured image is decomposed into a plurality of features in a hidden space using a trained image decomposition model. One or more of the features in the hidden space are manipulated into one or more manipulated features. A synthesized tube assembly image having at least one of the manipulated features is generated using the trained image synthesis model. Other methods and systems are disclosed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 268,846, filed on March 3, 2022, entitled “DIAGNOSTIC LABORATORY SYSTEMS AND METHODS OF IMAGING TUBE ASSEMBLIES,” the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] Embodiments of the present disclosure relate to diagnostic laboratory systems and methods of imaging tube assemblies in diagnostic laboratory systems. Background Art

[0003] Diagnostic laboratory systems perform clinical chemistry or assays to identify analytes or other components in biological samples, such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. The sample may be received in the system in a sample tube assembly and / or transported throughout the system in a sample tube assembly.

[0004] Many of the laboratory systems process large volumes of tube assemblies and samples in the tube assemblies. Some laboratory systems utilize machine vision and machine learning to facilitate sample processing and tube assembly identification (e.g., characterization). For example, vision-based machine learning models (e.g., AI models) have been employed to provide a fast and non-invasive method for tube assembly identification and fluid characterization. However, the training cost for adding new tube assemblies may be excessive because a large amount of training data may be required to retrain or adapt the machine learning model to characterize the new tube assembly.

[0005] Tube assembly manufacturers continuously produce new tube assembly configurations to introduce new features or save on production costs. Additionally, third-party low-cost tube assembly alternatives may be utilized by some laboratories due to availability and / or financial reasons. With so many tube assembly configurations, it is difficult to collect every possible tube assembly variation on a global scale, which means that it is likely that a machine learning model cannot be trained on all tube assembly configurations. Accordingly, there is a limit to the number of tube assembly configurations that a machine learning model can actually handle. Therefore, there is a need for laboratory systems and methods that facilitate the introduction of new tube assembly configurations into a laboratory system. Summary of the invention

[0006] According to a first aspect, a method for synthesizing an image of a pipe assembly is provided. The method includes: capturing an image of the pipe assembly, the capturing generating a captured image; decomposing the captured image into a plurality of features in a latent space using a trained image decomposition model; manipulating one or more of the features in the latent space into manipulated features; and using the trained image synthesis model to generate a synthesized pipe assembly image having at least one of the manipulated features.

[0007] In another aspect, a method for synthesizing an image of a pipe assembly is provided. The method includes: constructing an image decomposition model configured to receive an input image of a pipe assembly and decompose the input image into a plurality of features in a latent space; constructing an image synthesis model configured to synthesize a synthesized pipe assembly image based on the plurality of features in the latent space; and training the image decomposition model and the image synthesis model using at least an image of a first pipe assembly and an image of a second pipe assembly, wherein there are one or more known changes between the image of the first pipe assembly and the image of the second pipe assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

[0008] In another aspect, a diagnostic laboratory system is provided. The diagnostic laboratory system includes an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; and an image synthesis model configured to synthesize a synthetic tube assembly image based on the plurality of features in the latent space, wherein the image decomposition model and the image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, wherein there are one or more known changes between the images of the first tube assembly and the second tube assembly.

[0009] Still other aspects, features and advantages of the present disclosure may be apparent from the following description and illustration of a plurality of example embodiments, including the best modes contemplated for implementing the present disclosure. The present disclosure may also have the capability of other and different embodiments, and may modify its several details in various aspects, all without departing from the scope of the present disclosure. The present disclosure is intended to cover all modifications, equivalents and alternatives that fall within the scope of the claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described below are provided for illustration purposes and are not necessarily drawn to scale. Accordingly, the drawings and descriptions should be regarded as illustrative in nature and not as restrictive. The drawings are not intended to limit the scope of the present disclosure in any way.

[0011] Figure 1Illustrated is a block diagram of a diagnostic laboratory system including multiple instruments according to one or more embodiments.

[0012] Figure 2 Illustrated is a top view of an imaging instrument provided in a diagnostic laboratory system according to one or more embodiments.

[0013] Figures 3A-3C Different types of tube assemblies including caps affixed to tubes that may be used within a diagnostic laboratory system are illustrated in accordance with one or more embodiments.

[0014] Figures 4A-4C Different types of tubes of a tube assembly that may be used within a diagnostic laboratory system are illustrated in accordance with one or more embodiments.

[0015] Figure 5 A block diagram illustrating a process for implementing a method of synthesizing images of a tube assembly according to one or more embodiments is illustrated.

[0016] Figure 6 A block diagram illustrating a process for implementing a method of integrating images of an exterior and interior of a tube assembly in accordance with one or more embodiments is illustrated.

[0017] Figure 7 Illustrated is a process that may be used to train an image decomposition model and an image synthesis model in accordance with one or more embodiments.

[0018] Fig. 8A and 8B An image of a similar tube assembly is shown, where Fig. 8A The image has blue caps and Figure 8B The image has a green cap.

[0019] Figure 8C The present invention illustrates a method according to one or more embodiments. Fig. 8A Example of an exploded image of a tube assembly.

[0020] Figures 9A-9B Illustrated are images of tube assemblies having different internal and / or fluid characteristics according to one or more embodiments.

[0021] Fig.10 is a flow chart illustrating a method of synthesizing images of a tube assembly according to one or more embodiments.

[0022] Fig.11 is a flow chart illustrating another method of synthesizing images of a tube assembly in accordance with one or more embodiments. DETAILED DESCRIPTION

[0023] The diagnostic laboratory system performs clinical chemistry and / or assays to identify analytes or other components in biological samples, such as serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc. The sample is collected in a tube assembly (e.g., a sample tube) and transported via the tube assembly to a laboratory system where the sample is analyzed. In some embodiments, the tube assembly is transported throughout the laboratory system so that various instruments can perform the analysis.

[0024] The laboratory system can use tube assemblies from various manufacturers to collect samples and transport samples to and throughout the laboratory system. The tube assembly can include a sample tube, such as a bottom closed tube. Some tube assemblies can include a cap that seals the tube. The tube assembly can also include the content of the tube. Different tube assembly types can have different characteristics, such as different sizes and different chemical additives therein. For example, many tube assembly types are chemically active, which means that the tube assembly contains one or more additive chemicals for changing or retaining the state of the sample or otherwise assisting the sample processing. In certain embodiments, the inner wall of the tube can be coated with the one or more additives, or the additive can be provided elsewhere in the tube. The type of additive included in the tube can be a serum separator, a coagulant (such as, thrombin), an anticoagulant (such as, EDTA or sodium citrate), an anti-sugar additive, or other additives for changing or retaining the characteristics of the sample. The tube assembly manufacturer can associate the color of the cap on the tube and / or the shape of the tube or cap with the specific type of chemical additive included in the tube.

[0025] Different manufacturers may have their own standards for associating the attributes of a tube assembly (such as cap color, cap shape, and tube shape) with specific properties of the tube assembly. For example, the attributes may be related to the contents of the tube, or may be related to whether the tube is provided with vacuum capability. In some embodiments, a manufacturer may associate all tube assemblies with gray caps with tubes including potassium oxalate and sodium chloride configured to test glucose and lactate. Tube assemblies with green caps may include heparin for electrostatic electrolytes such as sodium, potassium, chloride, and bicarbonate. Tube assemblies with lavender caps may identify tubes containing EDTA (ethylenediaminetetraacetic acid-anticoagulant) that are configured to test CBC w / diff., HgBA1c, and parathyroid hormone. Other cap colors such as red, yellow, light blue, royal blue, pink, orange, and black may be used to symbolize other additives or the lack of additives. In other embodiments, a combination of cap colors may be used, such as: yellow and lavender to indicate a combination of EDTA and a gel separator; or green and yellow to indicate lithium heparin and a gel separator. The laboratory system can use the tube assembly properties for further processing of the tube and / or sample. Since the tube can be chemically active, it is important to associate a specific test that can be performed on the sample with a specific tube assembly type because the test can depend on the contents of the tube. Thus, the laboratory can confirm that the test being run on the sample in the tube is correct by analyzing the color and / or shape of the cap and / or tube.

[0026] The transport mechanism, such as a robot and a tube assembly carrier within the laboratory system, may have specific hardware and processes for moving different types of tube assemblies. For example, a robot may grip a first type of tube assembly differently than a second type of tube assembly. Additionally, the laboratory system may utilize different carriers depending on the type of tube assembly to be transported throughout the laboratory system. Therefore, the laboratory system needs to identify the tube assemblies.

[0027] The laboratory system described herein may use a vision system to capture images of tube assemblies and identify tube assemblies and / or the contents of tube assemblies. For example, the laboratory system may include: a vision-based artificial intelligence (AI) model configured to provide a fast and non-invasive method for tube assembly characterization. Embodiments of the AI ​​model can be trained to characterize different types of tube assemblies and slight variations of tubes and / or caps. As new types of tube assemblies are introduced into the laboratory system, the AI ​​model needs to be updated to be able to classify the new types of tube assemblies. Retraining the AI ​​model in a traditional laboratory system can be costly and time consuming because multiple tube assembly types need to be imaged and manually classified to retrain the AI ​​model.

[0028] The systems and methods described herein overcome the problem of tube assembly classification by synthesizing images of tube assemblies. Synthetic data and / or paired images of tube assemblies can be used with controlled changes in the images to train image decomposition models and image synthesis models. The trained decomposition model is used to decompose the image of the tube assembly into features (e.g., decomposed features) in a latent space. One or more of the features in the latent space can then be manipulated. The trained synthesis model can reassemble features including the manipulated features to generate a synthesized tube assembly image.

[0029] In some embodiments, image-to-image translation is used by decomposition models and synthesis models to synthesize images of tube assemblies, where the new or synthesized image is a controlled modification of a reference image. Image-to-image translation is a class of vision and graphics processing in which the goal is to learn a mapping between an input image of a tube assembly and an output image of the tube assembly using a training set of aligned image pairs. In some embodiments, processing may include using a generative adversarial network (GAN) to perform image-to-image translation. Figure 1-11 These and other systems and methods are described in greater detail below.

[0030] Reference now Figure 1 , Figure 1 An example embodiment of an automated diagnostic system 100 configured to process and / or analyze a biological sample stored in a tube assembly 102 (e.g., a sample container) is illustrated. The tube assembly 102 can be any suitable container (e.g., a tube), including a transparent or translucent container, such as a blood collection tube, a test tube, a sample cup, a vial, or other container capable of containing a sample and / or allowing imaging of the sample contained therein. The tube assembly may or may not have a cap or cover attached to it. As described herein, the tube assembly 102 may have different sizes and may have different cap colors and / or cap types. As described herein, the system 100 and the methods described herein enable the system 100 or a device coupled to the system 100 to generate an image that integrates different tube assemblies.

[0031] The system 100 may include a plurality of instruments 104 configured to process tube assemblies 102 and / or samples located in the tube assemblies 102. The tube assemblies 102 may be received at the system 100 at an input / output instrument 104A that may include one or more racks 106 and a robot 108. The robot 108 may transport the tube assemblies 102 between the racks 106 and a carrier 114 located on a track 112. The carrier 114 is configured to transport the tube assemblies 102 on the track 112.

[0032] The instrument 104 can process the sample and / or tube assembly 102. Processing can include pre-processing or pre-screening the sample and / or tube assembly 102 before being analyzed by one or more of the instruments 104. For example, pre-processing can prepare the sample for a specific assay or other analysis. Pre-processing can also identify the type of tube assembly 102 to prepare the sample for analysis in response to the identification. Other instruments in the instrument 104 can perform analysis on the sample.

[0033] One or more of the instruments 104 may include an imaging device that captures an image of the tube assembly 102 and / or the carrier 114. Figure 1 In some embodiments, imaging apparatus 104B may include an imaging device as described below and may be configured to capture images of tube assembly 102 and / or carrier 114. In some embodiments, imaging device 116 may be located in or near input / output apparatus 104A and may be configured to capture images of tube assembly 102 before tube assembly 102 is transferred to track 112 and / or loaded into carrier 114.

[0034] Additional references Figure 2 , Figure 2 A top view of imaging instrument 104B is shown. Other embodiments of imaging instruments may be implemented in system 100. Figure 2 In some embodiments, imaging instrument 104B is configured to capture images of tube assembly 102 while tube assembly 102 is located on track 112. Tube assembly 202 is shown located on track 112 and at imaging position 214 within imaging instrument 104B. Components of imaging instrument 104B may be controlled by computer 220. In some embodiments, computer 220 may be incorporated into computer 120 ( Figure 1 )middle.

[0035] The imaging apparatus 104B may include: three imaging devices 216 configured to capture images of the tube assembly 202 at the imaging position 214. The imaging devices 216 are individually referred to as a first imaging device 216A, a second imaging device 216B, and a third imaging device 216C. In other embodiments, the imaging apparatus 104B may include one or more imaging devices. The imaging devices 216 may be configured to generate image data representing the tube assembly 202. The computer 220 and / or the computer 120 ( Figure 1 ) can be configured to process image data generated by imaging device 216 as described herein.

[0036] The imaging instrument 104B may also include one or more lighting devices 217 configured to illuminate the imaging location 214. Thus, the lighting device 217 may be configured to illuminate the tube assembly 202. Figure 2 In the embodiment of FIG. 1 , the imaging instrument 104B may include three lighting devices 217 , which are individually referred to as a first lighting device 217A, a second lighting device 217B, and a third lighting device 217C.

[0037] Reference again Figure 1 , the system 100 may also include or be coupled to a computer 120 that may be configured to operate the instrument 104 and / or communicate with the instrument 104. In some embodiments, the computer 120 may analyze data received in the system 100 and / or generated by the instrument 104. The computer 120 and programs executed by the computer 120 may also analyze image data generated by the imaging instrument 104B and other imaging devices as described herein. The computer 120 may include: a processor 122 configured to execute computer-readable instructions (e.g., program code).

[0038] exist Figure 1In some embodiments, computer 120 may be connected to individual instruments in instruments 104. For example, computer 120 may be connected to computer 220 in imaging instrument 104B. In some embodiments, computer 120 may be coupled to one or more workstations (not shown) that are coupled to one or more of instruments 104. In some embodiments, computer 120 may be configured to transmit computer instructions and / or computer code to individual instruments in instruments 104. In some embodiments, computer 120 may be configured to receive and / or process data generated by instruments 104.

[0039] The computer 120 may include or be coupled to a memory 124, which may store or access one or more modules and / or programs 126, which are described herein as being stored in the memory 124, and the modules or programs 126 may be executable by the processor 122. The programs 126 may be configured to operate one or more of the instruments 104. Although the programs are described as individual programs, in some embodiments, the program 120 may be implemented as a single program. One or more of the programs 126 may execute an artificial intelligence (AI) algorithm. The programs 126 may include: a data processing program 126A, which may be configured to process data in at least one of the instruments 104, the at least one instrument including the imaging instrument 104B. For example, the data processing program 126A may process data input to one or more of the instruments in the system 100 and / or the instruments 104. The data processing program 126A may also process data generated by one or more of the instruments 104.

[0040] In some embodiments, the memory 124 may include a processing algorithm 126B. In some embodiments, the memory 124 may include two or more processing algorithms. In some embodiments, the two or more processing algorithms may be implemented as a single processing algorithm, such as processing algorithm 126B. Processing algorithm 126B may include: computer code configured to analyze image data and other data generated by an imaging device in or coupled to system 100. In some embodiments, processing algorithm 126B may use AI to process data, and AI may be implemented as a tube characterization model 126C that analyzes image data as described herein. Tube characterization model 126C may be updated as described herein. For example, image decomposition model 502 ( Figure 5 ) and / or image synthesis model 504 ( Figure 5 ) can be updated as described herein.

[0041] The imaging device 216 and a computer (e.g., computer 120) for processing images captured by the imaging device 216 may be part of the machine vision and machine learning used by the system 100 to facilitate the processing of the tube assembly 102. For example, as described herein, the machine vision and AI models that may be run in the tube characterization model 126C may identify a specific type of tube assembly 102 and / or a sample located in the tube assembly 102 to facilitate sample processing and analysis as described herein.

[0042] The computer 120 may be coupled to or implemented in a workstation 128. The workstation 128 may enable user interaction with one or more instruments in the system 100 and / or the instruments 104. The workstation 128 may include a display 128A and / or a keyboard 128B that enables a user to interface with individual instruments in the system 100 and / or the instruments 104.

[0043] In some embodiments, the system 100 can be coupled to a laboratory information system (LIS) 130, which can determine how the sample is to be tested by the system 100. In some embodiments, the LIS 130 can be implemented in the computer 120. The LIS 130 can be coupled to a hospital information system (HIS) 132, which can receive a specific test order for a specific sample. The HIS 132 can also receive the test results after the test is performed on the specific sample by the system 100.

[0044] In an example of the operation of system 100, a physician may order a specific test to be performed on a sample from a patient. The physician may enter the test order into HIS 132. HIS 132 may then transmit the test order to LIS 130. A sample may be drawn from the patient and placed into a tube assembly, such as tube assembly 202 ( Figure 2 ). The configuration of the tube assembly 202 (such as color, size, and shape) may correspond to the type of test in the test order. The tube assembly 202 may then be delivered to the system 100. The LIS 130 may determine how the sample is to be tested and may transmit this information to the computer 120 where instructions may be generated to operate the instrument 104 to perform the test. For example, the data processing program 126A may generate the instructions. The imaging device 216 ( Figure 2 ) can generate an image of the tube assembly 202, and based on the image data, the tube characterization model 126C can identify the type of the tube assembly 202. The testing and movement of the tube assembly 202 throughout the system 100 can be based at least in part on the type of the tube assembly 202. After the testing is completed, the LIS 130 can transmit the test results to the HIS 132.

[0045] Now additionally refer to Figures 3A-3C , Figures 3A-3CDifferent types of tube assemblies that may be used within system 100 are illustrated. A tube assembly refers to a tube with or without a cap attached to the tube. A tube assembly may also include a sample of a tube assembly or other content. Figures 4A-4C , Figures 4A-4C Pictured is without a cap Figures 3A-3C As shown in the figure, all tube assemblies have different configurations or geometries. The caps can all have different cap geometries.

[0046] Figure 3A The tube assembly 340 includes a cap 340A which is white with a red stripe and has an extended vertical portion. The cap 340A can fit over the tube 340B. The tube assembly 340 has a height H31. Figure 4A The tube 340B is shown without the cap 340A. The tube 340B has a height H41 and a width W41.

[0047] Figure 3B The tube assembly 342 includes a cap 342A which is blue with a rounded top. The cap 342A can fit over the tube 342B. The tube assembly 342 has a height H32. Figure 4B Tube 342B is shown without cap 342A. Tube 342B has a height H42 and a width W42.

[0048] Figure 3C The tube assembly 344 includes a cap 344A which is red and gray with a flat top. The cap 344A can fit on the tube 344B. The tube assembly 344 has a height H33. Figure 4C Tube 344B is shown without cap 344A. Tube 344B has a height H43 and a width W43.

[0049] Figures 3A-4C The diagram illustrates different types of tubes, caps, and tube assemblies that may be used in the system 100. The tube characterization model 126C may have been trained to characterize or identify Figures 3A-4C In some embodiments, the tube characterization model 126C may also determine whether the tube assembly cannot be characterized. In some embodiments, the tube may have Figures 3A-4C The tube and the cap may be made of different materials and may have different shapes, structures, albedos, and textures that may be represented by the tube representation model 126C.

[0050] Over time, the laboratory may utilize tube assemblies from a source that provides tube assemblies that deviate from the tube assemblies supplied by the manufacturer on which the tube characterization model 126C was trained. For example, the laboratory may use tube assemblies supplied by a local manufacturer or new tube assembly designs that are not stored in the tube characterization model 126C. The training data may be limited and may limit the number of tube variations that the tube characterization model 126C can identify or characterize. Retraining the tube characterization model 126C to handle different tube types typically requires training with a large number of images illustrating different configurations of the tube assembly under different views and lighting conditions. In some conventional embodiments, training is performed manually, which makes retraining very costly.

[0051] Embodiments disclosed herein synthesize tube assembly images to generate synthesized tube assembly images. In one example, the tube characterization model 126C is trained to characterize a specific type of tube assembly that can be used for a specific test. The tube assembly can have a specific tube material, tube color, tube shape, cap shape, cap color, and other unique attributes. Over time, one or more manufacturers may change these attributes of the tube assembly. Embodiments disclosed herein enable the system 100 (such as, processing algorithm 126B) to synthesize images of tube assemblies with new attributes, which can then be used by the tube characterization model 126C to characterize the new tube assembly.

[0052] In another example, the system 100 may receive a new tube assembly type from a new manufacturer. Each attribute of the new tube assembly type may be similar to the attributes of the specific tube assembly on which the tube characterization model 126C has been trained. For example, the new tube assembly may have the same tube material as the tube assembly on which the tube characterization model 126C has been trained, but with a different cap shape. Although another tube assembly type on which the tube characterization model 126C has been trained may have the same cap type as the new tube assembly type, but with a different tube material. Using the tube characterization model 126C, the system 100 may synthesize images of the new tube assembly with different cap shapes. In some embodiments, the system 100 may synthesize images using similar cap shapes from another type or tube assembly. In other embodiments, the system 100 may synthesize the cap shape of the new tube assembly.

[0053] Some embodiments disclosed herein may use unpaired or paired image-to-image translation to adapt existing data (e.g., image data) from an existing tube assembly type to a new tube assembly type. The existing data may be images used to train the tube characterization model 126C. The image-to-image translation may be performed using machine learning, such as a deep learning (DL) method. Examples of machine learning include generative adversarial networks (GANs). An example of a GAN is a cycleGAN that performs unpaired image-to-image translation using a cycle-consistent adversarial network. Some embodiments decompose a captured image of a tube assembly into decomposed features. One or more of the decomposed features may be translated into a synthesized tube assembly image or into a portion of a synthesized tube assembly. Thus, the systems and methods disclosed herein enable data (e.g., images) to be collected from existing tube assembly configurations to develop synthesized images of different tube assemblies.

[0054] Additional references Figure 5 , Figure 5 A block diagram of a process 500 implementing a method of synthesizing images of a tube assembly as described herein is illustrated. Figure 5 The images referred to in the figure may be image data, such as image data generated by the imaging instrument 104B. The process 500 uses synthesized data or images of paired tube assemblies with controlled variations to train an image decomposition model 502 and an image synthesis model 504. The trained image decomposition model 502 may then receive an input image of a tube assembly and decompose the input image into a latent space 506 having a plurality of features. The latent space manipulation module 508 may manipulate one or more features of the image in the latent space 506. For example, the latent space manipulation module 508 may manipulate one or more features in the latent space that are related to an attribute such as the cap color of the image of the tube assembly. The manipulated features may then be used by the image synthesis model 504 to generate a synthesized image of the tube assembly.

[0055] Figure 6 A block diagram of a process 600 of implementing a method for integrating images of an outer surface of a tube assembly and an interior of a tube assembly according to one or more embodiments is illustrated. The interior integration of the tube assembly can integrate the fluid in the tube assembly. In some embodiments, the process 600 can be viewed as two separate processes.

[0056] Process 600 may include: using Figure 5The image may be decomposed by an image decomposition model 502 that decomposes an input image into various features as described in the process 600. The process 600 may use a tube exterior latent space 610 that is configured to characterize the exterior properties of the image of the tube assembly. The tube exterior latent space 610 may include individual variables or features 612 that may be configured to isolate different properties of the image exterior of the tube assembly decomposed by the image decomposition model 502. The features 612 may be based on factors such as the geometry and material of the tube assembly, the surface properties of the tube assembly, the lighting conditions under which the image was captured, and the imaging device (e.g., Figure 2 The image may be isolated based on attributes such as camera properties of the imaging device 216. Other attributes of the image may be isolated or characterized.

[0057] The external latent space 610 can be manipulated by the latent space manipulation module 614. The latent space manipulation module 614 can be used with Figure 5 The latent space manipulation module 614 may be the same or substantially similar to the latent space manipulation module 508. The latent space manipulation module 614 may manipulate the features 612 isolated in the tube-external latent space 610. The image synthesis model 504 may then use the manipulated features manipulated by the latent space manipulation module 614 to synthesize an image.

[0058] Process 600 may include: using Figure 5 6 to decompose an image of a tube interior or fluid. Process 600 may use a tube interior latent space 618 configured to synthesize internal properties of an image of a tube assembly. In some embodiments, tube interior latent space 618 may characterize the contents of the tube assembly (e.g., samples stored in the tube assembly) as features 620. Tube interior latent space 618 may include individual variables or features 620, which may include the geometry and material of the tube assembly, fluid properties, lighting conditions under which the image was captured, imaging equipment (e.g., Figure 2 Other attributes of the image may be isolated or characterized as features 620.

[0059] The features 620 of the latent space 618 inside the tube can be manipulated by the latent space manipulation module 624. The latent space manipulation module 624 can be connected to the latent space manipulation module 614 or Figure 5 The latent space manipulation module 624 may be the same or substantially similar to the latent space manipulation module 508 of the embodiment. The latent space manipulation module 624 may manipulate the features 620 isolated in the tube interior latent space 618. The image synthesis model 504 may then use the manipulated features to synthesize a synthesized image of the tube assembly.

[0060] Additional references Figure 7 , Figure 7is an embodiment of a training process 700 that can be used to train the image decomposition model 502 and the image synthesis model 504. A first image A of a first tube assembly is input to an encoder network, which is implemented to generate latent variables L in a multi-dimensional latent space. A Decomposition model 730 (sometimes referred to as image decomposition model) of the latent features in the low-dimensional space. Decomposition model 730 can be similar to or the same as decomposition model 530. In some embodiments, decomposition model 730 generates latent features in a low-dimensional space. The latent variable L A The features of are partitioned into multiple feature groups, such as W, X, Y and Z, as Figure 7 Each of the features may correspond to a specific one of the properties of the tube assembly. In some embodiments, at least one of the features (e.g., feature Z) may be reserved for an intrinsic property of the image of the tube assembly where variation cannot be set to control. In some examples, all tubes under study may have a cylindrical structure that they all share in common, such that it is expected that they all share the same value in feature Z.

[0061] The synthesis model 732 (sometimes referred to as an image synthesis model), which may be implemented as a decoder network, may use the latent variable L A , to reconstruct the image A' of the tube assembly to be as similar as possible to the input image A. The synthesis model 732 can be similar to or the same as the synthesis model 532. The reconstruction loss module 734 can compare the input image A with the output image A' to determine whether the images are close to each other. For example, the reconstruction loss module 734 can analyze the properties of the image A and the image A' to determine the latent variable L A Is it correct and whether the decomposition model 730 and the synthesis model 732 are trained correctly.

[0062] The training process 700 may also include: receiving a second input image B of a second tube assembly into the decomposition model 730; and generating a latent variable L in a multidimensional space. B The latent features in . Images A and B can be paired tube component images. The latent variable L B The features of are partitioned into multiple feature groups, such as W', X', Y' and Z', as Figure 7 . Each of the features may correspond to a specific attribute in the properties of the tube assembly. At least one of the features (e.g., feature Z') may be reserved for an intrinsic property of the tube image that cannot be captured by one of the features. The synthesis model 732 may use the features to reconstruct the image B' of the tube assembly to be as similar as possible to the input image B. The reconstruction loss module 736 may compare the input image B with the output image B' to determine whether the images are close to each other. For example, the reconstruction loss module 736 may analyze the attributes of the image B and the image B' to determine the latent variable L BIs it correct and whether the decomposition model 730 and the synthesis model 732 are trained correctly.

[0063] Training the disentangled latent space (such as for use with decomposition model 730 and synthesis model 732) may include subjecting input image B to specific variations relative to input image A. For example, in the case of tube-hat synthesis, the latent variable L A and latent variable L B The first feature of (e.g., W and W') may be an albedo, such as the color of the tube cap. In this example, input image B is of the same type of tube assembly as input image A, with the only change being the color of the tube cap. Input image B may be synthetically generated (e.g., using computer generated imaging (CGI)) or captured from the same type of tube assembly and the same imaging settings used to acquire input image A. The tube cap in input image B may be very similar to the cap in input image A, with the only difference, or the only significant difference, being the difference in the color of the cap. In the latent space L A and L B In the embodiment, the remaining features (eg, X and X', Y and Y', Z and Z') are to be as similar as possible, while allowing the first features W and W' to vary as described above.

[0064] It can be used for the latent space L A and L B The above process is repeated for other changes in the latent variable, so that each of the features of the latent variable captures a specific change. In some embodiments, it is possible to have multiple changes between input image A and input image B at the same time. In such an embodiment, the corresponding latent features can be allowed to vary while the remaining features are kept consistent. As an example, input image A and input image B can be captured using the same camera, but include tube assemblies with different tube geometries, surface properties, and under different lighting conditions. In this embodiment, only the latent features corresponding to the camera properties are forced to be similar, and other features can vary.

[0065] The architecture of the training process 700 includes a decomposition model 730 and a synthesis model 732. Thus, training can be enhanced with those images by training on unpaired images to minimize the reconstruction-only loss. Such images can also be edited in the latent domain by replacing some of their latent features with latent features taken from another image. The decomposition model 730 and the synthesis model 732 can then be trained by an adversarial network that attempts to classify edited and unedited images. For example, there can be images acquired by a first imaging device C1 (e.g., Figure 2The set of tube assembly images S1 captured by the first imaging device 216A) under the lighting condition L1 provided by the first lighting device 217A. The latent features of the images S1 corresponding to the camera properties of the first imaging device 216A and the light conditions provided by the first lighting device 217A may be similar to each other. The two latent features may be regularized to approximate a mean vector (m C1 、m L1 ). For another set of tube assembly images S2 captured by a second imaging device C2 (e.g., second imaging device 216B) under illumination condition L2 provided by second illumination device 217B, the latent features associated with the camera and illumination conditions are forced to be approximated by a mean vector (m C2 、m L2 ) is a unit Gaussian distribution.

[0066] By using the vector (m C2 、m L2 ) replaces the latent features of the tube image in S1 and uses the synthesis model 732 to reconstruct the image of the tube assembly, synthesizing the same tube captured using C2 (the second imaging device 216B) and L2 (the second lighting device 217B). In some embodiments, the adversarial discriminator can be further trained to enforce the rule that the synthesized image is indistinguishable from the image in S2 in terms of camera and lighting. Similarly, one can replace (m C1 、m L1 ) to synthesize the images captured using C1 and L1. These processes ensure that image variations are disentangled by the latent space (e.g., latent space 610 and tube interior latent space 618). After training, the trained image decomposition model 730 learns how to perform image decomposition into the disentangled latent space, and the trained image synthesis model 732 learns how to perform image synthesis from latent variables in the latent space (such as latent variable L A or L B Since each latent space is disentangled using known variations, the latent vector of a given tube assembly image can be perturbed to generate many tube assembly images with desired variations.

[0067] In some embodiments, the latent space representation can encode data across different scales (e.g., decomposing an image). For example, normalized flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are examples of deep generative learning. In addition, the latent space can be constrained to a known parametric distribution (e.g., Gaussian or Gaussian mixture) or a non-parametric distribution, such as using a vector quantized variational autoencoder (VQ-VAE).

[0068] Additionally, images other than the generated images may be used as image data to characterize attributes, such as tube cap classification and tube fluid characterization. The disentangled features in the latent space may also provide useful information (used as ground truth annotations or considered as discriminant features) for downstream tasks that may be performed by the system 100. Furthermore, the disentangled features in the latent space may also be used to develop tools that may be used by laboratory personnel (such as suppliers, R&D personnel, field service technicians, and / or end users (e.g., laboratory technicians and managers)) to evaluate new tube assembly types and tube assembly variations to determine whether the system 100 ( Figure 1 )’s ability to appropriately characterize pipe assemblies.

[0069] In some embodiments, the system 100 can use unpaired image-to-image translation (such as cycleGAN) to adapt existing data from images of old tube assemblies to images of new tube assemblies. This process enables the system 100 and / or developers of tube assemblies to collect field data from existing tube assemblies when developing new types of tube assemblies.

[0070] In some embodiments, image data may be collected in a group that may include a dense collection of representative tube assembly types and a sparse collection of other similar tube assembly types. Enhancement data may be used to fill in missing attributes of the sparsely collected images of the tube assembly types. For example, color enhancement may be used to fill in missing colors in the sparsely collected images of the tube assembly types.

[0071] An example of a system 100 for integrating a pipe assembly is described below. Fig. 8A and Figure 8B , Fig. 8A and Figure 8B An image of a similar tube assembly is illustrated. Fig. 8A The tube assembly 840 is Figure 8B 842, except that the color of cap 840A of tube assembly 840 is blue and the color of cap 842A is green. For example, tube 840B may be similar to tube 842B, and both tube assembly 840 and tube assembly 842 may have the same height H81. Decomposition model 730 may receive an image of tube assembly 840 as input image A. Decomposition model 730 may also receive an image of tube assembly 842 as input image B. Decomposition model 730 decomposes input image A into variables represented as L A The feature set (W, X, Y, Z) in the latent space of the image B is decomposed into variables represented as L B The feature set (W', X', Y', Z') in the same latent space of .

[0072] Additional references Figure 8C , Figure 8CAn example of a decomposed image 844 of a tube assembly 840 is provided. It should be noted that other decompositions of the image of the tube assembly 840 may be performed. Figure 8C In the example of , feature W may be the cap color, which is blue. Feature X may be the cap height H82, feature Y may be the tube height H83, and feature Z may be the tube width W81. Other features may include other parts of the tube geometry or tube assembly geometry. The synthesis model 732 may reconstruct the deconstructed image to generate image A'. The input image A may be compared to the reconstructed image A' by a reconstruction loss module 734. If the loss or difference between the input image A and the reconstructed image A' is within a predetermined amount, then the decomposition model 730 and the synthesis model 732 may be considered to be properly trained and the latent variable L A This is true for process tube assembly 840 .

[0073] Tube assembly 842 may be selected as the paired image because the only significant difference between tube assembly 840 and tube assembly 842 is the color of the cap. B The image B of the tube assembly 842 is decomposed. The latent space L B The characteristics of the latent space L A The feature W' may be the cap color, which is green, the feature X' may be the cap height H82, the feature Y may be the tube height H83, and the feature Z may be the tube width W81. The synthesis model 732 may synthesize the deconstructed image to generate image B'. The input image B may be compared to the reconstructed image B' by the reconstruction loss module 736. If the loss or difference between the input image B and the reconstructed image B' is within a predetermined amount, then the decomposition model 730 and the synthesis model 732 may be considered to be properly trained and the latent variable L B This is true for process tube assembly 842.

[0074] Green cap 842A may be a new cap color. To reconstruct tube assembly 842 with a blue cap, feature W is used in place of feature W' which is a green cap. Image-to-image reconstruction and other methods may be used to reconstruct tube assembly 842 with a blue cap. The resulting reconstructed or synthesized image using features W, X', Y', and Z' would be tube assembly 842A with a blue cap for tube assembly 840. In some embodiments, cap 840A may be used as a whole to replace cap 842A, and in other embodiments, the color of cap 840A may be used to replace the color of cap 842A. In other embodiments, the color of cap 840A may be stored and used for cap 842A without imaging tube assembly 840.

[0075] Reference now Figures 9A-9B , Figures 9A-9BThe diagram shows a hidden space 618 ( Figure 6 ) can be used with tube assemblies of different interior and / or fluid characteristics. Tube assembly 940 can include one or more items (e.g., properties) located in the interior of tube 940B and can be sealed by cap 940A. One or more fluids 946 can be located in the interior of tube 940B and can include serum 946A, gel separator 946B, and blood 946C. Air gap 946D can be located above fluid 946.

[0076] Tube assembly 942 may include one or more items (e.g., properties) located in the interior of tube 942B and may be sealed by cap 942A. One or more fluids 948 may be located in the interior of tube 942B and may include serum 948A, gel separator 948B, and blood 948C. Air gap 948D may be located above fluid 948. The material of tube 940B and the material of tube 942B may be similar and may be identified by the following operation: generating image data through air gap 946D and air gap 948D. Accordingly, the properties of tube assembly 940 and tube assembly 942 include, but are not limited to, tube geometry, tube material, tube color, tube surface properties, sample fluid in the tube, labels attached to the tube, and lighting conditions of the tube during image capture.

[0077] All of the above attributes of tube assembly 940 and tube assembly 942 may be the same or substantially similar. The difference between tube assembly 940 and tube assembly 942 may be the labels attached to tube 940B and tube 942B. Tube assembly 940 has label 950 attached to tube 940B, and tube assembly 942 has label 952 attached to tube 942B. Figures 9A-9B As shown in FIG. 9A , label 950 is different from label 952 . For example, the size of label 950 is different from the size of label 952 , and the positioning of barcode 954 is different from the position of barcode 956 .

[0078] Decomposition model 730 and synthesis model 732 can be configured to identify and / or characterize the attributes of tube assembly 940 and the image of tube assembly 942 as features in a latent space. Accordingly, system 100 can be configured to replace the above features with a synthesized image (e.g., image A' or B'). Figures 9A-9B In the example of , system 100 can replace either label 950 or label 952 with a label from the original image to generate a synthesized image. In some embodiments, system 100 can replace the label but keep the original barcode or other label information in the synthesized image.

[0079] In some embodiments, the synthesized tube assembly image is used by the system 100 during the tube assembly identification process. For example, when the tube assembly 102 is loaded into the input / output instrument 104A, an image of the tube assembly 102 can be captured by the imaging device 116 or another imaging device. The synthesized image of the tube assembly can be used for tube assembly identification of tube assemblies that were not initially identified by the tube characterization model 126C. The tube assembly can then be characterized and transported to the appropriate instrument 102, where the sample is analyzed as described herein. In some embodiments, the instrument 102 can perform a process customized for the identified tube assembly, such as gas delivery. In other embodiments, the robot 108 ( Figure 1 ) can use specific techniques (such as, specific gripping techniques) depending on the type of tube assembly identified to move the sample containers to and from the carrier 114.

[0080] Reference now Fig.10 , Fig.10 1 is a flow chart illustrating a method 1000 of synthesizing an image of a tube assembly (e.g., tube assembly 202). Method 1000 includes, at 1002, capturing an image of the tube assembly, the capturing generating a captured image. Method 1000 includes, at 1004, decomposing the captured image into a plurality of features (e.g., features 612) in a latent space (e.g., latent space 610) using a trained image decomposition model (e.g., image decomposition model 502). Method 1000 includes, at 1006, manipulating one or more of the features in the latent space into manipulated features. For example, the cap color may be changed from red to blue and / or the cap shape may be changed from a circle to an ellipse. Additionally or alternatively, other features may be changed (i.e., manipulated). Method 1000 includes, at 1008, using a trained image synthesis model (e.g., image synthesis model 504) to generate a synthesized tube assembly image having at least one of the manipulated features.

[0081] Reference now Fig.11 , Fig.111 is a flow chart illustrating a method 1100 for synthesizing an image of a pipe assembly (e.g., pipe assembly 202). The method 1100 includes, at 1102, constructing an image decomposition model (e.g., image decomposition model 502) configured to receive an input image of a pipe assembly and decompose the input image into a plurality of features (e.g., features 612) in a latent space (e.g., latent space 610). The method 1100 includes, at 1104, constructing an image synthesis model (e.g., image synthesis model 504) configured to synthesize a synthesized pipe assembly image based on the plurality of features in the latent space. The method 1100 includes, at 1106, training an image decomposition model and an image synthesis model using at least an image of a first pipe assembly (e.g., pipe assembly 840) and an image of a second pipe assembly (e.g., pipe assembly 842), wherein there are one or more known changes between the image of the first pipe assembly and the image of the second pipe assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

[0082] Although the present disclosure is susceptible to various modifications and alternative forms, specific methods and device embodiments have been shown as examples in the drawings and described in detail herein. However, it should be understood that the specific methods and devices disclosed herein are not intended to limit the present disclosure, but on the contrary, are intended to cover all modifications, equivalents and alternatives that fall within the scope of the claims.

Claims

1. A method for synthesizing an image of a tube assembly, comprising: capturing an image of the tube assembly, the capturing generating a captured image; decomposing the captured image into a plurality of features in a latent space using a trained image decomposition model; Manipulating one or more of the features in the latent space into manipulated features; as well as A synthesized tube assembly image having at least one of the manipulated features is generated using the trained image synthesis model.

2. The method of claim 1, further comprising: An image decomposition model and an image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, wherein there are one or more known changes between the image of the first tube assembly and the image of the second tube assembly, and the training produces a trained image decomposition model and a trained image synthesis model.

3. The method of claim 2, wherein training the decomposition model is based at least in part on a reconstruction loss between the synthesized image and the input image.

4. The method of claim 1, wherein the tube assembly comprises a tube and a cap, and at least one of the plurality of characteristics is tube geometry, tube color, tube surface properties, cap geometry, cap color, or lighting conditions.

5. The method of claim 1, wherein the tube assembly comprises a tube and at least one of the plurality of characteristics is tube geometry, tube material, sample fluid in the tube, a label affixed to the tube, or lighting conditions of the tube.

6. A method of synthesizing an image of a tube assembly, comprising: constructing an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; constructing an image synthesis model configured to synthesize an integrated pipe assembly image based on the plurality of features in the latent space; as well as The image decomposition model and the image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, there being one or more known changes between the image of the first tube assembly and the image of the second tube assembly, wherein the training produces a trained image decomposition model and a trained image synthesis model.

7. The method of claim 6, further comprising: capturing an image of the tube assembly, the capturing generating a captured image; decomposing the captured image into a plurality of decomposed features in the latent space using a trained image decomposition model; manipulating one or more of the decomposed features in the latent space into manipulated features; as well as A synthesized tube assembly image having at least one of the manipulated features is generated using the trained image synthesis model.

8. The method of claim 6, wherein training the decomposition model is based at least in part on a reconstruction loss between the synthesized image and the input image.

9. The method of claim 6, wherein the image synthesis model is trained based on a reconstruction loss between a synthesized image and an input image.

10. The method of claim 6, wherein at least one of the features in the latent space is generated using computer generated imaging.

11. The method of claim 6, wherein images of at least two of the tube assemblies differ from each other with respect to one or more controlled properties.

12. The method of claim 6, wherein the images of at least two tube assemblies share one or more attributes in common.

13. The method of claim 6, wherein at least one of the image decomposition model or the image synthesis model is trained based on a reconstruction loss in the absence of an image of the tube assembly.

14. The method of claim 6, wherein the tube assembly comprises a tube and a cap, and at least one of the plurality of features in the latent space is tube geometry, tube color, tube surface properties, cap color, cap geometry, or lighting conditions.

15. The method of claim 6, wherein the tube assembly comprises a tube, and at least one of the plurality of features in the latent space is tube geometry, tube material, sample fluid in the tube, a label affixed to the tube, or lighting conditions of the tube.

16. The method of claim 6, wherein the one or more features in the latent space include properties of a fluid in a pipe assembly.

17. A diagnostic laboratory system comprising: an image decomposition model configured to receive an input image of a tube assembly and decompose the input image into a plurality of features in a latent space; as well as an image synthesis model configured to synthesize a comprehensive pipe assembly image based on the plurality of features in the latent space, The image decomposition model and the image synthesis model are trained using at least an image of a first tube assembly and an image of a second tube assembly, and there are one or more known changes between the image of the first tube assembly and the image of the second tube assembly.

18. The diagnostic laboratory system of claim 17, wherein the image decomposition model is trained based at least in part on a reconstruction loss between a synthesized image and an input image.

19. The diagnostic laboratory system of claim 17, wherein the image synthesis model is trained based on a reconstruction loss between a synthesized image and an input image.

20. The diagnostic laboratory system of claim 17, wherein the tube assembly comprises a tube and a cap, and at least one of the plurality of features in the latent space is tube geometry, tube color, tube surface property, cap color, cap geometry, or lighting condition.

Citation Information

Patent Citations

  • Processing fundus camera images using machine learning models trained using other modalities

    CN112868068A