Method and apparatus for hashing and retrieval of training images for use in hiln determination of a sample in an automated diagnostic analysis system

Abstract

A method of characterizing a test sample to be analyzed in an automated diagnostic analysis system provides a HILN classification (hemolysis, icterus, lipemia, normal) of the test sample along with the basis for that determination. The method includes assigning a hash code to each training image of a sample test used in a characterization training process. Responsive to a HILN determination of a test sample, the method can retrieve, via the hash code, one or more closest matching training images on which the HILN classification is based. The one or more closest matching training images can be displayed alongside one or more images of the test sample. A quality check module and system configured to carry out the method, among other aspects, are also described.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 929,066, filed October 31, 2019, entitled “METHODS AND APPARATUS FORHASHING AND RETRIEVAL OF TRAINING IMAGES USED IN AN HILN DETERMINATION OF ASPECIMEN IN AN AUTOMATED DIAGNOSTIC ANALYSIS SYSTEM”, the disclosure of which is incorporated herein by reference in its entirety for all purposes herein. Technical Field

[0003] This disclosure relates to methods and apparatus configured to characterize samples in an automated diagnostic analysis system. Background Technology

[0004] Automated diagnostic analysis systems can analyze samples such as urine, serum, plasma, interstitial fluid, and cerebrospinal fluid to identify analytes or other components contained in the sample. Such samples are typically contained in sample containers (e.g., sample collection tubes) and transported via automated tracks to various pretreatment, pre-screening (including imaging), and analyzers within the automated diagnostic analysis system.

[0005] Samples can be pre-screened and pretreated using one or more reagents and possibly other materials added therein, and then analyzed at one or more analyzers. Analytical measurements can be performed on samples via fluorescence absorption readings, such as by using an interrogation beam and obtaining fluorescence readings, or by obtaining luminescence readings. Analytical measurements allow the determination of the amount of analytes or other components contained in a sample using well-known techniques.

[0006] However, the presence of interfering substances in the sample (e.g., hemolysis, jaundice, and / or hyperlipidemia) (potentially due to patient condition or sample pretreatment) can adversely affect test results for analyte or component measurements obtained from one or more analyzers. For example, the presence of hemolysis (H) in the sample, which may be unrelated to the patient's disease state, can lead to different interpretations of the patient's disease state. Similarly, the presence of jaundice (I) and / or hyperlipidemia (L) in the sample can also lead to different interpretations of the patient's disease state.

[0007] Accordingly, a pre-screening process can be performed in an automated diagnostic analysis system for determining the presence of an interferent (such as H, I, and / or L) in a test sample to be analyzed, and in some cases the degree of the interferent. The pre-screening process involves automated detection of H, I, and / or L or normal (N) based on one or more images of the test sample captured at one or more imaging stations of the automated diagnostic analysis system. The pre-screening process can perform a HILN (hemolysis, icterus, and / or lipemia or normal) determination on the captured images; that is, the pre-screening process can determine the presence of H, I, and / or L in the test sample, and optionally the degree or index, or it can determine that the test sample is normal (N), and thus can be passed on for further analysis by one or more analyzers.

[0008] However, in some instances, the pre-processing for determining HILN can be incorrect. Accordingly, there is an unmet need for methods and apparatuses for improving HILN determination in an automated diagnostic analysis system. SUMMARY

[0009] According to a first aspect, a method of characterizing a test sample in an automated diagnostic analysis system is provided. The method includes receiving a plurality of training images for training a HILN (hemolysis, icterus, and / or lipemia or normal) network of a quality check module, the quality check module comprising a computer in the automated diagnostic analysis system, each training image depicting a sample test sample in a test sample container, and assigning a hash code to each training image via a hash network of the HILN network. The method further includes receiving one or more images of a test sample in a test sample container to be analyzed in the automated diagnostic analysis system; characterizing the test sample to be analyzed using the plurality of training images based on the one or more images received via the HILN network to determine a classification index comprising a hemolysis, icterus, lipemia, or normal class; and retrieving, via the hash code, one or more of the plurality of training images on which the determined classification index of the characterized test sample is based.

[0010] According to another aspect, a method of characterizing a test sample is provided. The method includes receiving a plurality of training images to train a HILN network, each training image depicting a sample test sample in a test sample container; assigning a hash code to each training image via a hash network; receiving one or more images of a test sample in a test sample container to be analyzed by the HILN network; characterizing the test sample to be analyzed using the plurality of training images based on the one or more images received via the HILN network to determine a classification index of a hemolysis, icterus, lipemia, or normal class; and retrieving, via the hash code of one or more of the plurality of training images, one or more of the plurality of training images on which the classification index is based.

[0011] According to yet another aspect, a quality check module of an automated diagnostic analysis system is provided. The quality check module includes a plurality of image capture devices arranged about an imaging location and a computer coupled to the plurality of image capture devices, the image capture devices configured to capture a plurality of images from a plurality of viewpoints of a sample container containing a sample therein. The computer is configured and operable via programmed instructions to input a first plurality of images captured by the plurality of image capture devices to a HILN (hemolytic, icteric, lipemic, normal) network executing on the computer, the first plurality of images representing a plurality of training images used to train the HILN network, each training image depicting a sample specimen in a sample container. The computer is further configured and operable via programmed instructions to assign a hash code to each training image via a hash network of the HILN network, and to input one or more second images captured by the plurality of image capture devices to the HILN network executing on the computer, wherein the one or more second images represent the same sample in a sample container to be analyzed in the automated diagnostic analysis system. The computer is further configured and operable via programmed instructions to characterize the sample to be analyzed based on the one or more second images via the HILN network using the plurality of training images to determine a classification index comprising a hemolytic, icteric, lipemic, and normal class; and retrieve, via the hash code, one or more of the plurality of training images on which the determined classification index of the characterized sample is based.

[0012] Further aspects, features, and advantages of the present disclosure can readily be made apparent from the following description and illustration of multiple exemplary embodiments, including the best mode contemplated for carrying out the present disclosure. The present disclosure can also be capable of other and different embodiments, and its several details can be modified in various respects, all without departing from the scope of the present disclosure. The present disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] The drawings described below are for purposes of illustration only and are not necessarily drawn to scale. Thus, the drawings and description are to be regarded as illustrative in nature, and not as restrictive. The drawings are not intended to be limiting in any way;

[0014] Figure 1 FIG. illustrates a top schematic view of an automated diagnostic analysis system including one or more quality check modules configured to implement a HILN determination method according to one or more embodiments;

[0015] Figure 2 FIG. illustrates a side plan view of a sample container including a separated sample having a serum or plasma portion that can contain an interferent;

[0016] Figure 3 FIG. illustratesFigure 2 a side plan view of a specimen container, held in a vertical orientation in a holder and which can facilitate imaging, the holder can be made of Figure 1 a carrier transport within an automated diagnostic analysis system according to one or more embodiments;

[0017] Figure 4A illustrates a schematic top view (with top removed for illustration purposes) of a quality check module according to one or more embodiments, including multiple viewpoints (1, 2, 3), and configured to capture and analyze multiple images to enable pre-screening to determine presence of an interferent;

[0018] Figure 4B illustrates a schematic side view (with front housing wall removed for illustration purposes) of a quality check module according to one or more embodiments, taken along section line 4B-4B of Figure 4A Figure 4A

[0019] Figure 5 illustrates a functional block diagram of a HILN network according to one or more embodiments, operable to output an interferent determination and classification index for a serum or plasma portion of a specimen in a specimen container;

[0020] Figure 6 illustrates a functional block diagram of a hash network process of a HILN network according to one or more embodiments; Figure 5

[0021] Figure 7 is a flowchart of a method of characterizing a specimen in an automated diagnostic analysis system according to one or more embodiments. DETAILED DESCRIPTION

[0022] Incorrect HILN determinations can be difficult to debug, as little information, if any known information, is provided explaining how the pre-screening process determined each HILN result. Embodiments provided herein can facilitate debugging of incorrect HILN determinations that have occurred in an automated diagnostic analysis system.

[0023] ​​​A test sample (e.g., a serum or plasma test sample) contained in a test sample container can be pre-screened at a quality check module of an automated diagnostic analysis system to determine the presence of an interferent in the serum or plasma portion of the blood test sample, and in some embodiments, the degree (index) of the interferent. Checking the quality of the test sample via imaging at the quality check module allows unacceptable test samples to be avoided from being routed to an analyzer, which can produce erroneous results and further can waste valuable analyzer time. The output of the pre-screening process can be a HILN index indicating the presence of one or more of hemolysis (H), icterus (I), or lipemia (L) in the serum or plasma portion of the test sample or a determination of normal (N) indicating that the serum or plasma portion can include an acceptably low amount of H, I, and L, or none at all. Hemolysis can be defined as a condition in the serum or plasma portion in which red blood cells are destroyed during pre-processing, which causes hemoglobin to be released from the red blood cells into the serum or plasma portion such that the serum or plasma portion takes on a reddish hue. The degree of hemolysis can be quantified by assigning a hemolysis index. Icterus can be defined as a blood condition in which the serum or plasma portion turns a dark yellow color, which can be caused by the accumulation of bile pigments (bilirubin). The degree of icterus can be quantified by assigning an icterus index. Lipemia can be defined as the presence of an abnormally high concentration of emulsified fat in the blood such that the serum or plasma portion has a whitish or milky appearance. The degree of lipemia can be quantified by assigning a lipemic index.

[0024] The pre-screening process can determine only the presence of HIL or the degree or subclass (index) of H (e.g., HO-H6 in some embodiments, and more or less in other embodiments), the degree or subclass (index) of I (e.g., IO-16 in some embodiments, and more or less in other embodiments), and / or the degree or subclass (index) of L (e.g., L0-L4 in some embodiments, and more or less in other embodiments). In some embodiments, the pre-screening process can include determining an uncentrifuged (U) class of the serum or plasma portion of a test sample that has not been centrifuged or has been improperly centrifuged.

[0025] Further, the pre-screening process can classify (or“segment”) the test sample container and various regions of the test sample in order to identify image regions corresponding to, for example, the serum or plasma portion, the settled blood portion, the gel separator (if used), the air, the label(s), and / or the test sample container top cover. The test sample container holder and / or the background can also be classified. As a result of the segmentation, the type of test sample container can be determined (e.g., via the height and / or width (diameter) obtained), and / or the type and / or color of the test sample container top cover.

[0026] A quality check module of an automated diagnostic analysis system configured to perform a pre-screening characterization method can include a HILN network, such as a Segmentation Convolutional Neural Network (SCNN), that receives as input one or more images of a sample test specimen acquired at an imaging station of the automated diagnostic analysis system (e.g., an imaging station of the quality check module) (described in more detail below). In some embodiments, the SCNN can include more than 100 operational layers, including, for example, BatchNorm, ReLU activation, convolution (e.g., 2D), dropout, and deconvolution (e.g., 2D) layers to extract features such as simple edges, textures, and partial serum or plasma portions and regions containing labels. Top layers such as fully convolutional layers can be used to provide correlations between the portions. The output of this layer can be fed to a SoftMax layer that produces an output on a per-pixel (or per-superpixel (patch) - including n x n pixels) basis as to whether each pixel or patch is classified as HIL. In some embodiments, the SCNN can provide only an output of HIL or N. In other embodiments, the output of the SCNN can include multiple sub-classes (indices) of HIL, such as greater than 20 HIL classes, such that for each interferent present, an estimate of the interferent level (index) of that interferent can also be obtained. In some embodiments, the SCNN can also include a front-end container segmentation network (CSN) to determine container type and container boundaries. In some embodiments, the Segmentation Convolutional Neural Network (SCNN) can include a Deep Semantic Segmentation Network (DSSN) or other deep learning convolutional neural network. Alternatively, other types of HILN networks can be used.

[0027] If it is found that the test specimen includes one or more of H, I, and L, then appropriate notifications can be provided to an operator, and / or the test specimen container can be down-lined (1) to perform remediation to correct one or more of H, I, or L, (2) to redraw the sample, or (3) to perform other processing. Thus, the ability to pre-screen for HILN prior to analysis by one or more analyzers can advantageously (a) minimize wasted time analyzing test specimens that are not of suitable quality for analysis, (b) avoid or minimize erroneous test results, (c) minimize patient test result delays, and / or (d) avoid waste of patient test specimens.

[0028] However, in some instances, the HILN determination for one or more test samples can be incorrect. Such incorrect HILN determinations can possibly be caused by an insufficient number and / or range of sample test images (training images) used to train the HILN network. To facilitate debugging of incorrect HILN determinations, the characterization method and quality check module configured to carry out a pre-screening process according to one or more embodiments herein assigns a hash code to each training image used to train the HILN network. The hash codes assigned in training can be indexed using a hash table or the like and later used to facilitate quick retrieval of the training image(s) on which the HILN determination is based. In addition, the characterization method and quality check module can provide retrieval of one or more training images via the hash code on which the HILN determination of a test sample characterized in the automated diagnostic analysis system is based. This allows comparison of the image(s) of the test sample specimen having an incorrect HILN determination to the particular training image(s) on which the incorrect HILN determination is based. The image(s) of the test sample specimen having an incorrect HILN determination and the image(s) of the training specimen on which the HILN determination is based can be displayed on a display screen, such as by side-by-side display for comparison. Additional training images based on the incorrectly characterized sample specimen can then be included in the HILN network, e.g., by retraining, as appropriate to improve its performance. Other corrective measures can also or alternatively be taken based on the retrieved training images. Still further, in some embodiments, the pre-screening process can assign a (determination accuracy) prediction confidence level to each HILN determination for a test sample based on the output of the classification network (e.g., SCNN), alerting a system operator to suspect (i.e., by a low confidence value) the HILN determination and possibly flagging such determinations for retraining.

[0029] Further details of the characterization method of the present invention, the quality check module configured to carry out the characterization method of the present invention and the automated diagnostic analysis system comprising one or more quality check modules will be described herein with reference to Figures 1-7 Further described.

[0030] Figure 1 Fig. 1 illustrates an automated diagnostic analysis system 100 capable of automatically processing a plurality of test sample containers 102 containing sample test specimens 212 (see Figure 2). The specimen containers 102 can be provided in one or more racks 104 at a loading area 105 before being transported around one or more analyzers (e.g., first analyzer 106, second analyzer 108, and / or third analyzer 110) of the automated diagnostic analysis system 100 and analyzed thereby. A greater or lesser number of analyzers can be in the system. The analyzers can be any number of clinical chemistry analyzers, assay instruments, and / or the like or combinations thereof. The specimen containers 102 can be any suitable transparent or translucent container such as a blood collection tube, test tube, sample cup, clear vial, or other transparent or non-transparent glass or plastic container that is capable of containing and allowing imaging of a specimen 212 contained therein. The size and type of the specimen containers 102 can vary.

[0031] The specimen 212 (see Figure 2 ) can be provided to the automated diagnostic analysis system 100 in the specimen container 102, which can be capped with a cap 214. The cap 214 can be of different types and / or colors (e.g., red, royal blue, light blue, green, gray, tan, yellow, or combinations of colors), which can be meaningful in terms of what test the specimen container 102 is used for, the type of additive included therein, whether the container includes a gel separator, vacuum capability, and the like. Other colors can also be used. In one embodiment, the cap color and / or cap type can be determined by the characterization methods described herein.

[0032] Each specimen container 102 can be provided with a label 218, which can include identification information 218i (i.e., indicia) such as a bar code, alphabetic characters, numeric characters, or combinations thereof thereon. The identification information 218i can be machine-readable at various locations of the automated diagnostic analysis system 100. The machine-readable information can be darker (e.g., black) than the label material (e.g., white material) so that it can be easily imaged. The identification information 218i can be indicative of or can be related via a laboratory information system (LIS) 147 or other test order database to the identification of a patient and the test to be performed on the specimen 212. The identification information 218i can be indicative of other or additional information. Such identification information 218i can be provided on the label 218, which can be adhered to or otherwise provided on the outer surface of the tube 215. As Figure 2As shown, the label 218 may not extend continuously around the sample container 102, or may not extend continuously along the length of the sample container 102, such that from the particular forward viewpoint shown, most of the serum or plasma fraction 212SP is visible (the portion shown by the dashed line) and is not obscured by the label 218. However, as should be apparent, the sample 212 can be observed and imaged from multiple viewpoints, and therefore images can be captured from multiple viewpoints such that at least one viewpoint will have the serum or plasma fraction 212SP observable.

[0033] Sample 212 may include a serum or plasma portion 212SP and a settled blood portion 212SB contained within tube 215. Air 216 may be provided above the serum and plasma portions 212SP, and the liquid-air boundary between the air 216 and the serum and plasma portions 212SP is defined as the liquid-air interface (LA). The boundary between the serum or plasma portion 212SP and the settled blood portion 212SB is defined as the serum-blood interface (SB). The interface between the air 216 and the top cap 214 is defined as the tube-cap interface (TC). The tube height (HT) is defined as the height from the bottom of tube 215 to the bottom of top cap 214 and may be used to determine the tube size. The height of the serum or plasma portion 212SP is HSP and is defined as the height from the top of serum or plasma portion 212SP to the top of settled blood portion 212SB. The height of the settled blood portion 212SB is HSB, and is defined as the height from the bottom of the settled blood portion 212SB to the top of the settled blood portion 212SB at SB. HTOT is the total height of sample 212, and is equal to HSP plus HSB.

[0034] More specifically, the automated diagnostic analysis system 100 may include a base 120 ( Figure 1 (e.g., frame, base plate, or other structure), track 121 can be mounted on base 120. Track 121 can be a combination of rails (e.g., single or multi-rail), conveyor belt, conveyor chain, movable platform, or any other suitable type of conveying mechanism. Track 121 can be circular or any other suitable shape, and in some embodiments can be a closed track (e.g., an endless track). In operation, track 121 can transport individual sample containers in sample container 102 to various locations spaced apart around track 121, such as those held in carrier 122. Figure 3 ).

[0035] The carriers 122 can be passive, non-motorized discs that can be configured to carry a single specimen container 102 on the track 121, or alternatively, are automated carriers that include an onboard drive motor, such as a linear motor programmed to move around the track 121 and stop at preprogrammed locations. Other configurations of the carriers 122 can be used. The carriers 122 can each include a holder 122H (see Figure 3 ) configured to hold the specimen container 102 in a defined vertical position and orientation. The holder 122H can include a plurality of fingers (or leaf springs) 122F (a few labeled) that secure the specimen container 102 on the carrier 122, but some of the fingers can be movable or flexible to accommodate specimen containers 102 of different sizes (e.g., diameters). In some embodiments, the carriers 122 can exit the loading area 105 after unloading from the one or more shelves 104. The loading area 105 can serve a dual function of also allowing the specimen containers 102 to be reloaded from the carriers 122 to the loading area 105 after the pre-screening and / or analysis is completed by the one or more analyzers 106-110.

[0036] A robot 124 can be provided at the loading area 105, and the robot 124 can be configured to grasp the specimen containers 102 from the one or more shelves 104 and load the specimen containers 102 onto the carriers 122, such as onto the input channel of the track 121. The robot 124 can also be configured to reload the specimen containers 102 from the carriers 122 to the one or more shelves 104. The robot 124 can include one or more (e.g., at least two) robot arms or assemblies capable of X (lateral) and Z (vertical - out of page, as shown), Y and Z, X, Y and Z, or r (radial) and theta (rotational) motion. The robot 124 can be a gantry robot, articulated robot, R-theta robot, or other suitable robot, where the robot 124 can be equipped with a robot gripper that is oriented, sized, and configured to pick up and place the specimen containers 102.

[0037] Upon loading onto the track 121, the specimen container 102 carried by the carrier 122 can be advanced to a first pre-processing station 125. For example, the first pre-processing station 125 can be an automated centrifuge configured to perform separation of the specimen 212 into a serum or plasma portion 212SP and a settled blood portion 212SB. The carrier 122 carrying the specimen container 102 can be diverted to the first pre-processing station 125 by an inflow channel or other suitable robotic diversion. After being centrifuged, the specimen container 102 can exit on an outflow channel, or otherwise removed by a robot, and continue along the track 121. In the depicted embodiment, the specimen container 102 in the carrier 122 can next be transported to a quality check module 130 configured to perform the methods of pre-screening and characterizing a specimen according to embodiments of the present disclosure, as will be further described herein.

[0038] The quality check module 130 is configured to perform pre-screening and to perform the characterization methods described herein to automatically determine the presence of H, I, and / or L contained in the specimen 212 and optionally to automatically determine the range or extent, or whether the specimen is normal (N). If found to contain a valid low amount of H, I, and / or L, and thus deemed normal (N), the specimen 212 can continue on the track 121 and can then be analyzed by one or more analyzers (e.g., first, second, and / or third analyzers 106, 108, and / or 110). Thereafter, the specimen container 102 can be returned to the loading area 105 for reloading onto one or more racks 104.

[0039] In some embodiments, in addition to the HILN determination, segmentation of the specimen container 102 and specimen 212 can also occur. From the segmentation data, post-processing by the computer 143 can be used for quantification of the specimen 212 (i.e., determination of HSP, HSB, HTOT, and determination of the location of SB or SG and LA). In some embodiments, characterization of the physical properties of the specimen container 102 (e.g., size - height and / or width / diameter) can also occur at the quality check module 130. Such characterization can include determination of HT and W, and possibly TC and / or W or Wi. From this characterization, the size of the specimen container 102 can be extracted. Further, in some embodiments, the quality check module 130 can also determine the cap color and / or cap type, which can be used as a safety check and can capture whether an incorrect tube type has been used for an ordered test.

[0040] In some embodiments, a remote station 132 can be provided on the automated diagnostic analysis system 100 that is not directly linked to the track 121. For example, a separate robot 133 (shown in dashed line) can carry the specimen container 102 containing the specimen 212 to the remote station 132 and return it after testing / preprocessing. Alternatively, the specimen container 102 can be manually removed and returned. The remote station 132 can be used to test certain constituents, such as hemolysis levels, or can be used for further processing, such as reducing lipemia levels by one or more additions and / or by additional processing, or removing clots, bubbles or foam, for example, that were previously determined by the quality check module 130. Other pre-screening using the HILN detection method described herein can be done at the remote station 132.

[0041] Additional station(s) can be provided on the track 121 or at one or more locations along the track 121. The additional station(s) can include a decap station, an aliquot station, one or more additional quality check modules 130, etc.

[0042] The automated diagnostic analysis system 100 can include a plurality of sensors 116 at one or more locations around the track 121. The sensors 116 can be used to keep track of the location of the specimen containers 102 on the track 121 by reading the identification information 218i or similar information (not shown) provided on each carrier 122. Any suitable means for tracking location can be used, such as proximity sensors. All of the sensors 116 can be interfaced with the computer 143 so that the location of each specimen container 102 can be known at all times.

[0043] The preprocessing stations and analyzers 106, 108 and 110 can be equipped with a robotic mechanism and / or an inflow channel that removes the carriers 122 from the track 121, and a robotic mechanism and / or an outflow channel that is configured to re-enter the carriers 122 into the track 121.

[0044] The automated diagnostic analysis system 100 can be controlled by a computer 143, which can be a microprocessor-based central processing unit CPU, with suitable memory and suitable conditioning electronics and drivers for operating the various system components. The computer 143 can be disposed as part of the base 120 of the automated diagnostic analysis system 100 or separate therefrom. The computer 143 can operate to control the movement of the carriers 122 to and from the loading area 105, movement about the track 121, movement to and from the first pre-processing station 125, and operation of the first pre-processing station 125 (e.g., centrifuge), movement to and from the quality check module 130, and operation of the quality check module 130, and movement to and from each of the analyzers 106, 108, 110. In some embodiments, the operation of each of the analyzers 106, 108, 110 for performing various types of testing (e.g., assays or clinical chemistry) can be provided by the computer 143, or alternatively, each of the analyzers 106-110 can include its own server or computer, which can interface and communicate with the computer 143 via a suitable network, such as a LAN or WAN.

[0045] For all modules other than the quality check module 130, the computer 143 can control the automated diagnostic analysis system 100 in accordance with software, firmware, and / or hardware commands or circuitry, such as those used on the Dimension® clinical chemistry analyzers sold by Siemens Healthcare Diagnostics, Inc. of Tarrytown, New York, and such control is typical to those skilled in the art of computer-based electromechanical control programming, and will not be further described herein. Other suitable systems for controlling the automated diagnostic analysis system 100 can be used. Control of the quality check module 130 can also be provided by the computer 143, but in accordance with the characterization method of the present application described in detail herein.

[0046] The computer 143, as used for image processing and carrying out the characterization method described herein, can include a CPU or GPU, sufficient processing power and RAM, and suitable storage devices. In one example, the computer 143 can be a multi-processor PC equipped with one or more GPUs, 8 GB or more of RAM, and 1 TB or more of storage. In another example, the computer 143 can be a PC equipped with a GPU, or alternatively, a PC equipped with a CPU operating in parallel mode. A mathematical kernel library (MKL) can also be used, 8 GB or more of RAM, and suitable storage devices.

[0047] Embodiments of the present disclosure can be implemented using a computer interface module (CIM) 145 that allows a user to easily and quickly access various control and status displays of the display 145D. These control and status displays can display and enable control of some or all aspects of the plurality of interrelated automated devices for pre-screening, pre-treatment preparation, and analysis of a test sample 212. The CIM 145 can be employed to provide information regarding the operational status of the plurality of interrelated automated devices, as well as information describing the location of any test sample 212 and the status of tests to be performed or being performed on the test sample 212. The CIM 145 is thus adapted to facilitate interaction between an operator and the automated diagnostic analysis system 100. The display 145D of the CIM 145 can be operable to display a menu including icons, scroll bars, boxes, and buttons through which an operator can interact with the automated diagnostic analysis system 100. The menu can include a plurality of functional elements that are programmed to display and / or operate functional aspects of the automated diagnostic analysis system 100. As will be apparent from the following, the display 145D can be used to display training images on which characterization of a test sample 212 is based.

[0048] Figure 4A and Figure 4B Embodiments of a quality check module 130 configured to carry out the characterization methods as shown and described herein are shown. The quality check module 130 and the computer 143 can be configured with programmed instructions to pre-screen test samples 212 (e.g., serum or plasma portions 212SP thereof) for the presence and extent of interferents (e.g., H, I, and / or L) prior to analysis by one or more analyzers 106, 108, 110. Pre-screening in this manner allows for additional processing, additional quantification or characterization, and / or discarding and / or replotting of test samples 212 without wasting valuable analyzer resources or potentially allowing the presence of interferents to affect the accuracy of test results.

[0049] In addition to the interference detection methods described herein, other detection methods can be performed on the sample 212 contained in the sample container 102 at the quality inspection module 130. For example, a method can be implemented at the quality inspection module 130 to provide segmentation data. The segmentation data can be used in post-processing steps to quantify the sample 212, such as determining certain physical dimensional characteristics of the sample 212, such as LA and SB, and / or determining HSP, HSB, and / or HTOT. Quantization can also involve estimating, for example, the volume of serum or plasma fraction (VSP) and / or the volume of sedimented blood fraction (VSB). Furthermore, the quality inspection module 130 can be used to quantify the geometry of the sample container 102, i.e., to quantify certain physical dimensional characteristics of the sample container 102, such as the location of TC, HT, and / or W or Wi of the sample container 102. Other quantifiable geometric features can also be determined.

[0050] The quality inspection module 130 may include a housing 446 that may at least partially surround or cover the track 121 to minimize the influence of external lighting. During the image acquisition sequence, the sample container 102 may be located inside the housing 446. The housing 446 may include one or more doors 446D to allow the carrier 122 to enter and / or exit the housing 446. In some embodiments, the top plate may include an opening 446O to allow a robot, including movable robotic fingers, to load the sample container 102 into the carrier 122 from above.

[0051] like Figure 4A and 4B As shown, the quality inspection module 130 may include a plurality of image capture devices 440A-440C configured to capture lateral images of the sample container 102 and the sample 212 at imaging position 432 from a plurality of viewpoints (labeled 1, 2, and 3). While three image capture devices 440A-440C are shown and preferred, two, four, or more may optionally be used. As shown, viewpoints 1-3 may be arranged such that they are approximately equally spaced from each other, such as approximately 120° apart. For example, images may be captured in a cyclic manner, wherein one or more images may be captured from viewpoint 1, followed by views 2 and 3 in sequence. Other image capture sequences may be used. Light sources 444A-444C may illuminate the sample container 102 from behind (as shown). Multiple viewpoints are advantageous because one or more images captured from viewpoints 1-3 may be partially or completely obscured by one or more labels 218 (i.e., no clear view of the serum or plasma fraction 212SP). Among multiple viewpoints, at least one unobstructed viewpoint can be identified.

[0052] As depicted, image capture devices 440A, 440B, 440C can be arranged around the track 121. Other arrangements of multiple image capture devices 440A, 440B, 440C can be used. In this manner, when a test sample container 102 resides in the carrier 122 at the imaging location 432, an image of the test sample 212 in the test sample container 102 can be taken. When three or more viewpoints are used, the field of view of the multiple images obtained by the image capture devices 440A, 440B, 440C can slightly overlap in a circumferential range.

[0053] The image capture devices 440A-440C can be any suitable device for capturing clear digital images, such as a conventional digital camera, a charge-coupled device (CCD), an array of photodetectors, one or more CMOS sensors, etc., capable of capturing a pixelated image. The captured image size can be, for example, approximately 2560 x 694 pixels. In another embodiment, the image capture devices 440A, 440B, 440C can capture an image size of, for example, approximately 1280 x 387 pixels. Other image sizes and pixel densities can be used.

[0054] Each image can be triggered and captured at the quality check module 130 in response to receiving a trigger signal provided in the communication lines 443A, 443B, 443C from the computer 143. According to one or more embodiments, each captured image can be processed by the computer 143. In one particularly effective method, high dynamic range (HDR) processing can be used to capture and process image data from the captured images.

[0055] Figure 5 A HILN network architecture 500 configured to carry out the HILN characterization methods described herein is shown. The architecture 500 can be implemented in the quality check module 130 controlled by the computer 143 via programmed instructions stored in memory. As discussed above, the image capture devices 440A-440C can be arranged around the track 121 at the imaging location 432 of the quality check module 130. Figure 4A and 4BA specimen container 102 is provided, as represented in functional block 502. A plurality of image capture devices 440A-440C can capture one or more images (e.g., multi-view images), as represented in functional block 504. The image data of each multi-view image can be processed to provide a plurality of optimally exposed and normalized image data sets (hereinafter “image data sets”), as described in U.S. Patent Application Publication Nos. 2018 / 0372648 and 2019 / 0041318 by Wissmann et al., as represented in functional block 506. The image data (i.e., pixel data) of the images of the specimen (and specimen container) can be provided as input to a HILN network 535, which can be a Segmentation Convolutional Neural Network (SCNN). Other types of HILN networks can be employed to provide the HILN determination.

[0056] One task that can occur during pre-processing is detailed characterization of a specimen container, such as, for example, specimen container 102. This can include, for example, separating the specimen container from its background, understanding the contents and location of the serum or plasma portion 212SP, and segmenting any labels adhered to the specimen container. All of these tasks can be accomplished with a HILN network 535, which can perform pixel-level classification. Given an input image (i.e., pixel data), the HILN network 535 is operable to assign a classification index to each pixel of the image based on its local appearance as indicated by its pixel data values. The extracted pixel index information can be further processed by the HILN network 535 to determine a final HILN classification index. In some embodiments, the classification index can include 21 serum classes, including an un-centrifuged class, a normal class, and 19 HIL classes / subclasses, as described in more detail below.

[0057] A challenge in determining the HILN classification index can be caused by the subtle appearance differences within each subclass (index) of the H, I, and L classes. That is, the pixel data values of adjacent subclasses can be very similar. To overcome these challenges, the HILN network 535 can include a deep semantic segmentation network (DSSN) 538, which includes, in some embodiments, more than 100 operational layers. The deep semantic segmentation network (DSSN) 538 is a deep learning network (also known as deep structured learning) and is part of a broader family of machine learning methods based on artificial neural networks with representation learning. The learning can be supervised, semi-supervised, or unsupervised.

[0058] To overcome appearance differences that can be caused by changes in sample container type (e.g., size and / or shape), the HILN network 535 can also include a container segmentation network (CSN) 536 at the front end of the DSNN 538. The CSN 536 is configured and operated to determine container type and boundary information. The container type and boundary information 537 can be input to the DSNN 538 via an additional input channel, and in some embodiments, the HILN network 535 can provide the determined container type and boundary segmentation 539 as an output. In some embodiments, the CSN 536 can have a similar network structure as the DSNN 538, but shallower (i.e., with much fewer layers).

[0059] As shown in Figure 5 , the output of the HILN network 535 can be a classification index 540, which in some embodiments can include an un-centrifuged class 540U, a normal class 540N, a hemolyzed class 540H, a icteric class 540I, and a lipemic class 540L. In some implementations, the hemolyzed class 540H can include hemolyzed sub-classes H0, H1, H2, H3, H4, H5, and H6. The icteric class 540I can include icteric sub-classes I0, I1, I2, I3, I4, I5, and I6. And the lipemic class 540L can include lipemic sub-classes L0, L1, L2, L3, and L4. In other embodiments, each of the hemolyzed class 540H, the icteric class 540I, and / or the lipemic class 540L can have other numbers of fine-grained sub-classes.

[0060] As also shown in Figure 5 , according to one or more embodiments, the HILN network 535 can include a front-end hashing network 508 (HN). The front-end hashing network 508 (HN) assigns a unique hash code to the image before being processed by the DSNN 538. The front-end hashing network 508 (HN) can be configured to operate (via programmed instructions) as described below in connection with the hashing network 601 of Figure 6 . In alternative embodiments, the hashing network 508 can be a back-end hashing network that is configured to receive segmented regions of the input image of the sample in the sample container (e.g., the serum or plasma portion 212SP) from the DSNN 538, rather than the complete input image itself. The hash code is assigned and indexed to a hash table to allow easy retrieval of the corresponding training image later.

[0061] Figure 6 illustrates a HILN network 535 incorporating a front-end hashing network 508 (HN) and a back-end hashing network 601, according to one or more embodiments. Figure 5hashing network process 600 in the HILN network 535. The hashing network process 600 includes a hashing network 601, which can be implemented during a training phase 602 and a testing phase 604. During the training phase 602, the hashing network 601 provides a hash / index for training specimen images 605, where such images 605 are used to train the HILN network 535, such as the DSSN 538. For example, the DSSN 538 is trained to perform at least HILN determinations for the serum or plasma portion 212SP of each sample specimen 212 to be analyzed in the automated diagnostic analysis system 100. The hashing network 601 assigns a unique hash code 603 (e.g., code-1, code-2, code-3,... code-n) to each training image or each training image group 605 representing the same training sample specimen 212. Likewise, the hashing network 601 assigns a unique hash code 608 to each test image or test image group 606. The value returned by the hash function of the hashing network 601 is referred to as a hash code. Machine learning algorithms, and in particular hashing neural networks, are hash functions that produce hash codes. The hash codes are used to index the training images 605. For example, Figure 6 The three training images 605 shown in FIG. 6 represent the same sample specimen, where the three images 605 can have been captured by the image capture devices 440A, 440B, 440C, respectively, from the respective viewpoints 1-3 of the image capture devices 440A, 440B, 440C. In some embodiments, there can be a minimum distance (difference) between the hash codes 603 for training images in a similar HILN class index or group, while there can be a large distance between the hash codes 603 across HILN classes or groups. The distance between respective hash codes can be determined by any suitable routine, such as a Hamming function. The assigned training image hash codes 603 (possibly along with other identifiers for their respective training images) can be stored in memory, such as in a database of the computer 143, and can be retrieved later using a hash table. Figure 4A The three training images 605 shown in FIG. 6 represent the same sample specimen, where the three images 605 can have been captured by the image capture devices 440A, 440B, 440C, respectively, from the respective viewpoints 1-3 of the image capture devices 440A, 440B, 440C. In some embodiments, there can be a minimum distance (difference) between the hash codes 603 for training images in a similar HILN class index or group, while there can be a large distance between the hash codes 603 across HILN classes or groups. The distance between respective hash codes can be determined by any suitable routine, such as a Hamming function. The assigned training image hash codes 603 (possibly along with other identifiers for their respective training images) can be stored in memory, such as in a database of the computer 143, and can be retrieved later using a hash table.

[0062] During the testing phase 604, the hashing network 601 provides retrieval of the particular training image upon which each HILN determination is based. As Figure 6As shown, test specimen images 606 representing viewpoints 1-3 of the same specimen to be analyzed can be input to the hashing network 601, which assigns a unique hash code 608 (code-k) to the test specimen images 606. If the HILN network 535 has been thoroughly trained, then the assigned hash code 608 (code-k) can be such that it is the minimum distance (e.g., close or identical) to the particular hash code(s) 603 of the training images that are the closest matches to the test specimen images 606 as determined by the hashing network 508 or otherwise. For example, the matching / retrieval feature 610 of the hashing network process 600 (executed by the computer 143) can then retrieve the particular training images 612, determine the HILN index of the specimen depicted in the test specimen images 606 based on the particular training images 612, and can present those images to the user via the CIM 145, such as on a suitable display screen. The confidence level of the classification can also be displayed.

[0063] The closest hash code can be determined by a Hamming distance function or other measure of difference between the test sample hash code 608 and the closest training hash code(s) 603 of the training images 605. The closer the hash codes, the more similar and better the match of the images. In some embodiments, the first two or first few training images 605 can be selected as the closest for display to the user. Once the hash codes are learned, the retrieval technique can be as simple as retrieving the K (K is an integer) nearest neighbors, or can be based on a more advanced retrieval model. K can be set or selectable by the user.

[0064] The network 535 can determine that the classification of the test image 606 is incorrect. The incorrect HILN determination, along with the hashing network process 600 hashing / indexing and matching / retrieving features, can advantageously facilitate debugging of the incorrect HILN determination. This can be achieved, for example, by providing a basis or rationale for each HILN determination made by the HILN network 535 (via the particular training image 612 retrieved). In some embodiments, the incorrect HILN determination is determined based on a confidence level assigned by the HILN network 535, such as by a Softmax function. For example, if the confidence level is below a preselected value, such as 0.75 in a 0.0 to 1.0 scale, then the HILN determination can be deemed an incorrect HILN determination. In other embodiments, if the distance between the test hash code 608 and the training hash code 603 is greater than a preselected distance, then the HILN determination can be deemed an incorrect HILN determination. Thus, the incorrect HILN determination can flag the test image 606 for comparison with the particular training image 612 used by the HILN network 535 in the HILN determination, and appropriate corrective measures can be taken. For example, the corrective measures can include retraining of the HILN network 535. Retraining of the HILN network 535 can involve providing a mix of new image data from those images 606 deemed incorrectly classified and old image data previously used to train the model of the HILN network 535. The training can occur locally, or can be trained on a remote server / cloud. The trained artificial intelligence model can be tested again for validation / confirmation of the old data for regulatory approval, along with at least some newly collected data from those images 606 deemed incorrectly classified. The HILN network 535 generates a performance report that highlights the improvements to the previously trained model that complies with the regulatory process. Based on the report, a user can approve the update, or the HILN network 535 can automatically update. These updates can occur without interrupting existing workflows. For example, the update can simply replace the old model of the HILN network 535 with the new model. The update can be performed by a service technician, or it can be downloaded remotely. Thus, the HILN network 535 can be trained with additional training images that represent the incorrectly represented specimen. The hashing network process 600 can also advantageously identify outliers based on the hash codes, can flexibly add new HILN class sets to the HILN network with minimal effort, and provide a report via the CIM 145 regarding failed cases (e.g., incorrectly determined HILN index) and / or specimen samples that need more attention (e.g., more closely matching training images) via the assigned confidence level.

[0065] Note that in some embodiments, the hashing network 601 can be a front-end hashing network (such as a CNN) that is trained to recognize a particular class of images, and the HILN network 535 can be a back-end hashing network that is trained to recognize a particular class of images. In other embodiments, the hashing network 601 can be a back-end hashing network that is trained to recognize a particular class of images, and the HILN network 535 can be a front-end hashing network that is trained to recognize a particular class of images. Figure 5a back-end hash network, coupled to receive segmented regions of input images of specimens and specimen containers from the DSSN 538 (operating directly on segmented regions of specimens), or a stand-alone HILN network that incorporates and performs some or all of the functions of the CSN 536 and / or DSSN 538 of the HILN 535.

[0066] Figure 7 A flowchart of a characterization method 700 is illustrated in accordance with one or more embodiments. As described herein, the characterization method 700 can be carried out by the quality check module 130 of the automated diagnostic analysis system 100 (in conjunction with the computer 143 and programming instructions), and can include receiving, at process block 702, a plurality of training images for training a HILN (hemolytic, icteric, lipemic, normal) network of a quality check module comprising a computer in an automated diagnostic analysis system, each training image depicting a sample specimen in a specimen container. The training images can be, for example, the training images 605 of Figure 6 and / or can be captured by one or more of the image capture devices 440A-440C (of Figure 4A and 4B ), each of which can be a digital pixelated image. The training images are used to train the HILN network of the quality check module, such as, for example, the HILN network 535 of the quality check module 130.

[0067] The characterization method 700 can further include, in process block 704, assigning a hash code to each training image via a hash network of the HILN network. The hash code can be assigned by the hash network 601 (see Figure 5 and 6 ), and can be, for example, the hash code 603 of Figure 6 , where the hash codes of the training images 605 in a similar HILN class or group can have a minimum distance between them, while the hash codes 605 across HILN classes or groups can have a large distance.

[0068] In process block 706, the characterization method 700 can include receiving one or more images of a specimen in a specimen container to be analyzed in the automated diagnostic analysis system. For example, with reference to Figure 4A and 4B , specimen images taken from viewpoints 1-3 can be received from the image capture devices 440A, 440B, 440C. The specimen images can be, for example, the specimen images 606 of Figure 6 .

[0069] In process block 708, the characterization method 700 can include characterizing the specimen to be analyzed using the plurality of training images to determine a classification index including a hemolytic, icteric, lipemic, and normal class based on the one or more images received via the HILN network. For example, the classification index can be the classification index 540 (see Figure 5 ), which in some embodiments can include the following classes (and subclasses): 540U, 540N, 540H (H0, H1, H2, H3, H4, H5, H6), 540I (I0, I1, I2, I3, I4, I5, I6), and 540L (L0, L1, L2, L3, and L4).

[0070] In process block 710, the characterization method 700 can include retrieving one or more of the plurality of training images on which the determined classification index of the characterized specimen is based via the hash code. For example, with reference to Figure 6 , the hash network process 600 can include retrieving the features 610, which can retrieve the particular training image 612 on which the HILN determination of the specimen image 606 is based from the memory of the computer 143 (of FIGS. Figure 1 , 4A .

[0071] Accordingly, based on the foregoing, it should be apparent that an improved characterization method 700 is provided that facilitates debugging incorrect HILN determinations.

[0072] As should also be apparent, the above characterization method can be implemented using a quality check module (e.g., the quality check module 130) that includes a plurality of image capture devices (e.g., the image capture devices) 440A-440C arranged about an imaging location (e.g., the imaging location 432) and configured to capture one or more images from one or more viewpoints (e.g., the viewpoints 1-3 of FIG. Figure 4A ) of the specimen container 102 including one or more labels 218 and containing a specimen 212. The quality check module further includes a computer (e.g., the computer 143) coupled to the plurality of image capture devices and configured to process pixel data of the one or more images. The computer (e.g., the computer 143) can also be configured and operable to assign a hash code to each training image via the hash network, store the assigned hash codes in a memory or database of the computer 143, and provide the HILN determination and the particular training image on which the HILN determination is based (retrieved via the assigned hash code).

[0073] While the disclosure is susceptible to various modifications and alternative forms, specific methods and device embodiments have been shown by way of example in the drawings and are described in detail herein. However, it should be understood that the specific methods and device described herein are not intended to limit the disclosure but on the contrary, are intended to cover all modifications, equivalents, and alternatives falling within the scope of the claims.

Claims

1. A method for characterizing a sample in an automated diagnostic analysis system, comprising: The HILN network for training a quality inspection module, which includes a computer in an automated diagnostic analysis system, receives multiple training images, each training image depicting a sample specimen in a sample container. and Hash codes are assigned to each training image via a hash network of the HILN network, which determines a classification index for hemolytic, jaundiced, lipemic, or normal class.

2. The method for characterizing a sample according to claim 1, comprising: Receive one or more images of a sample in a sample container to be analyzed in an automated diagnostic analysis system; Using multiple training images via the HILN network, a sample to be analyzed is characterized based on one or more images to determine the classification index, including hemolytic, jaundiced, lipemic, or normal classes.

3. The method for characterizing a sample according to claim 2, comprising: The classification index of a sample is retrieved from one or more of the plurality of training images based on a hash of one or more of the plurality of training images.

4. The method for characterizing a sample according to claim 1, comprising: One or more images of a sample in a sample container with an incorrect HILN determination are compared with one or more of the plurality of training images on which the incorrect HILN determination is based.

5. The method for characterizing a sample according to claim 1, wherein the retrieval via hash is conditional upon determination that the sample's classification index is incorrect.

6. The method for characterizing a specimen according to claim 5, wherein the determination of an incorrect classification index for characterizing the specimen is based on a confidence level.

7. The method for characterizing a specimen according to claim 1, wherein the plurality of training images comprises multi-view images captured by a plurality of image capture devices.

8. The method for characterizing a sample according to claim 1, wherein the HILN network comprises a deep semantic segmentation network (DSSN).

9. The method for characterizing a sample according to claim 1, wherein each of the plurality of training images includes a classification, said classification including one of hemolytic, jaundiced, lipemic, and normal categories.

10. The method for characterizing a sample according to claim 1, wherein each of the plurality of training images includes a hemolytic subclass, a jaundice subclass, or a lipemia subclass.

11. The method for characterizing a sample according to claim 1, wherein the hash network includes a front-end hash network that assigns hash codes before segmentation.

12. The method for characterizing a specimen according to claim 1, wherein the hash network includes a back-end hash network configured to receive segmented regions of each training image and assign hash codes to segmented regions of the specimen.

13. The method for characterizing a specimen according to claim 1, wherein the assigned training image hash is stored in a computer database and retrieved later.

14. The method for characterizing a specimen according to claim 1, wherein the hash network assigns hash codes to each training image group representing the same sample specimen.

15. The method for characterizing a specimen according to claim 1, comprising retrieving via a hash of one or more of a plurality of training images on which the classification of the specimen is based, and presenting one or more of the plurality of training images to a user on a display screen.

16. The method for characterizing a specimen according to claim 1, comprising providing the HILN network with one or more additional training images representing specimens that are incorrectly characterized.

17. The method for characterizing a sample according to claim 1, wherein the characterization method is performed by a quality inspection module.

18. The method for characterizing a sample according to claim 1, wherein hash codes for training images within the same HILN class have a first distance between them, while hash codes across HILN classes have a greater distance between them.

19. A method for characterizing a sample, comprising: Multiple training images are received to train the HILN network, each training image depicting a sample specimen in a sample container. Hash codes are assigned to each training image via a hash network; Receive one or more images of a sample in a sample container to be analyzed by the HILN network; Multiple training images are used via the HILN network to characterize the sample to be analyzed based on one or more images to determine the classification index of hemolytic, jaundiced, lipemic, or normal class. as well as One or more of the training images on which the classification index is based are retrieved via the hash code of one or more of the training images.

20. A quality inspection module for an automated diagnostic analysis system, comprising: Multiple image capturing devices are arranged around the imaging position and configured to capture multiple images from multiple viewpoints of a sample container containing the sample. as well as A computer coupled to multiple image capture devices, the computer being configured and operable via programming instructions to perform the following operations: A first plurality of images, captured by multiple image capture devices, are input into a HILN network executed on a computer. These first plurality of images represent multiple training images used to train the HILN network, each training image depicting a sample specimen in a sample container. Hash codes are assigned to each training image via a hash network. One or more second images, captured by multiple image capture devices, are input into a HILN network executed on a computer. These one or more second images represent test specimens in a sample container to be analyzed in the automated diagnostic analysis system. Multiple training images are used via the HILN network to characterize the test specimen to be analyzed based on one or more second images to determine a classification index including hemolytic, jaundiced, lipemic, or normal categories. One or more of the training images on which the classification index of the test specimen is based are retrieved via hashing.

Images