Hypothesis and verification network and method for sample classification

Abstract

The invention relates to a hypothesis and verification network and method for sample classification. A method of characterizing a sample as comprising hemolysis, jaundice or lipemia is provided. The method includes capturing one or more images of the sample, wherein the one or more images include a serum or plasma portion of the sample. Pixel data is generated by capturing an image. Pixel data of one or more images of a sample is processed using a first network executing on a computer to predict a classification of serum or plasma portions, where the classification includes hemolysis, jaundice, and lipemia. The classification of the prediction is validated using one or more validation networks. Quality inspection modules and sample testing devices adapted to perform the methods are described, as are other aspects.

Description

[0001] This application is a divisional application of the invention patent application with national application number 201980061643.8, the application date of which is September 19, 2019, and the name of the invention is “Hypothesis and Verification Network and Method for Sample Classification”.

[0002] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 734,007, filed on September 20, 2018, and entitled “HYPOTHESIZE ANDHIERARCHICAL VERIFICATION NETWORK FOR CLASSIFICATION / REGRESSION TASK,” which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] Embodiments of the present disclosure relate to methods and apparatus for characterizing a sample in an automated diagnostic analysis system. Background Art

[0004] The automated diagnostic analysis system may use one or more reagents to perform an assay or clinical analysis to identify an analyte or other component in the serum or plasma portion of a blood sample. Improvements in automated testing technology have been accompanied by corresponding advances in the pre-analytical sample preparation and processing operations of automated sample preparation systems known as laboratory automation systems (LASs), such as sorting, batch preparation, centrifugation of sample containers to separate sample components, cover removal to facilitate fluid entry, pre-screening for HILN (hemolysis, icterus and / or lipemia or normal), and the like. LASs can also automatically transport samples in sample containers to multiple sample processing stations so that various operations (e.g., pre-analysis or analytical testing) can be performed thereon.

[0005] In the case where the sample is whole blood, a gel separator can be added to the sample container to help separate the settled blood portion from the serum or plasma portion. After pretreatment, the sample container can be transported to an appropriate analyzer, which can extract a portion of the biological fluid (e.g., serum or plasma portion) from the sample container and combine the fluid with one or more reagents in a reaction vessel (e.g., a cuvette) and possible other materials. Analytical measurements can then be performed via photometric or fluorescent absorption readings using interrogating radiation or beams of the like. This measurement allows determination of an end-point rate or other value, thereby determining the amount of an analyte or other component in the biological fluid.

[0006] However, the presence of any interferents (e.g., hemolysis, icterus, and / or lipemia) in a sample that may be derived from the patient's condition or the sampling process may adversely affect the test results of the analyte or component measurement obtained from one or more analyzers. For example, the presence of hemolysis (H) in a sample that may be unrelated to the patient's disease state may cause a different interpretation of the patient's disease state. In addition, the presence of icterus (I) and / or lipemia (L) in a sample may also cause a different interpretation of the patient's disease state.

[0007] In some systems, a skilled laboratory technician can visually inspect the integrity of the serum or plasma portion of the sample and rate the integrity of the serum or plasma portion of the sample as normal (N) or having a degree of H, I, and / or L (e.g., by assigning an index). This process may involve a review of the color of the serum or plasma portion against a known standard. However, such manual visual inspection is highly subjective, labor intensive, and may involve possible human error.

[0008] Because manual inspection can be problematic, efforts have been made to assess sample integrity without manual visual inspection by a laboratory technician, but rather by using automated machine vision inspection devices, where such assessment occurs during pre-analytical testing (hereinafter referred to as "pre-screening"). Pre-screening involves automated detection of interferents such as H, I, and / or L in serum or plasma fractions obtained from whole blood by fractionation (e.g., by centrifugation).

[0009] In some cases, one or more labels can be directly affixed to the sample container. Such labels may partially block and cover certain side viewpoints of the sample, so that there may be certain orientations that do not provide clear opportunities for visual observation of the serum or plasma portion. Therefore, the automation of this pre-screening includes, for example, rotating the oriented sample in such a way that automatic pre-screening of H, I and / or L or N is considered (see, for example, U.S. Patent No. 9,322,761). In other systems, the sample container and sample are imaged from multiple viewpoints and processed with a model-based system so that rotation of the sample container is not required (see, for example, WO 2016 / 133,900).

[0010] In some cases, only a small portion of the serum or plasma portion may be visible, so that any H, I and / or L or N readings taken on the serum or plasma portion may not involve a high level of confidence. Moreover, such systems may be complex and the processing of the image data may be computationally heavy.

[0011] Therefore, there is an unmet need for a robust and efficient method and apparatus for characterizing the serum or plasma portion of a sample to determine the presence and extent of hemolysis (H), icterus (I), and / or lipemia (L), or whether the serum or plasma portion of the sample is normal (N). More particularly, there is an unmet need for improved methods and apparatus for determining whether a sample includes H, I, and / or L and to what extent or whether it is normal (N). Summary of the invention

[0012] According to a first aspect, a method for characterizing a sample is provided. The method includes capturing one or more images of the sample with one or more image capture devices, the one or more images including a serum or plasma portion of the sample, the capturing generating pixel data; processing the pixel data of the one or more images of the sample using a first network executed on a computer to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and validating the predicted classification using one or more validation networks executed on the computer.

[0013] According to another aspect, a quality check module is provided. The quality check module includes a plurality of image capture devices operable to capture one or more images from one or more viewpoints of a sample container containing a serum or plasma portion of the sample; and a computer coupled to the plurality of image capture devices, the computer being configured and operable to: capture one or more images of the sample, the one or more images including the serum or plasma portion of the sample, capture to generate pixel data; process the pixel data of the one or more images of the sample using a first network executed on the computer to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and validate the predicted classification using one or more validation networks executed on the computer.

[0014] In another aspect, a sample testing device is provided. The sample testing device includes a track; a carrier that is movable on the track and configured to contain a sample container, wherein the sample container contains a serum or plasma portion of the sample; a plurality of image capture devices arranged around the track and operable to capture one or more images from one or more viewpoints of the sample container and the serum or plasma portion of the sample; and a computer coupled to the plurality of image capture devices, the computer configured and operable to: capture one or more images of the sample, the one or more images including the serum or plasma portion of the sample, capture to generate pixel data; use a first network executed on the computer to process the pixel data of the one or more images of the sample to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and use one or more validation networks executed on the computer to validate the predicted classification.

[0015] Other aspects, features and advantages of the present disclosure may be readily apparent from the following detailed description of multiple example embodiments and implementations of the best modes contemplated. The present disclosure may also be capable of other and different embodiments, and several details thereof may be modified in various aspects. Therefore, the drawings and descriptions are to be considered illustrative in nature and not restrictive. The drawings are not necessarily drawn to scale. The present disclosure encompasses all modifications, equivalents and substitutes that fall within the scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described below are for illustrative purposes and are not necessarily drawn to scale. Therefore, the drawings and descriptions should be considered illustrative in nature and should not be considered restrictive. The drawings are not intended to limit the scope of the present invention in any way.

[0017] Figure 1A A schematic top view of a quality inspection module (top removed) is shown according to one or more embodiments, which includes multiple viewpoints of a sample container and is configured to capture and analyze multiple backlit images of the sample container to enable determination of the presence and extent of interferents.

[0018] Figure 1B According to one or more embodiments, Figure 1A The section line 1B-1B is obtained Figure 1A Schematic side view of a quality inspection module (front housing wall removed).

[0019] Figure 1C A schematic top view of a sample testing device according to one or more embodiments is shown, wherein the sample testing device includes one or more quality inspection modules, and the quality inspection modules are configured to perform a HILN determination method.

[0020] Figure 2 A side view of a sample container including a separated sample having a serum or plasma portion and a settled blood portion is shown, wherein the sample container includes a label thereon, according to one or more embodiments.

[0021] Figure 3A A block diagram illustrating functional components of a quality check module including a hypothesis network and a verification network operable to output an interferent classification of a serum or plasma portion of a sample in a sample container and a degree of certainty of the classification, according to one or more embodiments.

[0022] Figure 3B A block diagram of functional components of a quality check module including a hypothesis network and an index verification network operable to output an interferent classification index and a degree of certainty of the classification index for a serum or plasma portion of a sample in a sample container is shown according to one or more embodiments.

[0023] Figure 4 A block diagram illustrating functional components of a quality check module including a segmentation convolutional neural network (SCNN) operable to output an interferent classification index for a serum or plasma portion of a sample in a sample container according to one or more embodiments.

[0024] Figure 5 According to one or more embodiments, Figure 4 Block diagram of the architecture of the Deep Semantic Segmentation Network (DSSN) of SCNN.

[0025] Figure 6 is a flow chart of a method of characterizing and testing a sample within a sample container according to one or more embodiments. DETAILED DESCRIPTION

[0026] As described, the presence of any interferents (e.g., hemolysis, icterus, and / or lipemia) in a sample that may be derived from the patient's condition or the sampling process may adversely affect the test results of the analyte or component measurements obtained from one or more analyzers. For example, the presence of hemolysis (H) in a sample that may be unrelated to the patient's disease state may cause a different interpretation of the patient's disease state. In addition, the presence of icterus (I) and / or lipemia (L) in a sample may also cause a different interpretation of the patient's disease state.

[0027] As used herein, interferences such as H, I and / or L or determination of normal (N) (hereinafter referred to as "HILN") refers to the presence of at least one of hemolysis (H), icterus (I) or lipemia (L) in the serum or plasma portion of the sample. "N" means "normal", which can be defined as a serum or plasma portion that includes an acceptable low amount of H, I and L. Hemolysis can be defined as a condition in the serum or plasma portion in which red blood cells are destroyed during processing, which causes hemoglobin to be released from the red blood cells into the serum or plasma portion, causing the serum or plasma portion to have a reddish hue. The extent of hemolysis can be quantified by assigning a hemolysis index. Jaundice can be defined as a condition of the blood in which the serum or plasma portion is decolorized to a dark yellow due to the accumulation of bile pigment (bilirubin). The extent of jaundice can be quantified by assigning a icteric index. Lipemia can be defined as the presence of abnormally high concentrations of emulsified fat in the blood, causing the serum or plasma portion to have a whitish or milky appearance. The degree of lipemia can be quantified by assigning a lipemia index.

[0028] The networks, systems and methods disclosed herein include networks (e.g., neural networks) that validate HIL predictions. For example, a first network (e.g., a hypothesis network) assumes or predicts the HILN classification of a sample and a network (e.g., a validation network) at a later stage validates the prediction. Image data or pixel data representing the sample may be input to a first network in which an assumption or prediction of the HILN classification is made. Based on the assumption of the HILN classification, the pixel data is further analyzed by one or more specialized validation networks. In some embodiments, one or more specialized networks may further analyze the last layer or a layer close to the last layer generated by the first network. In some embodiments, there may be a specialized validation network for each of the HIL or HILN classifications. For example, if the first network assumes a hemolysis classification in a sample, the pixel data may be further analyzed by a validation network trained to validate and / or detect hemolysis. Other validation networks may validate icteric and / or lipemia assumptions. In some embodiments, the system (s) may include similar validation networks to validate normal classifications. In some embodiments, the first network may predict or assume an uncentrifuged (U) category for the serum or plasma portion of a sample that has not yet been centrifuged.

[0029] In addition to the foregoing, methods, systems, and apparatus according to embodiments disclosed herein may include a first network that predicts or assumes a classification index for hemolysis (e.g., H0-H6 in some embodiments and more or less in other embodiments), a classification index for jaundice (e.g., I0-I6 in some embodiments and more or less in other embodiments), and / or a classification index for lipemia (e.g., L0-L4 in some embodiments and more or less in other embodiments). In some embodiments, classification may be performed by regression to output HIL concentration. The classification index may be referred to as the degree or subclass of a specific HIL classification. The results of the first network may be input into one or more index validation networks specific to the classification index of the interferent predicted by the first network. In some embodiments, the validation network may provide a weighted response (e.g., a percentage) identifying whether the prediction of the first network is correct. In other embodiments, the validation network may provide a binary output identifying whether the prediction of the first network is correct or incorrect.

[0030] In some embodiments, the first network can classify (or "segment") various regions of the sample container and sample, such as a serum or plasma portion, a sedimented blood portion, a gel separator (if used), air, a label, the type of sample container (indicating, for example, height and width / diameter), and / or the type and / or color of the sample container cap. The sample container holder and / or background can also be classified.

[0031] In view of the above challenges, in a first broad aspect, embodiments of the present disclosure provide methods and apparatus configured to determine the presence and / or extent of HILN. A first network (e.g., a hypothesis network) may use a segmentation convolutional neural network (SCNN) that includes a deep semantic segmentation network (DSSN) that outputs a predicted classification and / or classification index of a sample. In some embodiments, the DSSN may predict a classification index of a serum or plasma portion of a sample identified in a sample container based at least in part on a color represented by pixel data of the serum or plasma portion. In some embodiments, the DSSN may include more than 100 operational layers, which are further described below. The SCNN may also include a front-end container segmentation network (CSN) to determine container types and container boundaries. The determined container information may then be input to the DSSN via an additional input channel, as further described below. Based on the classification prediction, further analysis may be performed by a professional verification network to verify the prediction. In some embodiments, the apparatus and method may include at least one index verification network for each HIL classification index.

[0032] In some embodiments, the input to the SCNN can be multi-spectral, multi-exposure pixel data or image data, which can be merged and normalized and obtained from one or more image capture devices. The image capture device can be any device capable of capturing a pixelated image (e.g., a digital image) for analysis, such as a digital camera, a CCD (charge coupled device), one or more CMOS (complementary metal oxide semiconductor) sensors, a sensor array, or the like. Two or more image capture devices can be arranged and configured to capture images from multiple viewpoints (e.g., three viewpoints; other numbers of viewpoints are possible). In some embodiments, the methods, apparatuses, and systems described herein can use high dynamic range (HDR) image processing of sample containers and serum or plasma portions as input to the SCNN. HDR imaging may involve capturing multiple exposures when using multiple spectral illumination. In some embodiments, the SCNN is trained to identify areas occluded by one or more labels on the sample container so that the SCNN can better account for the presence of labels on the back side of the sample container from any viewpoint in characterizing H, I, L, or N.

[0033] According to some aspects, the characterization method can be performed by a quality inspection module and performed in a sample test system, each sample test system including a SCNN. The SCNN may include an operation layer, including, for example, BatchNorm, ReLU activation, convolution (e.g., 2D), dropout, and deconvolution (e.g., 2D) layers to extract features such as simple edges, textures, and parts of the serum or plasma portion and the region containing the label. Top layers such as full convolutional layers can be used to provide correlations between parts. The output of the layer can be fed to a SoftMax layer, which generates an output on a per-pixel (or per-patch-including nxn pixels) basis about whether each pixel or patch contains H, I, L, or N. In some embodiments, only H, I, L, or N outputs can be provided from the SCNN. In other embodiments, the output of the SCNN may be H, I, L, or N with a fine granularity of 20 serum categories or indices greater than H, I, L, or N, so that for each interferent present, an estimate of the level (index) of the interferent is also obtained.

[0034] If the sample is found to contain one or more of H, I, and L, an appropriate notification may be provided to the operator, and / or the sample container may be taken off line (1) to perform remediation to correct one or more of H, I, or L, (2) to redraw the sample, or (3) to perform other processing. Thus, the ability to pre-screen for H, I, L, or N at the first possible instance, such as after centrifugation and prior to analysis by one or more analyzers, may advantageously (a) minimize time wasted analyzing samples that are not of appropriate quality for analysis, (b) avoid or minimize erroneous test results, (c) minimize delays in patient test results, and / or (d) avoid wasting patient samples.

[0035] Reference this article Figure 1A-6 Further details of the characterization and validation methods, apparatus, systems and quality check modules configured to perform the characterization and validation methods, and sample testing apparatus including one or more quality check modules are further described.

[0036] Reference now Figure 1A and Figure 1B, which illustrate an embodiment of a quality check module 100 configured to perform the characterization and verification methods shown and described herein. The quality check module 100 can be configured to pre-screen the presence and extent (e.g., index) of interferents (e.g., H, I, and / or L) in a sample 104 located in a sample container 106 before analysis by one or more analyzers, one or more analyzers being coupled to the quality check module. This manner of pre-screening allows for additional processing, additional quantification or characterization, and / or discarding and / or redrawing of the sample 104 without wasting valuable analyzer resources or the possibility of interferents affecting the veracity of the test results. The quality check module 100 can also determine whether the sample 104 is normal.

[0037] In addition to the interferent detection methods described herein, other detection methods may be performed at the quality inspection module 100 on the sample 104 contained in the sample container 106. For example, a method may be performed at the quality inspection module 100 to provide segmentation as a result of a container segmentation network (CSN 462- Figure 4 The segmented data may be used in a post-processing step to quantify the sample 104, ie, to determine certain physical dimensional characteristics of the sample 104.

[0038] Additional references Figure 2 , which shows a side view of a sample container 106 including a sample 104 according to one or more embodiments. The sample 104 may include a gel separator 207 that separates a serum or plasma portion 204SP of the sample 104 from a settled blood portion 204SB of the sample 104. Air 208 may be provided above the serum or plasma portion 204SP, and a line of demarcation between them is defined as a liquid-air interface (LA). A cap 209 may seal the sample container 106. Physical properties determined at least in part by segmentation may include a position of a top (TC) of the sample container 106, a height (HT) of the sample container 106, a width (W) of the sample container 106, an inner width (Wi) of the sample container 106, and a thickness (Tw) of a wall of the sample container 106. In addition, the segmentation data may provide a position liquid-air interface (LA), a total height (HTOT) of the sample 104, and an upper position (GU) and a lower position (GL) of the gel separator 207. The difference between the upper position (GU) and the lower position (GL) provides the height of the gel separator 207. The characteristics may further include the height HSP of the serum or plasma portion 204SP and the height HSB of the settled blood portion 204SB. Quantification may also involve estimating the volume of the serum or plasma portion and / or the volume of the settled blood portion. Other quantifiable geometric features, such as the size and color of the cap 209, may also be determined.

[0039] The sample container 106 can be provided with a label 210, which can include identification information 210i (i.e., markings) thereon, such as a bar code, letters, numbers, or a combination thereof. The identification information 210i can be machine readable at various locations including at the quality inspection module 100. The machine-readable information can be darker (e.g., black) than the label material (e.g., white paper) so that it can be easily imaged. The identification information 210i can indicate the patient's identification and the tests to be performed on the sample 104 or may otherwise be associated with them via a laboratory information system (LIS). The identification information 210i can indicate other or additional information. The above-mentioned identification information 210i can be provided on the label 210, which can be affixed to or otherwise provided on the outer surface of the sample container 106. As shown in Figure 2 As shown in , the label 210 may not extend all the way around the sample container 106 or all the way along the length of the sample container 106, so that from the particular front viewpoint shown, a majority of the serum or plasma portion 204SP is visible (shown in dashed lines) and is not obstructed by the label 210.

[0040] In some embodiments, multiple labels may have been provided (such as from multiple facilities that have processed the sample container 106), and the multiple labels may overlap with each other to some extent. For example, two labels (e.g., a manufacturer's label and a barcode label) may be provided and the two labels may overlap and may obstruct (block) some or all of one or more viewpoints. Therefore, it should be understood that in some embodiments, although (one or more) labels 210 may obstruct a portion of the sample 104 (the obstructed portion), from at least one viewpoint (the unobstructed portion), the sample 104 and a portion of the serum or plasma portion 204SP may still be visible. Therefore, according to another aspect of the present disclosure, an embodiment of a SCNN configured to perform a characterization method can be trained to identify obstructed portions and unobstructed portions, so that improved HILN detection can be provided.

[0041] Reference again Figure 1A and 1B , the quality inspection module 100 may include one or more image capture devices 110A, 110B, 110C, which image the sample container 106 through viewpoints 1, 2, and 3, respectively. Figure 1A Three image capture devices 110A-110C are shown, but alternatively, two or four or more may be used. The image capture devices 110A-110C may be any suitable device for capturing a well-defined digital image, such as a conventional digital camera capable of capturing a pixelated image, a charge coupled device (CCD), an array of photodetectors, one or more CMOS sensors, or the like. Figure 1AThe three image capture devices 110A-110C shown in FIG. 1 are configured to capture images from three different lateral viewpoints (viewpoints labeled 1, 2, and 3). In some embodiments, the captured image size may be, for example, approximately 2560×694 pixels. In another embodiment, the image capture devices 110A-110C may capture an image size that may be, for example, approximately 1280×387 pixels. Other image sizes and pixel densities may be used.

[0042] Each image capture device 110A-110C can be configured and operable to capture a lateral image of at least a portion of the sample container 106 and at least a portion of the sample 104. For example, the image capture device 110A-110C can capture a portion of the label 210 and part or all of the serum or plasma portion 204SP. In some cases, the label 210 can partially obstruct portions of the viewpoints 1-3. In some embodiments, one or more of the viewpoints 1-3 can be completely obstructed, that is, a clear view of the serum or plasma portion 204SP may be possible without. However, even in the case where one side (front or back) of the viewpoint 1-3 is completely obstructed by one or more labels 210, the characterization method may still be able to distinguish the boundaries of the serum or plasma portion 204SP through one or more labels 210.

[0043] In the illustrated embodiment, a plurality of image capture devices 110A-110C are configured to capture lateral images of a sample container 106 and a sample 104 at an imaging position 116 from a plurality of viewpoints 1-3. As shown, the viewpoints 1-3 may be arranged so that they are approximately equally spaced from one another, such as approximately 120° from one another. As depicted, the image capture devices 110A-110C may be arranged around a track 118 on which the sample container 106 is transported. Other arrangements of the plurality of image capture devices 110A-110C may be used. In this way, an image of the sample 104 in the sample container 106 may be acquired while the sample container 106 resides in a carrier 120 at the imaging position 116. The fields of view of the plurality of images obtained by the image capture devices 110A-110C may overlap slightly in a circumferential extent.

[0044] In one or more embodiments, the carrier 120 may be stopped at a predetermined position in the quality inspection module 100, such as the imaging position 116. At this position, the normal vectors from each image capture device 110A-110C intersect each other. A gate or linear motor (not shown) of the carrier 120 may be provided to stop the carrier 120 at the imaging position 116 so that multiple images can be captured there. In embodiments where a gate is present at the quality inspection module 100, one or more sensors (not shown) may be used to determine the presence of the carrier 120 at the quality inspection module 100.

[0045] In some embodiments, the quality inspection module 100 may include a housing 128 that may at least partially surround or cover the track 118 to minimize external lighting effects. During the image capture sequence, the sample container 106 may be located inside the housing 128. The housing 128 may include one or more openings (and / or doors) 128D to allow the carrier 120 to enter the housing 128 and / or exit from the housing 128. In some embodiments, the top plate may include an opening 128D to allow the sample container 106 to be loaded into the carrier 120 from above by a robot (not shown) including a movable robot finger.

[0046] The image capture devices 110A-110C may be provided in close proximity and trained or focused to capture an image window at an imaging location 116, where the image window is an area that includes the expected location of the sample container 106. Thus, in some embodiments, the sample container 106 may be stopped so that it is approximately centered in the image window.

[0047] In operation of the quality inspection module 100, each image may be triggered and captured in response to a trigger signal provided in the communication lines 122A, 122B, 122C that may be sent by the computer 124. Each captured image may be processed by the computer 124 in accordance with one or more embodiments described herein. In some methods, high dynamic range (HDR) processing may be used to capture and process image data from the captured images. In more detail, multiple images of the sample 104 are captured at the quality inspection module 100 at multiple different exposures (e.g., at different exposure times) while being sequentially illuminated with one or more different spectra. For example, at different exposure times with each of the multiple spectra, each image capture device 110A-110C may capture a portion 204SP ( Figure 2). For example, when sample 104 is backlit by light source 126A having a red spectrum, image capture device 110A may capture 4-8 images at viewpoint 1. Additional similar images may be captured sequentially at viewpoints 2 and 3. Other numbers of images may be captured.

[0048] In some embodiments, different light sources 126A, 126B, and 126C that emit different spectrum illumination can be used to complete capturing multiple spectrum images. Light sources 126A-126C can background illuminate sample container 106 (as shown). In some embodiments, a light diffuser can be used in conjunction with light sources 126A-126C. Multiple light sources 126A-126C of different spectrums can be red, green, and blue (RGB) light sources, such as light emitting diodes (LEDs) emitting nominal wavelengths of 634nm+ / -35nm (red), 537nm+ / -35nm (green), and 455nm+ / -35nm (blue). In other embodiments, light sources 126A-126C can be white light sources. In the case where label 210 covers multiple viewpoints, infrared (IR) backlighting or near infrared (NIR) backlighting can be used. In addition, even when there is a label obstruction, RGB light sources can also be used in some cases. In other embodiments, the light sources 126A-126C may emit one or more spectra having a nominal wavelength between about 700 nm and about 1200 nm.Other light sources and / or wavelengths may be used.

[0049] As a non-limiting example, to capture images at a first wavelength, three red light sources 126A-126C (wavelengths of approximately 634 nm + / - 35 nm) may be used to sequentially illuminate the sample 104 from three lateral positions. The red illumination of the light sources 126A-126C may occur when each image capture device 110A-110C captures multiple images (e.g., 4-8 images or more) at different exposure times from each viewpoint 1-3. In some embodiments, the exposure time may be between approximately 0.1 ms and 256 ms. Other exposure times may be used. In some embodiments, for example, each of the individual images of each image capture device 110A-110C may be acquired sequentially. Thus, for each viewpoint 1-3, a group of images having red spectrum backlighting and multiple exposures (e.g., 4-8 exposures, such as different exposure times) may be sequentially obtained. Images may be captured in a round robin manner, for example, where all images from viewpoint 1 are captured, and then viewpoints 2 and 3 are captured in sequence.

[0050] Once in Figure 1A-1BIn an embodiment where an image of red illumination is captured, another spectrum of light, such as from a green spectrum light source 126A-126C (nominal wavelength of about 537 nm with a bandwidth of about + / - 35 nm) may be turned on, and multiple images (e.g., 4-8 or more images) at different exposure times may be captured sequentially by each image capture device 110A-110C. This may be repeated for each image capture device 110A-110C with a blue spectrum light source 126A-126C (nominal wavelength of about 455 nm with a bandwidth of about + / - 35 nm). For example, the different nominal wavelength spectrum light sources 126A-126C may be accomplished by a light panel that includes a bank of different desired spectrum light sources (e.g., R, G, B, W, IR, and / or NIR) that may be selectively turned on and off. Other means for backlighting may be used.

[0051] In some embodiments, multiple images acquired at multiple exposures (e.g., exposure times) for each corresponding wavelength spectrum may be obtained in rapid succession, such that an entire collection of backlit images of sample container 106 and sample 104 from multiple viewpoints 1-3 may be obtained in, for example, less than a few seconds. In some embodiments, four different exposure images for each wavelength at three viewpoints 1-3 using image capture devices 110A-110C and background illumination with RGB light sources 126A-126C will result in 4 images x 3 spectra x 3 image capture devices = 36 images. In another embodiment, four different exposure images for each wavelength at three viewpoints using image capture devices 110A-110C and background illumination with R, G, B, W, IR, and NIR light sources 126A-126C will result in 4 images x 6 spectra x 3 cameras = 72 images. Other numbers of images may be used.

[0052] According to an embodiment of the characterization and verification method provided herein, processing of image data or pixel data may involve a preprocessing step that includes, for example, selecting optimally exposed pixels from multiple captured images at different exposure times at each wavelength spectrum and for each image capture device 110A-110C so as to generate optimally exposed pixel data for each spectrum and for each viewpoint 1-3.

[0053] For each corresponding pixel (or patch of pixels), a pixel (or patch) exhibiting the best image intensity may be selected from each of the different exposure images for each viewpoint 1-3 according to each image captured by each image capture device 110A-110C. In some embodiments, the best image intensity may be a pixel (or patch) that falls within a predetermined range of intensities, such as, for example, between 180 and 254 on a scale of 0-255. In another embodiment, the best image intensity may be, for example, between 16 and 254 on a scale of 0-255. For example, if more than one pixel (or patch) in the corresponding pixel (or patch) positions of two exposure images is determined to be best exposed, the higher of the two may be selected.

[0054] The selected pixels (or patches) exhibiting the best image intensities may be normalized by their corresponding exposure times. The result is a plurality of normalized and merged spectral image data sets for the illumination spectrum (e.g., R, G, B, white light, IR and / or NIR - depending on the combination used) and for each image capture device 110A-110C, in which all pixels (or patches) are optimally exposed (e.g., one image data set per spectrum) and normalized. In other words, for each viewpoint 1-3, the data pre-processing performed by the computer 124 may result in a plurality of optimally exposed and normalized image data sets, one for each illumination spectrum employed.

[0055] Figure 1C An example sample detection device 150 is shown that can automatically process multiple sample containers 106 containing samples 104. Sample containers 106 can be provided in one or more racks 152 at a loading area 154 before being transported to one or more analyzers (e.g., a first analyzer 156, a second analyzer 158, and / or a third analyzer 160) and analyzed by one or more analyzers (e.g., a first analyzer 156, a second analyzer 158, and / or a third analyzer 160), and one or more analyzers (e.g., a first analyzer 156, a second analyzer 158, and / or a third analyzer 160) are arranged around the sample detection device 150. A greater or lesser number of analyzers can be used. The analyzer can be a clinical chemistry analyzer and / or an assay instrument or any combination of the like. The sample container 106 may be any suitable transparent or translucent container, such as a blood collection tube, test tube, sampling cup, cuvette, or other transparent or opaque glass or plastic container that is capable of containing and allowing imaging of the sample 104 contained therein. The size of the sample container 106 may vary.

[0056] In more detail, the sample detection device 150 may include a base 162 (e.g., a frame, floor, or other structure) on which the track 118 may be mounted. The track 118 may be a railed track (e.g., a single track or multiple tracks), a conveyor belt, a conveyor chain, a collection of movable platforms, or any other suitable type of conveying mechanism. In some embodiments, the track 118 may be circular or any other suitable shape, and may be a closed track (e.g., an endless track). The track 118 may transport individual sample containers in the sample container 106 in the carrier 120 to various locations spaced around the track 118 during operation.

[0057] The carrier 120 can be a passive, non-motored puck that can be configured to carry a single sample container 106 on the track 118, or alternatively, can be an automated carrier that includes an onboard drive motor, such as a linear motor, that is programmed to move around the track 118 and stop at a preprogrammed position. Other configurations of the carrier 120 can be used. In some embodiments, after being unloaded from one or more racks 152, the carrier 120 can leave the loading area 154. The loading area 154 can serve a dual function by also allowing the sample container 106 to be reloaded from the carrier 120 to the loading area 154 after pre-screening and / or analysis is completed.

[0058] A robot 166 may be provided at the loading area 154 and may be configured to grab sample containers 106 from one or more racks 152 and load the sample containers 106 onto the carrier 120, such as onto an input channel of the track 118. The robot 166 may also be configured to reload the sample containers 106 from the carrier 120 to one or more racks 152. The robot 166 may include one or more (e.g., at least two) robotic arms or components capable of X (lateral) and Z (vertical - out of the page, as shown), Y and Z, X, Y and Z, or r (radial) and theta (rotational) motion. The robot 166 may be a gantry robot, a multi-jointed robot, an R-theta robot, or other suitable robot, wherein the robot 166 may be equipped with a robotic gripper finger that is oriented, sized, and configured to pick up and place the sample container 106.

[0059] Upon being loaded onto the track 118, the sample containers 106 carried by the carrier 120 may proceed to a first pre-processing station 168. For example, the first pre-processing station 168 may be an automated centrifuge configured to perform fractionation of the sample 104. The carrier 120 carrying the sample containers 106 may be transferred to the first pre-processing station 168 via an inflow channel or other suitable robot. After being centrifuged, the sample containers 106 may exit on an outflow channel, or otherwise be removed by the robot, and continue along the track 118. In the depicted embodiment, the sample containers 106 in the carrier 120 may then be transported to the quality inspection module 100 to perform a pre-screening as further described herein.

[0060] Additional station(s) may be provided at one or more locations on or along the track 118. The additional station(s) may include a de-capping station, an aliquoting station, one or more additional quality inspection modules 100, and the like.

[0061] The sample detection device 150 may include a plurality of sensors 170 at one or more locations around the track 118. The sensors 170 may be used to detect the position of the sample container 106 on the track 118 by reading identification information 210i or similar information (not shown) provided on each carrier 120. Any suitable means for tracking the position may be used, such as proximity sensors. All sensors 170 may be interfaced with a computer 172 so that the position of each sample container 106 is always known.

[0062] The pre-processing stations and analyzers 156 , 158 , and 160 may be equipped with robotic mechanisms and / or inflow channels configured to remove carriers 120 from the track 118 , and with robotic mechanisms and / or outflow channels configured to re-enter the track 118 .

[0063] The sample detection device 150 can be controlled by a computer 172, which can be a microprocessor-based central processing unit (CPU) with suitable memory and suitable regulating electronics and drivers for operating various system components. The computer 172 can be housed or separated as part of the base 162 of the sample detection device 150. The computer 172 can be operable to control the movement of the carrier 120 to and from the loading area 154, the movement around the track 118, the movement to and from the first pre-processing station 168 and the operation of the first pre-processing station 168 (e.g., a centrifuge), the movement to and from the quality inspection module 100 and the operation of the quality inspection module 100 and the movement to and from each analyzer 156, 158, 160 and the operation of each analyzer 156, 158, 160 for performing various types of tests (e.g., assays or clinical chemistry). Figure 1A and 1B The computer 124 may be Figure 1C Part of or separate from computer 172.

[0064] Additional References Figure 3A , which shows an apparatus 330A, which includes functional components configured to perform the HIL or HILN characterization and verification method described herein. The apparatus 330A may be embodied in a quality inspection module 100 ( Figures 1A-1C ). For example, each functional component can be implemented in software, hardware, or a combination of software and hardware. As described above, the sample container 106 can be provided at the imaging position 116 ( Figure 1A ). Multi-view images of the sample container 106 may be captured in the image collection subsystem 332 by one or more of the plurality of image capture devices 110A-110C. As described above, pixel data of each of the multi-view, multi-spectral, multi-exposure images may be pre-processed in the pre-processing subsystem 334 to provide a plurality of optimally exposed and normalized image data sets (hereinafter referred to as "image data sets"). The hypothesis network 338 may use one or more image data sets of the images of the sample container to predict the sample classification. In some embodiments, the hypothesis network 338 may be a segmentation network and / or a classification network.

[0065] Given one or more sets of image data, the hypothesis network 338 is operable to assign a HILN classification to each pixel of the image based on the local appearance of each pixel of the image indicated by the pixel data value of each pixel of the image. In some embodiments, the hypothesis network 338 can predict the classification of the serum or plasma portion of the sample based on color or at least in part based on the color represented by the pixel data of the serum or plasma portion. In some embodiments, the classification can include an uncentrifuged category. The classification can include fewer or more categories and subclasses. In some embodiments, the hypothesis network can determine whether the sample has been centrifuged. If the sample has not been centrifuged, a notification can be generated to notify the laboratory technician that the HIL classification cannot be performed. In addition, the device 330A can skip the HIL classification of uncentrifuged samples.

[0066] The device 330A verifies the predicted classification of the sample in a verification subsystem 340. The verification subsystem 340 may include or be coupled to one or more verification networks 342. Figure 3A The embodiment shown in includes four validation networks 342. A hemolysis validation network 342A (sometimes referred to as a "first validation network") can be trained on a hemolysis classification to validate a hemolysis prediction. A jaundice validation network 342B (sometimes referred to as a "second validation network") can be trained on a jaundice classification to validate a jaundice prediction. A lipemia validation network 342C (sometimes referred to as a "third validation network") can be trained on a lipemia classification to validate a lipemia prediction. A fourth validation network 342D can be trained to validate a normal prediction. Some embodiments do not include a fourth validation network 342D and do not validate a normal classification. Other numbers and types of validation networks 342 may be used.

[0067] For example, during operation of the device 330A, the hypothesis network 338 may predict a hemolysis classification of the sample. The validation subsystem 340 may then transmit one or more sets of image data to a hemolysis validation network 342A, which may be separately trained to validate hemolysis. The hemolysis validation network 342A may generate a signal indicating a correct or incorrect hemolysis prediction.

[0068] Device 330A can output a verified classification (using output subsystem 344), which can be generated based on the output of verification network 342. The verified output of output subsystem 344 can be based on the positive result generated by verification network 342. For example, verification network 342 can verify or reject the prediction of the HIL classification generated by hypothesis network 338. If verification network 342 verifies hypothesis network 338, output subsystem 344 can output the HIL determination of hypothesis network 338 with a certainty greater than a predetermined level of certainty. For example, the output of verification and therefore the output of device 330A can determine with 98% certainty that the HIL classification is correct. Other levels of certainty can be used. In some embodiments, the output of verification can be a signal indicating that device 330A cannot determine the HIL classification within a predetermined level of certainty. In some cases, the output of verification can indicate that the sample should be analyzed manually.

[0069] Additional references Figure 3B , which shows an embodiment of a device 330B for predicting a classification index and verifying the predicted classification index of a sample. For example, the device 330B predicts and verifies the classification index within the HIL category. The device 330B is similar to Figure 3A The device 330A includes a hypothetical network 348, which replaces Figure 3A The hypothesis network 338 is constructed and the classification index of the sample is predicted. In some embodiments, the classification index may include 21 serum categories, including an uncentrifuged category, a normal category, and 19 HIL categories / subcategories. The classification index may include fewer or more or different categories and subcategories.

[0070] The device 330B includes a Figure 3A The validation subsystem 340 of the HIL classification index is used to validate the predicted classification index, such as by using the index validation network 352. The index validation network 352 can be trained to validate a specific HIL classification index. For example, one or more index validation networks 352 can be binary classifiers that determine whether a hypothesis generated by the hypothesis network 339 or the hypothesis network 348 is correct.

[0071] The hemolytic index validation network 352A (sometimes referred to as the "first index validation network") can be trained to validate an individual hemolytic classification index. The icteric index validation network 352B (sometimes referred to as the "second index validation network") can be trained to validate an individual icteric classification index. The lipemia index validation network 352C (sometimes referred to as the "third index validation network") can be trained to validate an individual lipemia classification index. The index validation network 352 may include a fourth validation network 340D, which may be trained to validate a normal classification as described above. Some embodiments do not include the fourth validation network 342D and do not validate a normal classification. Other numbers and types of index validation networks may be used.

[0072] The hemolytic index validation network 352A can validate multiple individual predicted hemolytic classification indices. In some embodiments, the hemolytic index validation network 352A can include an individual neural network, such as a CNN, to validate each predicted hemolytic classification index. The icteric index validation network 352B can validate multiple individual predicted icteric classification indices. In some embodiments, the icteric index validation network 352B can include an individual neural network, such as a CNN, to validate each predicted icteric classification index. In the same manner, the lipidemia index validation network 352C can validate multiple predicted lipidemia classification indices. In some embodiments, the lipidemia index validation network can include an individual neural network, such as a CNN, to validate each lipidemia classification index.

[0073] As an example of the operation of the device 330B, assume that the network 348 can, for example, predict the hemolysis classification index of H3 of the sample. The verification subsystem 350 can then transmit one or more sets of image data to the hemolysis index verification network 352A, which can be trained separately to verify the hemolysis classification index of H3. The hemolysis index verification network 352A can generate a signal indicating a correct or incorrect prediction. In some embodiments, the index verification network 352 can generate a weighted response related to the certainty of the prediction, such as a percentage. For example, the weighted response can determine whether the prediction exceeds a predetermined confidence level.

[0074] The device 330B may use an output subsystem 354 to output a verified classification index, which may be generated based on the output of the index verification network 352. The verified output of the output subsystem 354 may be based on a positive result generated by the index verification network 352. For example, the index verification network 352 may verify or reject the prediction of the HIL classification index predicted by the hypothesis network 348. If the index verification network 352 verifies the hypothesis network 348, the output subsystem 354 may output the HIL classification index predicted by the hypothesis network 348 at a predefined level of certainty. For example, the output of the verification and therefore the output of the device 330B may have a 98% certainty that the predicted HIL classification index is correct. Other levels of certainty may be used. In some embodiments, the output of the verification may be a signal indicating that the device 330B cannot determine the HIL classification index within a predetermined certainty. In some cases, the output of the verification may indicate that the sample should be analyzed manually.

[0075] In some cases, the first classification index may be close to the second classification index. Confusion may occur between the first classification index and the second classification index, such as identifying the first classification index as the second classification index or vice versa. To overcome these problems, the index verification network 352 may include a verification network that distinguishes between the classification indices. For example, the index verification network may include a network such as an expert network, which is trained to distinguish the above-mentioned first classification index from the second classification index. Therefore, confusion between close classification indices is alleviated.

[0076] Figure 4 An embodiment of a hypothesis network 458 is shown that includes functional components configured to perform the HIL(N) hypothesis method described herein. Figure 4 The assumption network 458 depicted in FIG. 1 is coupled to a plurality of index verification networks 352 (eg, 352A-C). The assumption network 458 may be embodied in a quality check module 100 ( Figures 1A-1C ) (e.g., in communication with or in a memory within computer 124 and / or 172). Assume that network 458 can be a neural network, such as, for example, a convolutional neural network. As described above, sample container 106 can be provided at imaging position 116 ( Figure 1A ) at the location. The multi-view images may be captured by one or more image capture devices 110A-110C ( Figure 1A ) capture. As described above, the image data or pixel data of each of the multi-view, multi-spectral, multi-exposure images may be pre-processed to provide multiple optimal exposure and / or normalized image data sets. Computers 124 and / or 172 may pre-process the pixel data to generate image data sets. The image data sets may be provided as input to a hypothesis network 458, which may be or include a segmentation convolutional neural network (SCNN) 460. The hypothesis network 458 and / or the SCNN 460 may be a program running on the computer 124 and / or 172.

[0077] The task performed by hypothesis network 458 is to characterize samples in a sample container such as sample container 106 (e.g., Figure 1A-1B This task may include separation of the sample container from its background and segmentation of any label affixed to the sample container, followed by the serum or plasma portion 204SP ( Figure 2) classification. All of these tasks can be accomplished using a SCNN 460 that performs pixel-level classification. Given an input image data set including pixel data, the SCNN 460 is operable to assign a classification index to each pixel of the image based on the local appearance of each pixel of the image as indicated by the pixel data value of each pixel of the image. The extracted pixel index information can be further processed by the SCNN 460 to determine the final HILN classification and / or the final HILN classification index. In some embodiments, the classification index may include 21 serum categories, including an uncentrifuged category, a normal category, and 19 HIL categories / subcategories (indices), as described in more detail below. The classification and / or classification index may include different, fewer, or more categories and subcategories. As described above, the challenge of determining the HIL classification or classification index may stem from small appearance differences in each subcategory of the H, I, and L categories. As described above, additional challenges stem from different sample container sizes and shapes and various sample container label occlusions that may occur. For example, a sample container label may block the view of the sample in the sample container or may generate a shadow area on the back of the sample container, which may darken the appearance of the sample.

[0078] To overcome these challenges, SCNN 460 may include a (very) deep semantic segmentation network (DSSN) 462, which in some embodiments includes more than 100 operating layers. The benefit of having such a deep network is that it can increase the network receptive field, utilizing more contextual information rather than small local areas. In addition, the densely connected network concatenates features from low-level to high-level layers, as well as from encoding to decoding areas. By doing so, DSSN 462 can be more easily trained to recognize more detailed features from an image than other convolutional neural networks configured for the same task. SCNN 460 may include other neural networks.

[0079] To overcome appearance differences that may be caused by variations in sample container types (e.g., size and / or shape), SCNN 460 may include a small container segmentation network (SCN) 464 at the front end of DSSN 462. SCN 464 may be configured and operable to determine container types and container boundary information 466. Container types and container boundary information 466 may be input to DSSN 462 via additional input channels, and in some embodiments, SCNN 460 may provide as output determined container types and boundaries 468. In some embodiments, SCN 464 may have a similar network structure as DSSN 462, but is shallower (i.e., has far fewer layers).

[0080] To overcome the appearance differences that may be caused by (one or more) sample container labels, the network loss function can be configured to support multi-label losses. That is, in some known networks, only one value can be output for each image pixel. In SCNN 460, two values ​​can be output for each image pixel - one for the serum class and one for the sample container label class. The value of the sample container label class can range from 0 (meaning no sample container label) to 3 (meaning that the pixel may be covered by three layers of labels). By combining the two values, SCNN 460 is able to better determine the serum or plasma portion class of the sample fluid area obscured behind one or more sample container labels.

[0081] like Figure 4 As shown in , the output of SCNN 460 can be a predicted classification (or classification index) 470, which in some embodiments can include an uncentrifuged class 470U, a normal class 470N, a hemolysis class 470H, an icteric class 470I, and a lipemia class 470L (or corresponding class indexes). In some embodiments, the hemolysis class 470H can include subclasses or indices H0, H1, H2, H3, H4, and H5. The icteric class 470I can include subclasses or indices I0, I1, I2, I3, I4, and I5. The lipemia class 470L can include subclasses L0, L1, L2, and L3. In other embodiments, each of the hemolysis class 470H, the icteric class 470I, and / or the lipemia class 470L can have other numbers of subclasses or indices.

[0082] The results of the predicted classification (or classification index) 470 may be forwarded to the reference verification subsystem 340 ( Figure 3A ) described verification network 342 ( Figure 3A ) or forwarded to the reference verification subsystem 350 ( Figure 3B ) The index validation network 352 described in the foregoing. The SCNN 460 can employ a neural network, such as a convolutional neural network, running on the computer 124 and / or 172 to generate a predicted classification or classification index 470.

[0083] Figure 4The index validation network 352 depicted in the figure may include an uncentrifuged validation network 472 that validates the following prediction of the hypothesis network 458: the sample 104 has not yet been separated into the serum or plasma portion 204SP and the sedimented blood portion 204SB. The index validation network 352 may include multiple individual validation networks, each of which may be trained to validate an individual classification index. For example, the first individual validation network may be trained separately to validate H2, and the second individual validation network may be trained separately to validate H3. In some embodiments, each individual validation network may be a neural network, such as a convolutional neural network. In some embodiments, the individual validation networks may be DenseNet and / or ResNet networks.

[0084] exist Figure 4 In the embodiment depicted in , each classification index generated by the hypothesis network 458 can have a corresponding validation network. For example, if the hypothesis network 458 predicts an H4 classification index for the sample 104, the pixel data and / or image data set can be processed by a network within the hemolytic index validation network 352A that is specifically trained on H4. The index validation network 352 can then analyze the pixel data and / or image data set to validate or nullify the predicted H4 classification index.

[0085] Figure 5 DSSN 462 is shown in the following figure. DSSN 462 may be encoded using any suitable scientific computing framework, program, or toolkit, such as, for example, Caffe available from Berkley Vision and Learning Center (BVLC), Theano Python library developed by the Montreal Institute for Learning Algorithms, TensorFlow, Torch, and the like. Architecture 500 may include, for example, Figure 5 The following operational layers are arranged as shown in the diagram: two convolution (CONV) layers 501 and 547; eleven dense block layers DB4 503, DB5 507, DB7 511, DB10 515, DB12 519, DB15 523, DB12 527, DB10 531, DB4 535, DB5539, and DB4 543; five downward transition layers TD 505, TD 509, TD 513, TD 517, and TD 521; five upward transition layers TU 525, TU 529, TU 533, TU 537, and TU 541, and a fully connected layer 545, wherein a classification index 470 is output. Other numbers, types, and / or arrangements of layers may be used.

[0086] In some embodiments, each dense block layer DB4503, 535 and 543 may include four dense layers. Each of the dense block layers DB5507 and 539 may include five dense layers. The dense block layer DB7511 may include seven dense layers. Each of the dense block layers DB10515 and 531 may include ten dense layers. Each of the dense block layers DB12519 and 527 may include twelve dense layers, and the dense block layer DB15523 may include fifteen dense layers. In some embodiments, each dense layer may include a batch normalization operation, a ReLU layer, and a 3×3 convolutional layer with a dropout p=0.2, wherein the first dense layer receives an input and outputs a plurality of pixel label maps, which are connected to the input. Then, the second dense layer receives the output of the connection as its input and outputs a plurality of pixel label maps, which are again connected to the previous pixel label map. This may be repeated for each dense layer in the dense block layer. Other numbers and / or types of dense layers may be used.

[0087] In some embodiments, each downward transition layer TD 505, TD 509, TD 513, TD 517, and TD 521 may include a batch normalization operation, followed by a ReLu layer, followed by a 3x3 convolution layer with dropout p=0.2, followed by a 2x2 max pooling layer. Note that the dropout probability may vary from 0 to 1, depending on the experimental test run that results in the best results. In some embodiments, the number of pixel label maps at the output of layer TD 505 may be 112, at the output of TD 509 may be 192, at the output of TD 513 may be 304, at the output of TD 517 may be 464, and at the output of TD 521 may be 656. In other embodiments, other types of downward transition layers may be used and / or other numbers of pixel label maps may be output. In some embodiments, each upward transition layer TU 525, TU 529, TU 533, TU 537, and TU 541 may include a 3x3 transposed convolutional layer with stride 2. Other upward transition layer parameters may be used.

[0088] In some embodiments, the number of pixel label maps at the output of DB15523 may be 896, at the output of DB12527 may be 1088, at the output of DB10531 may be 816, at the output of DB4535 may be 578, at the output of DB5539 may be 384, and at the output of DB4543 may be 256. In other embodiments, other numbers of pixel label maps may be output.

[0089] Reference now Figure 6, which shows a flow chart depicting a method 600 for characterizing a sample (e.g., sample 104). Method 600 includes capturing one or more images of a sample at 602, the one or more images including a serum or plasma portion of the sample (e.g., serum or plasma portion 204SP), capturing to generate pixel data. Method 600 includes, at 604, processing the pixel data of the one or more images of the sample using a first network (e.g., SCNN 460) executed on a computer (e.g., computer 124 and / or 172) to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia. The method includes validating the predicted classification using one or more validation networks (e.g., validation network 342) at 606. A similar method may be performed to validate a predicted classification index.

[0090] As used herein, a convolutional layer is a processing step that may apply a filter (also called a kernel) to input image data (e.g., pixel intensity values) to output an activation map that may indicate that certain specific types of colors or features are detected at certain spatial locations in the input image data (e.g., from a simple curve after applying the first convolutional layer to slightly more complex features after applying several convolutional layers).

[0091] A max pooling layer is a processing step that applies a filter to generate an output activation map with the maximum pixel value that appears in one or more activation maps received from the convolutional layer.

[0092] A ReLU (Rectified Linear Unit) layer is a processing step that can apply a non-linear function to all values ​​in a received activation map, resulting in, for example, all negative activation values ​​being assigned a value of zero.

[0093] A fully connected layer is a processing step that aggregates previous activation maps (each of which may indicate the detection of a lower level feature) to indicate the detection of a higher level feature.

[0094] The softmax layer is typically the final processing step, which outputs a probability distribution that highlights or identifies the most likely features of one or more images from the category of image features. Another challenge in determining the HIL classification index is to make the SCNN 460 output a homogeneous structure. That is, the challenge is to avoid classifying the serum or plasma portion as a combination of different HIL categories. This may occur because the loss function in the semantic segmentation network may evaluate each pixel individually. Although the DSSN 462 uses contextual information in response to a large receptive field, there may otherwise be no restriction on outputting homogeneous categories for each pixel of the same image of the sample container.

[0095] Although the present disclosure is described herein with reference to specific embodiments, the present disclosure is not intended to be limited to the details described. Instead, various modifications may be made in the details within the scope of equivalents of the present disclosure without departing from the present disclosure.

Claims

1. A method for characterizing a sample, include: capturing one or more images of the sample, the one or more images comprising a serum or plasma portion of the sample, the capturing generating pixel data; processing pixel data of the one or more images of the sample using a first network executed on a computer to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and validating the predicted classification using one or more validation networks executed on a computer, the one or more validation networks having been trained based on the predicted classification; as well as Responsive to the validated classification of hemolysis, icterus, and lipemia, a notification is output to indicate that the sample is not of appropriate quality for analysis.

2. The method according to claim 1, further comprising: include: In response to the first network predicting a hemolysis classification of the sample, validating the predicted classification by a validation network trained on hemolysis; In response to the first network predicting a jaundice classification of the sample, verifying the predicted classification by a validation network trained on jaundice; and In response to the first network predicting a lipemia classification of the sample, the predicted classification is validated by a validation network trained on lipemia.

3. The method according to claim 1, in: Processing the pixel data includes predicting a classification index of at least one of hemolysis, icterus, and lipemia; and Validating the predicted classification includes validating the predicted classification index using one or more index validation networks trained on the predicted classification index.

4. The method according to claim 1, in: Processing pixel data includes predicting a hemolysis classification index; and Validating the predicted classification includes validating the predicted hemolytic classification index using an index validation network trained on the predicted hemolytic classification index.

5. The method according to claim 1, in: Processing pixel data including predicting a jaundice classification index; and Validating the predicted classification includes validating the predicted jaundice classification index using an index validation network trained on the predicted jaundice classification index.

6. The method according to claim 1, in: Processing pixel data includes predicting a lipemia classification index; and Validating the predicted classification includes validating the predicted lipemia classification index using an index validation network trained on the predicted lipemia classification index.

7. The method according to claim 1, in, The classification also includes normal classification.

8. The method of claim 7, further comprising validating the normal classification using a validation network trained on the normal classification.

9. The method according to claim 1, in, Processing the pixel data of the one or more images of the sample using a first network includes processing the pixel data of the one or more images of the sample using a segmentation network.

10. The method according to claim 1, in, Processing the pixel data of the one or more images of the sample using a first network includes processing the pixel data of the one or more images of the sample using a classification network.

11. The method according to claim 1, in, Processing pixel data includes: Identifying the serum or plasma fraction in the sample using a deep semantic segmentation network; and Based at least in part on the color represented by the pixel data of the serum or plasma portion, predicting a classification index for the serum or plasma portion of the sample using the deep semantic segmentation network.

12. The method according to claim 1, in, The first network includes an architecture having at least eleven dense block layers.

13. The method of claim 1, further comprising generating a confidence level in response to validating the predicted classification using the one or more validation networks.

14. The method of claim 1, further comprising generating a confidence level in response to validating the predicted classification using one or more validation networks, and generating a signal in response to the confidence level being below a predetermined level.

15. The method according to claim 1, in, The one or more verification networks include one or more convolutional neural networks.

16. A quality inspection module, include: a plurality of image capture devices operable to capture one or more images from one or more viewpoints of a sample container containing a serum or plasma portion of the sample; and a computer coupled to the plurality of image capture devices, the computer being configured and operable to: capturing one or more images of the sample, the one or more images comprising a serum or plasma portion of the sample, the capturing generating pixel data; processing pixel data of the one or more images of the sample using a first network executing on the computer to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and validating the predicted classification using one or more validation networks executed on the computer, the one or more validation networks having been trained based on the predicted classification; as well as Responsive to the validated classification of hemolysis, icterus, and lipemia, a notification is output to indicate that the sample is not of appropriate quality for analysis.

17. The quality inspection module according to claim 16, in, The computer is also configured and operable to process the pixel data to predict a normal classification of the serum or plasma fraction.

18. The quality inspection module according to claim 16, in, The computer is configured and operable to: Processing pixel data to predict a classification index; and The predicted classification index is validated using one or more validation networks trained on the predicted classification index.

19. The quality inspection module according to claim 16, in, The computer is configured and operable to: In response to the first network predicting a hemolysis classification of the sample, validating the predicted classification by a validation network trained on hemolysis; In response to the first network predicting a jaundice classification of the sample, verifying the predicted classification by a validation network trained on jaundice; and In response to the first network predicting a lipemia classification of the sample, the predicted classification is validated by a validation network trained on lipemia.

20. The quality inspection module according to claim 16, in, The one or more verification networks include one or more convolutional neural networks.

21. A sample testing device, include: track; a carrier movable on the track and configured to contain a sample container containing a serum or plasma portion of a sample therein; a plurality of image capture devices arranged about the track and operable to capture one or more images from one or more viewpoints of the sample container and the serum or plasma portion of the sample; and a computer, coupled to the plurality of image capture devices, the computer being configured and operable to: capturing one or more images of the sample, the one or more images comprising a serum or plasma portion of the sample, the capturing generating pixel data; processing pixel data of the one or more images of the sample using a first network executing on the computer to predict a classification of the serum or plasma portion, wherein the classification includes hemolysis, icterus, and lipemia; and validating the predicted classification using one or more validation networks executed on the computer, the one or more validation networks having been trained based on the predicted classification; as well as Responsive to the validated classification of hemolysis, icterus, and lipemia, a notification is output to indicate that the sample is not of appropriate quality for analysis.

Images