Method and device for determining viewpoint for inspecting sample in sample container

Abstract

In some embodiments, there is provided a method of determining a viewpoint for optically inspecting a sample within a sample container, the method comprising (a) capturing image data of the sample container comprising the sample with a sensor, where a portion of the sample container comprises a tag having a first end and a second end; (b) rotating at least one of the sample container and the sensor about a central axis of the sample container such that the sensor captures image data comprising at least the first end and the second end of the tag; (c) generating an expanded image of the sample container using the captured image data; (d) processing the expanded image to characterize the label and generate label characterization information; and (e) employing the label characterization information to identify a viewpoint through which the sample within the sample container is optically inspected. Numerous other aspects are provided.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 365,189, filed on May 23, 2022, entitled “METHODS AND APPARATUS FOR DETERMINING AVIEWPOINT FOR INSPECTING A SAMPLEWITHIN ASAMPLE CONTAINER,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to methods and apparatus for testing samples, and more particularly, to methods and apparatus for sample analysis and viewpoint determination. Background Art

[0003] Automated testing systems can be used to perform clinical chemistry or assay tests using one or more reagents to identify analytes or other components in samples such as urine, serum, plasma, interstitial fluid, cerebrospinal fluid, etc. For reasons of convenience and safety, these samples can be contained in sample containers (e.g., blood collection tubes). The assay or test reactions generate various changes that can be read and / or manipulated to determine the concentration of the analyte or other component present in the sample.

[0004] Improvements in automated testing technology have been accompanied by corresponding advances in pre-analytical sample preparation and processing operations, such as sorting, batch preparation, centrifugation of sample containers to separate sample components, removal of caps to facilitate sample access, etc., by automated pre-analytical sample preparation systems, which may be part of a laboratory automation system (LAS). The LAS may automatically transport samples in sample containers to one or more pre-analytical sample processing stations and analysis stations comprising clinical chemistry analyzers and / or laboratory test instruments (hereinafter collectively referred to as "analyzers").

[0005] These LAS can process multiple different samples at a time, which can be contained in barcode-labeled or otherwise labeled (hereinafter referred to as "labeled") sample containers. The label can contain a registration number, which can be associated with demographic information entered into the hospital's laboratory information system (LIS) along with the test order and / or other information. The operator can place the labeled sample container on the LAS system, which can automatically route the sample container for one or more pre-analytical operations, such as centrifugation, decapping, and aliquot preparation, and all of this is performed before the sample is actually subjected to clinical analysis or assayed by one or more analyzers that can be part of the LAS.

[0006] Sample quality checking is an important pre-analytical task used to ensure the validity of the test to be performed on the sample in the sample container. For example, the presence of interferents (e.g., hemolysis, icterus, and / or lipemia) in the sample, which may be caused by the patient's condition or sample pretreatment, may adversely affect the test results of the analyte or component measurement obtained from one or more analyzers. The presence of hemolysis (H) in the sample, which may be unrelated to the patient's disease state, may result in a different interpretation of the patient's disease condition. Similarly, the presence of icterus (I) and / or lipemia (L) in the sample may also result in a different interpretation of the patient's disease condition.

[0007] Image analysis is commonly used in laboratory automation systems to determine sample quality. However, optical perception can be susceptible to various properties of the sample container, such as container material, HIL interference, label condition, and / or sample container orientation. For example, labels on sample containers may scatter light, and cylindrical sample containers may behave as lenses whose optical properties depend heavily on sample container orientation. Therefore, it is beneficial to find the best orientation for a particular sample quality inspection to be performed; and methods and apparatus are needed for determining the best viewpoint for inspecting a sample within a sample container. Summary of the invention

[0008] In some embodiments, a method for determining a viewpoint for optically inspecting a sample within a sample container includes: (a) using a sensor to capture image data of a sample container including the sample, wherein a portion of the sample container includes a label having a first end and a second end; (b) rotating at least one of the sample container and the sensor around a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; (c) using the captured image data to generate an expanded image of the sample container; (d) processing the expanded image to characterize the label and generate label characterization information; and (e) using the label characterization information to identify a viewpoint through which to optically inspect the sample within the sample container.

[0009] In some embodiments, a device suitable for determining a viewpoint for optically inspecting a sample within a sample container is provided. The device includes a sensor configured to capture image data of a sample container including the sample, wherein a portion of the sample container includes a label having a first end and a second end. The device also includes a rotation mechanism configured to rotate at least one of the sample container and the sensor around a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label. The device also includes a computer coupled to the sensor and the rotation mechanism, the computer including computer program code, which, when executed by the computer, causes the computer to (a) direct the rotation mechanism to rotate at least one of the sample container and the sensor around the central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; (b) use the captured image data to generate an expanded image of the sample container; (c) process the expanded image to characterize the label and generate label characterization information; and (d) use the label characterization information to identify a viewpoint through which to optically inspect the sample within the sample container.

[0010] In some embodiments, a diagnostic analysis system includes (a) a tracker; (b) a carrier movable on a track and configured to contain a sample container including a sample, wherein a portion of the sample container includes a label having a first end and a second end; (c) a sensor configured to capture image data of the sample container; (d) a rotation mechanism configured to rotate at least one of the sample container and the sensor around a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; and (e) a computer coupled to the sensor and the rotation mechanism. The computer includes computer program code that, when executed by the computer, causes the computer to (i) direct the rotation mechanism to rotate at least one of the sample container and the sensor around a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; (ii) use the captured image data to generate an expanded image of the label; (iii) process the expanded image to characterize the label and generate label characterization information; and (iv) use the label characterization information to identify a viewpoint through which to optically inspect the sample in the sample container.

[0011] By showing a number of example embodiments and implementations, other features, aspects and advantages of embodiments according to the present disclosure will become more apparent from the following detailed description, the appended claims and the accompanying drawings. Various embodiments according to the present disclosure can also have other and different applications, and several details thereof can be modified in various aspects, all without departing from the spirit and scope of the claims. Therefore, the drawings and descriptions are to be considered illustrative in nature, rather than restrictive. The drawings are not necessarily drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described below are for illustrative purposes and are not necessarily drawn to scale. Therefore, the drawings and description are to be regarded as illustrative rather than restrictive in nature. The drawings are not intended to limit the scope of the present invention in any way.

[0013] Figure 1 An automated diagnostic analysis system capable of automatically processing a plurality of sample containers containing samples according to embodiments provided herein is shown.

[0014] Figure 2 Shown is a side view of a sample container according to embodiments provided herein that can include a separated sample having a serum or plasma portion that can contain interfering substances.

[0015] Figure 3 Shows Figure 2 A side view of a sample container held in a vertical orientation in a holder that can be used in accordance with embodiments provided herein Figure 1 Transported within an automated diagnostic analysis system.

[0016] Figure 4A and 4B An example of a quality inspection module configured to perform the methods as shown and described herein is shown.

[0017] Figure 5 A functional quality check module architecture according to embodiments provided herein is shown, which is configured to perform characterization of a sample carrier and / or a sample.

[0018] Figure 6 A method for determining a viewpoint for optical inspection of a sample within a sample container according to embodiments provided herein is shown.

[0019] Fig. 7A A motor system for rotating a sample container according to embodiments provided herein is shown.

[0020] Figures 7B-7F Capturing multiple images from multiple viewpoints around a sample container according to embodiments provided herein is shown.

[0021] Figure 7G According to the embodiment provided in this article, Figures 7B-7F Generate expanded image labels for multiple captured images.

[0022] Figure 8 The use of a segmentation network to generate a segmentation mask according to embodiments provided herein is shown.

[0023] Fig. 9 The embodiment provided in this article is shown based on Figure 8 The segmentation mask is used to identify the appropriate viewpoint.

[0024] Fig. 10A An embodiment of a segmentation network according to embodiments provided herein is shown, where an input image is processed by a feature extractor and a segmentor.

[0025] Fig. 10B An embodiment of a classification network according to embodiments provided herein is shown, where an input image is processed by a feature extractor and a classification part.

[0026] Fig. 10C An embodiment of a segmentation and classification network according to embodiments provided herein is shown, where an input image is processed by a feature extractor and both a segmenter and a classification part. DETAILED DESCRIPTION

[0027] Many variations can occur in a sample processing system that may change the orientation of a sample container, such as misplacement by a human technician upon initial entry, or rotation caused by a machine when moving a sample container from one location to another (e.g., such as when moving between a pre-analytical processing station or an analyzer station). In addition, defects in any labels placed on the sample container, such as skew, bunching, and tears, may obscure a large portion of the contents of the sample container from view. Despite all of these complicating factors, the sample processing system must determine the best way to orient the sample container in front of the optical analysis system to generate an accurate analysis of the contents of the sample container. Correct placement of the sample container can lead to a more accurate understanding of the composition of the sample and can improve the performance of downstream tasks (such as chemical analysis), leading to a better understanding of the patient's health.

[0028] Embodiments provided herein include methods and apparatus for more accurately determining a viewpoint for optically inspecting a sample within a sample container, regardless of variations in rotation of the sample container or the presence of or defects in one or more labels on the sample container.

[0029] In one or more embodiments, multiple images of the sample container are obtained by the imaging sensor as the sample container rotates in front of the imaging sensor and / or the imaging sensor rotates around the sample container (e.g., in some embodiments, images are obtained from 360 degrees around the sample container). The images can be snapshots of the sample container or frames extracted from a video. Thereafter, the images are stitched together to produce an expanded image that depicts the sample container and any labels thereon, and if a label is present, depicts both ends of the label and any gap therebetween. The expanded image can then be processed to characterize the label, such as by passing the expanded image through a segmentation network to generate a segmentation mask that represents the expanded image pixels as label pixels or non-label pixels. The segmentation mask can be used to identify the gap between the ends of the label, and the center (e.g., midpoint) of the gap between the ends of the label can be determined and used as a viewpoint for inspecting the sample within the sample container. It can be determined how far the sample container (or imaging sensor) should be rotated to align the identified viewpoint (e.g., the center of the label gap) with the center of the field of view of the imaging sensor (e.g., for optical analysis of any sample within the sample container). Thereafter, the sample in the sample container may be inspected using the identified viewpoint, and the sample may be characterized.

[0030] Refer to the following Figure 1-10C These and other embodiments are described.

[0031] Figure 1 It is shown that the automatic processing of the sample 212 (see Figure 2 ) of a plurality of sample containers 102. The sample containers 102 may be provided in one or more racks 104 at a loading area 105 before being transported to and analyzed by one or more analyzers (e.g., a first analyzer 106, a second analyzer 108, and / or a third analyzer 110) arranged around the automated diagnostic analysis system 100. More or fewer analyzers may be used in the system 100. The analyzer may be any combination of any number of clinical chemistry analyzers, assay instruments, and the like. For example, the term "analyzer" as used herein refers to a device for analyzing the presence, quantity, or functional activity of a target entity (analyte) of a chemical or assay, such as DNA or RNA. Analytes commonly tested in clinical chemistry analyzers include enzymes, substrates, electrolytes, specific proteins, drugs of abuse, and therapeutic drugs. The sample container 102 may be any suitable transparent or translucent container, such as a blood collection tube, a test tube, a sample cup, a test tube, or other clear or opaque glass or plastic container that is capable of containing and allowing imaging of the sample 212 contained therein. The sample containers 102 may vary in size and may have different cap colors and / or cap types.

[0032] Sample 212 (see Figure 2) can be provided to the automated diagnostic analysis system 100 in a sample container 102, which can be covered with a lid 214. The lid 214 can be of different types and / or colors (e.g., red, royal blue, light blue, green, gray, brown, yellow, or a combination of colors), which can have significance in terms of what test each sample container 102 is used for, the type of additives included therein, whether the container includes a gel separator, etc. Other colors can be used. In one or more embodiments, the lid type can be determined by the characterization methods described herein. For example, the lid type can be used to determine whether the sample 212 is provided under vacuum and / or the type of additives therein. Figure 2 , the sample container 102 is shown as a tube 215. Other sample container shapes and / or types may be used.

[0033] Each sample container 102 may be provided with one or more labels 218, which may include identification information 218i (i.e., indicia), such as a barcode, alphabetic characters, numeric characters, or a combination thereof. Example identification information 218i may include or be associated with (e.g., by, for example, Figure 1 The laboratory information system (LIS) 112 database shown may include patient information (e.g., name, date of birth, address, and / or other personal information), tests to be performed, time and date the sample was obtained, medical institution information, tracking and routing information, etc. Other information may also be included. The identification information 218i may be machine readable at various locations regarding the automated diagnostic analysis system 100. For example, the machine-readable information may be darker (e.g., black) than the label material (e.g., white paper) so that it can be easily imaged. The identification information 218i may indicate or otherwise be associated with the identification of the patient and the test to be performed on the sample 212 via the LIS 112 or other test ordering system. Such identification information 218i may be provided on a label 218, which may be adhered to or otherwise provided on the outer surface of the tube 215. As shown in FIG. Figure 2 As shown, the label 218 may not extend all the way around the sample container 102, or may not extend all the way along the length of the sample container 102, such that from the particular side-front viewpoint shown, some or most of the sample 212 (e.g., the serum or plasma portion 212SP) is visible (the portion shown by the dashed line) and is not obscured by the label 218.

[0034] The sample 212 may include any fluid to be tested and / or analyzed (e.g., serum, plasma, urine, interstitial fluid, cerebrospinal fluid, etc.). In some embodiments, the sample 212 may include a serum or plasma portion 212SP and a sedimented blood portion 212SB contained in a tube 215. Air 216 may be provided above the serum and plasma portions 212SP, and the boundary therebetween is defined as a liquid-air interface (LA). The boundary between the serum or plasma portion 212SP and the sedimented blood portion 212SB is defined as a serum-blood interface (SB). The interface between the air 216 and the cap 214 is defined as a tube-cap interface (TC). The height (HT) of the tube is defined as the height from the bottom of the tube 215 to the bottom of the cap 214, and can be used to determine the tube size (tube height). The height of the serum or plasma portion 212SP is HSP, and is defined as the height from the top of the serum or plasma portion 212SP at LA to the top of the sedimented blood portion 212SB at SB. The height of the settled blood portion 212SB is HSB, and is defined as the height from the bottom of the settled blood portion 212SB to the top of the settled blood portion 212SB at SB. HTOT is the total height of the sample 212, and is equal to HSP plus HSB.

[0035] In more detail, the automated diagnostic analysis system 100 may include a base 116 ( Figure 1 ) (e.g., a frame, floor, or other structure), the track 118 can be mounted on the base 116. The track 118 can be a collection of rail tracks (e.g., a single track or multiple tracks), a conveyor belt, a conveyor chain, a movable platform, or any other suitable type of conveying mechanism. The track 118 can be circular or any other suitable shape, and in some embodiments can be a closed track (e.g., a circular track). In operation, the track 118 can transport individual sample containers in the sample container 102 in the carrier 122 to various locations spaced around the track 118.

[0036] The carrier 122 can be a passive, non-motorized tray that can be configured to carry a single sample container 102 on the track 118, or alternatively, an automated carrier that includes an onboard drive motor, such as a linear motor that is programmed to move around the track 118 and stop at a preprogrammed position. Other configurations of the carrier 122 can be used. The carriers 122 can each include a holder 122H configured to hold the sample container 102 in a defined upright position and orientation (see Figure 3 ). Retainer 122H ( Figure 3) may include a plurality of fingers or leaf springs that secure the sample container 102 to the carrier 122, but some may be removable or flexible to accommodate sample containers 102 of different sizes (widths). In some embodiments, the carrier 122 may leave the loading area 105 ( Figure 1 ). The loading area 105 can be used for dual functions, namely, it also allows the sample container 102 to be reloaded from the carrier 122 to the loading area 105 after the pre-screening and / or analysis is completed.

[0037] The robot 124 may be provided in the loading area 105 and may be configured to grab sample containers 102 from one or more racks 104 and load the sample containers 102 onto a carrier 122, such as onto an input channel of a track 118. The robot 124 may also be configured to reload sample containers 102 from a carrier 122 to one or more racks 104. The robot 124 may include one or more (e.g., at least two) robotic arms or assemblies capable of X (lateral) and Z (vertical off the page, as shown), Y and Z, X, Y and Z, or r (radial) and theta (rotational) motion. The robot 124 may be a gantry robot, an articulated robot, an R-theta robot, or other suitable robot, wherein the robot 124 may be equipped with robotic gripper fingers oriented, sized, and configured to pick up and place sample containers 102.

[0038] When loaded onto the track 118, the sample containers 102 carried by the carrier 122 can proceed to the first pre-processing station 125. For example, the first pre-processing station 125 can be an automatic centrifuge configured to perform fractionation on each sample 212. The carrier 122 carrying the sample containers 102 can be transferred to the first pre-processing station 125 via an inflow channel or a suitable robot. After being centrifuged, the sample containers 102 can exit on an outflow channel, or otherwise be removed by the robot, and continue along the track 118. In the depicted embodiment, the sample containers 102 in the carrier 122 can then be transported to a quality inspection module 130, which is configured to perform a pre-screening, as will be further described herein.

[0039] The quality inspection module 130 is configured to pre-screen and perform one or more characterization methods described herein. For example, the quality inspection module 130 can automatically determine the presence of H, I and / or L contained in the sample 212 and the optional range or degree, or whether the sample is normal (N). If it is found that H, I and / or L are contained in an effective low amount, it is considered to be normal (N), then the sample 212 can continue on the track 118, and then it can be analyzed by one or more analyzers (e.g., the first, second and / or third analyzers 106, 108 and / or 110). Other pre-treatment operations can be performed on the sample 212 and / or the sample container 102. After being analyzed by one or more analyzers (e.g., the first, second and / or third analyzers 106, 108 and / or 110), the sample container 102 can return to the loading area 105 to be reloaded to one or more racks 104 or unloaded in other ways.

[0040] In some embodiments, in addition to detecting HILN, segmentation of the sample container 102 and sample 212 can also be performed (e.g., at the quality inspection module 130). Based on the segmented data, post-processing can be used for quantification of the sample 212 (e.g., determining HSP, HSB, HTOT, and / or possibly determining the location of SB, LA and / or TC). In some embodiments, characterization of the physical properties of the sample container 102 (e.g., dimensions - height and width (or diameter)) can be performed at the quality inspection module 130. Such characterization can include determining HT and W, and possibly TC and / or Wi. Based on such characterization, the dimensions of the sample container 102 can be extracted. In addition, in some embodiments, the quality inspection module 130 can also determine the cap type, which can be used as a safety check and can capture whether the wrong tube type has been used for one or more predetermined tests.

[0041] In some embodiments, a remote station 132 may be provided on the automated diagnostic analysis system 100 that is not directly linked to the track 118. For example, an independent robot 133 (shown in dashed lines) may transport sample containers 102 containing samples 212 to the remote station 132 and return them after testing / preprocessing. Alternatively, the sample containers 102 may be removed and returned manually. The remote station 132 may be used to test certain components, such as hemolysis levels, or may be used for further processing, such as reducing blood lipid levels by one or more additions and / or by additional processing, or removing clots, bubbles, or foam, which are identified, for example, in the characterization at the quality inspection module 130. Other pre-screening using the HILN detection method may optionally be completed at the remote station 132.

[0042] Additional station(s) may be provided at one or more locations on or along the track 118. The additional station(s) may include a decapping station, an aliquoting station, one or more additional quality inspection modules 130, and the like.

[0043] The automated diagnostic analysis system 100 may include a plurality of sensors 134 at one or more locations around the track 118. The sensors 134 may be used to detect the position of the sample container 102 on the track 118 by reading information such as identification information 218i provided on each carrier 122 (not shown). Any suitable means for tracking position may be used, such as proximity sensors. All sensors 134 may be interfaced with a computer 143 so that the position of each sample container 102 along the track 118 may be known at all times.

[0044] The pre-processing station 125 and analyzers 106 , 108 , and 110 may be equipped with robotic mechanisms and / or inflow channels configured to remove carriers 122 from the track 118 , and be equipped with robotic mechanisms and / or outflow channels configured to re-enter the carriers 122 into the track 118 .

[0045] The automated diagnostic analysis system 100 can be controlled by a computer 143, which can be a microprocessor-based central processing unit CPU or other suitable controller with suitable memory and suitable regulating electronic devices and drivers for operating various system components. The computer 143 can be housed as part of the base 116 of the automated diagnostic analysis system 100 or separate therefrom. The computer 143 can be operated to control the movement of the carrier 122 to and from the loading area 105, the movement around the track 118, the movement to and from the first pre-processing station 125 and the operation of the first pre-processing station 125 (e.g., a centrifuge), the movement to and from the quality inspection module 130 and the operation of the quality inspection module 130, and the movement to and from each analyzer 106, 108, 110. In some embodiments, the operation of each analyzer 106, 108, 110 for performing various types of tests (e.g., assays or clinical chemistry) may be performed by a local workstation computer at each analyzer 106, 108, 110 that is in digital communication with computer 143, such as via network 145 ( Figure 1 ), such as, for example, a local area network (LAN) or a wireless local area network (WAN) or other suitable communication network. Optionally, some or all of the operations of the above-mentioned analyzers 106, 108, 110 can be provided by a computer 143.

[0046] For all components except the quality check module 130, the computer 143 may control the automated diagnostic analysis system 100 according to software, firmware, and / or hardware commands or circuits, such as those sold by Siemens Healthcare Diagnostics, Inc. of Tarrytown, New York. Those used on clinical chemistry analyzers. Other suitable systems for controlling the automated diagnostic analysis system 100 may be used. Control of the quality check module 130 may also be provided by the computer 143 (or another suitable computer) according to the embodiments described in detail herein.

[0047] Computer 143 can be used for image processing and execution of the characterization method described herein. For example, the computer can include a CPU or GPU, sufficient processing power and RAM, and suitable storage. In one example, computer 143 can be a PC equipped with a multiprocessor with one or more GPUs, 8GB RAM or more RAM, and terabytes or more storage. In another example, computer 143 can be a PC equipped with a GPU, or alternatively a PC equipped with a CPU operating in parallel mode. Mathematical Kernel Library (MKL), 8GB RAM or more RAM, and suitable storage can also be used.

[0048] Embodiments of the present disclosure may be implemented using a computer interface module (CIM) 147 that allows a user to easily and quickly access various control and status display screens. These control and status display screens may display and implement control of some or all aspects of a plurality of interrelated automated devices for preparation, prescreening, and analysis of a sample 212. The CIM 147 may be used to provide information about the operational status of a plurality of interrelated automated devices, as well as information describing the location of any sample 212 and the status of the prescreening and (multiple) tests that are to be performed on the sample 212 or are being performed on the sample 212. The CIM 147 is therefore suitable for facilitating interaction between an operator and the automated diagnostic analysis system 100. The CIM 147 may include, for example, a display screen that is operable to display a menu including icons, scroll bars, boxes, and / or buttons through which an operator may interact with the automated diagnostic analysis system 100. The menu may include a number of functional elements that are programmed to display and / or operate the functional aspects of the automated diagnostic analysis system 100.

[0049] Figure 4A and 4BAn embodiment of a quality check module 130 is shown, which is configured to perform the method as shown and described herein. The quality check module 130 can be configured with programming instructions that, when executed by the computer 143, perform pre-screening to ensure the validity of the test to be performed on the sample 212 in the sample container 102. For example, before being analyzed by one or more of the analyzers 106, 108, 110, etc., the quality check module 130 can be pre-screened for the presence and optional extent of interfering substances (e.g., H, I and / or L) in the container material, label status, sample container orientation, and sample 212 (e.g., its serum or plasma portion 212SP). Pre-screening in this manner allows the sample 212 to be additionally processed, additionally quantified or characterized, and / or discarded and / or re-drawn without wasting valuable analyzer resources or potentially causing the presence of interfering substances to affect the accuracy of the test results. In addition, in some aspects, pre-screening can improve the characterization of future samples 212.

[0050] In addition to the interference detection methods described herein, other detection methods may be performed on the sample 212 contained in the sample container 102 at the quality inspection module 130. For example, a method may be performed at the quality inspection module 130 to provide segmentation data. The segmentation data may be used in a post-imaging step to quantify the sample 212, for example, to determine certain physical dimensional characteristics of the sample 212, such as the location of LA and / or SB, and / or the determination of HSP, HSB, HT, Wi and / or HTOT. Quantification may also involve estimating, for example, the volume of the serum or plasma portion (VSP) and / or the volume of the sedimented blood portion (VSB) based on the quantification of the internal width Wi. In addition, the quality inspection module 130 may be used to quantify the geometry of the sample container 102, i.e., to quantify certain physical dimensional characteristics of the sample container 102, such as the location of TC, HT and / or W or Wi of the sample container 102. Other quantifiable geometric features may also be determined.

[0051] The quality inspection module 130 may include a housing 402 that may at least partially surround or cover the track 118 to minimize external lighting effects. During the image capture sequence, the sample container 102 may be located inside the housing 402 in an imaging position. The housing 402 may include one or more doors 404 to allow the carrier 122 to enter and / or exit the housing 402. In some embodiments, the top plate may include an opening 406 ( Figure 4B ) to allow sample containers 102 to be loaded from above into carrier 122 by a robot including movable robotic fingers.

[0052] like Figure 4A and 4BAs shown, the quality inspection module 130 may include an image capture device, referred to as a sensor 408, which is configured to capture a lateral image of the sample container 102 and the sample 212 from a viewpoint (e.g., a lateral viewpoint labeled 1) at an imaging position 410. Although one sensor 408 is shown, two, three, four, or more sensors may be used. The viewpoint 1 may be arranged at any suitable location. In some embodiments, the sensor 408 may be configured to be relative to a central axis 412 ( Figure 4B ) rotation. For example, the sensor 408 can rotate on a track (not shown) driven by a motor controlled by the computer 143.

[0053] The light source 414 can backlight the sample container 102 (as shown) for imaging to complete segmentation and / or HILN characterization. In other cases, such as for characterizing the sample container 102, front illumination of the imaging location 410 can be used. In embodiments where the sensor 408 rotates, the light source 414 can be configured to rotate with the sensor 408 relative to the central axis 412 of the sample container 102.

[0054] In one or more embodiments, the sample carrier 122 may be rotated within the quality inspection module 130 during imaging of the sample container 102 and the sample 212. For example, a motor or other rotating mechanism (e.g., Fig. 7A The motor 703 of the sensor 408 can rotate the sample carrier 122. In some embodiments, the sensor 408 and the sample carrier 122 can rotate relative to each other. Rotation of the sensor 408 and / or the sample container 102 (via the sample carrier 122) allows the sample container 102 and the sample 212 to be imaged from multiple perspectives (e.g., the sample container 102 is rotated up to 360 degrees relative to the sensor 408, and the sample container 102 and the sample 212 are imaged up to 360 degrees).

[0055] By using the sensor 408, an image of the sample 212 in the sample container 102 can be taken while the sample container 102 resides in the carrier 122 at the imaging position 410. The fields of view of the multiple images obtained by the sensor 408 can overlap in a circumferential range. In some embodiments, portions of the images can be digitally added to obtain a complete image of the sample 212 for analysis. In particular, in the embodiments described below, the multiple images captured by the sensor 408 can be combined (e.g., stitched together) to form an expanded image of the sample container 102 (and any labels on the sample container 102), and the viewpoint for imaging the sample 212 can be determined. For example, in some embodiments, slices or "windows" having a width of a predetermined number of pixels and a full (or other) image height can be obtained for each image, and the slices can be sequentially connected together to generate a stitched image. Alternatively, in some embodiments, a certain percentage of the current image slice can overlap with the previous image slice. For example, the final value of the pixel in the overlapping area can be a linear combination of the pixel values ​​from the previous slice and the current slice.

[0056] The sensor 408 may be any suitable device configured to capture a well-defined digital image, such as a conventional digital camera capable of capturing pixelated images, a charge coupled device (CCD), a photodetector array, a CMOS sensor, etc. For example, the captured image size may be, for example, approximately 2560×694 pixels. In another embodiment, the sensor 408 may capture an image size of, for example, approximately 1280×387 pixels. Other image sizes and pixel densities may be used for the captured image.

[0057] Each image may be triggered and captured at the quality inspection module 130 in response to receiving a trigger signal provided in the communication line 416 from the computer 143. According to one or more embodiments, each captured image may be processed by the computer 143. In some embodiments, image data from the captured images may be captured and processed using high dynamic range (HDR) processing.

[0058] Reference now Figure 5-9 The operation of the quality check module 130 is described.

[0059] Figure 5 5 shows a functional quality check module architecture 500 according to an embodiment provided herein, which is configured to perform characterization of a sample carrier and / or a sample. In some embodiments, the functional quality check module architecture 500 can be in the quality check module 130 ( Figure 1 ) is implemented as computer programming instructions stored in, for example, a memory 501 of the computer 143. In general, the functional quality check module architecture 500 can be implemented across one or more computing devices and / or one or more memories.

[0060] refer to Figure 5 , the functional quality inspection module architecture 500 includes an image capture rotation module 502 that controls the rotation of the sample container 102 and / or the sensor 408 during imaging within the quality inspection module 130. For example, the image capture rotation module 502 may include programming instructions that direct one or more motors (e.g., Fig. 7A The motor 703 in the quality inspection module 130 rotates the sample container 102 and / or the sensor 408 within the quality inspection module 130. The image capture module 504 controls the imaging within the quality inspection module 130 (e.g., by programming instructions that instruct the sensor 408 when to take an image of the sample container 102). The image captured by the sensor 408 is provided to the expanded image generator 506, which includes programming instructions for combining the captured images to generate an expanded image of the sample container 102 (and / or any label on the sample container 102). Any suitable method for stitching the images together can be used, such as the imaging processing tools of Open-CV in Python (see also Matthew Brown and David G. Lowe, "Automated Panoramic Image Stitching using Invariant Features", International Journal of Computer Vision 74 (2007)). The expanded image can then be fed through a segmentation network 508, which creates a segmentation mask based on the expanded image (as further described below). The segmentation mask is similar to the unfolded image, but each pixel within the segmentation mask is identified as either a label pixel or a non-label pixel. The segmentation mask processing module 510 includes programming instructions for analyzing the segmentation mask and identifying gaps between the ends of any unfolded labels on the sample container. In particular, a viewpoint can be identified within the label gap (e.g., the midpoint of the gap), and the amount by which the sample container 102 (and / or sensor 408) should be rotated so that the sensor 408 images the sample container 102 through the identified viewpoint (e.g., so that the identified viewpoint is aligned with the center of the field of view of the sensor 408). The quality inspection program 514 can then perform the desired quality inspection and / or characterization of the sample container 102 and / or sample 212. Referring below to Figure 6-9 Additional details regarding the operation of the functional quality check module architecture 500 are described.

[0061] Figure 6 A method 600 for determining a viewpoint for optically inspecting a sample in a sample container according to an embodiment provided herein is shown. Figure 7A-9 A method 600 is described, wherein Fig. 7AA motor system for rotating a sample container is shown, Figures 7B-7F shows capturing multiple images from multiple viewpoints around the sample container, Figure 7G Shows from Figures 7B-7F The multiple captured images generate an expanded image, Figure 8 shows the generation of segmentation masks using a segmentation network, and Fig. 9 Shown based on Figure 8 The segmentation masks identify appropriate viewpoints, each according to embodiments provided herein.

[0062] refer to Figure 6 , method 600 includes (1) at 602, using a sensor to capture image data of a sample container including a sample, wherein a portion of the sample container includes a label having a first end and a second end; and (2) at 604, rotating at least one of the sample container and the sensor about a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label. For example, sample container 102 and / or sensor 408 may be rotated relative to each other while sensor 408 captures images of sample container 102 from a plurality of different viewpoints.

[0063] In some embodiments, the sample container 102 can be rotated while the sensor 408 remains stationary. Fig. 7A As shown, the rotating platform 702 (eg, the image capture rotating module 502 ( Figure 5 ) can be used to securely hold the sample container 102 and rotate it into position along the central axis 704 of the sample container. In this arrangement, the sensor 408 can remain stationary with the sample container 102 centered in the field of view of the sensor 408 as the sample container 102 rotates. In some embodiments, the sensor 408 and sample container 102 can be adjusted so that the entire sample container 102 is within the field of view of the sensor 408. External lighting, such as from a light source 414 ( Figure 4A ) to provide adequate illumination. Images captured during the rotation of the sample container 102 and / or the sensor 408 may be stored (e.g., in the memory 501 of the computer 143 ( Figure 5 )).

[0064] exist Figures 7B-7F4 shows five example images 705a-705e of the sample container 102 taken from five different viewpoints (e.g., five different rotation amounts). More or fewer images and / or viewpoints may be used. In some embodiments, the sample container 102 and / or the sensor 408 may be rotated a full 360 degrees relative to each other. Larger or smaller rotation angles may be used (e.g., sufficient rotation to capture the gap between the ends of the label on the sample container 102).

[0065] At 606, method 600 includes using the captured image data to generate an expanded image of the sample container including the label (e.g., Figure 7G Any suitable image compositing algorithm may be employed to aggregate the images into a single image representing the expanded label and / or sample container 102. In some embodiments, line scans of a small window of pixels of the sample container 102 extracted from each image (or from selected video frames) may be stitched together to form the expanded image 706. Figures 7B-7F 1 and 2. Example image windows 708a-708e are shown in FIG. By using multiple images, or over the course of an entire video, these smaller image windows can capture the entire geometry of the sample container 102. In some embodiments, approximately 2 to 360 image windows may be used, ranging in size from approximately 1 to 20 mm wide by approximately 1 to 60 mm high, or approximately 1 to 640 pixels wide by approximately 1 to 1920 pixels high. In some embodiments, each image window 708a-e may represent approximately 0.4% to approximately 1.6% of the pixels of each image. Other numbers, widths, and / or lengths of image windows may be used. The size of the image window employed may depend on many factors, such as the capture rate of the image capture device employed, the resolution at which the image capture device is capable of capturing images, and how the sample container is oriented relative to the image capture device.

[0066] The image windows 708a-e can be stitched and blended together to generate the expanded image 706. The width of each image window can be changed according to the geometry of the sample container 102, such as according to how much curvature appears in each image. In some embodiments, during image stitching, not all frames of the video (or image snapshot) can be used. For example, it is possible to selectively select the frames from which each image window is extracted so that a smaller set of images is used. The amount of image data input to the algorithm can be adjusted based on the relevant geometry. As described above, any suitable stitching algorithm can be used and implemented in computer program instructions (e.g., executed by the computer 143 as part of the expanded image generator 506).

[0067] At 608, method 600 includes processing the expanded image to characterize the tag (and generate tag characterization information). For example, the expanded image 706 may be analyzed to determine tag characterization information, such as which portions of the image are tags ( Figure 7G The information about the labels 710 and their positions in the image can be used to determine the optimal orientation for the target task (described below). The label characterization step extracts key features of the label 710 that distinguish the label 710 from the rest of the sample container 102 and the background of the image. In some embodiments, the expanded image 706 can be used to generate a segmentation mask that characterizes the label. For example, the expanded image 706 can be passed through a deep learning-based segmentation network (e.g., Figure 8 Such a network can be trained to recognize labels in an expanded image of a sample container and generate a segmentation mask that identifies the location of the label in the image. Figure 8 As shown, the expanded image 706 may be fed into the segmentation network 508 to generate a segmentation mask 802 .

[0068] The segmentation network 508 is trained to learn the physical characteristics of labels on sample containers within images acquired with the same settings as the training data (e.g., the quality inspection module 130). The segmentation network 508 can be similar to a standard neural network. For example, an existing data set (e.g., unfolded images) can be provided to train the network. These images include unfolded labels, which can be annotated to indicate relevant information, such as the location of the barcode, the edges of the label, other label markings, tearing or bunching of the label, etc.

[0069] The segmentation network 508 can process an expanded image containing one or more labels and output an image of similar (or identical) height and width, wherein each pixel in the output image is identified as a label pixel or a non-label pixel in the form of a segmentation mask (such as the segmentation mask 802). For example, the brighter areas in the segmentation mask 802 identify label pixels 804, while the darker areas identify non-label pixels 806. During training, the segmentation network 508 can be taught to produce accurate segmentation masks by adjusting the internal connection weights so that the expanded images used during training output the expected segmentation mask. In some embodiments, thousands of training images can be used to train the segmentation network 508 to produce accurate segmentation masks that identify label and non-label areas of the expanded image. For example, the training images can be generated by imaging many sample containers, each sample container having different label attributes (e.g., different label gap size, flatness, bunching, orientation, etc.), and different sample container orientations. For each sample container, a training image set can be generated by imaging many different areas of the sample container at different rotation amounts, and then stitching the image sets together to create a large number of expanded images for each sample container.

[0070] In some embodiments, the segmentation network 508 may include a segmentation convolutional neural network (SCNN) that receives as input an expanded image of the sample container 102. In some embodiments, the SCNN may include more than 100 operation layers, including, for example, BatchNorm, ReLU activation, convolution (e.g., 2D), drop and deconvolution (e.g., 2D) layers to extract features such as edges, textures, and parts of any labels present on the sample container 102. Top layers, such as full convolutional layers, can be used to provide correlations between parts. A SoftMax layer can produce an output on a per-pixel (or per-patch - including n×n pixels) basis as to whether each pixel or patch includes a label. In some embodiments, the segmentation network 505 or a separate SCNN may include a front-end container segmentation network (CSN) to determine the sample container type and container boundaries. More specifically, the CSN can classify (or "segment") various regions of sample containers and samples, such as a serum or plasma fraction, a sedimented blood fraction, a gel separator (if used), an air region, one or more label regions, the type of sample container (indicating, for example, height and width / diameter), and / or the type and / or color of the sample container cap. The sample container holder or background may also be classified. There may be situations where the SCNN can perform all of these tasks without ultimately requiring an additional CSN. In some embodiments, the CSN may be included in the SCNN. In other embodiments, the CSN may be used as a separate model following the SCNN.

[0071] At 610, method 600 includes employing the label representation information (e.g., the segmentation mask) to identify a viewpoint through which to optically inspect a sample within a sample container. For example, once the segmentation network 508 determines the segmentation mask 802 (e.g., performing label segmentation on the expanded image 706), the output segmentation (segmentation mask 802) can be processed in the segmentation mask processing module 510 ( Figure 5 ) are analyzed to determine the best viewpoint for imaging the sample container 102 and the sample 212 stored therein. Figure 7G ) associates information about the geometry of the sample container 102 with rotation along the horizontal axis of the image 706, so the position of key features in the expanded image 706 can be used to determine the current orientation of the sample container 102. When acquiring images used to generate the expanded image 706, the initial position of the sample container 102 during imaging can be considered to be 0 degrees of rotation, and successive images can be mapped to the amount of rotation of the sample container 102 (or sensor 408) based on how fast the sample container 102 is rotated by the motor 703. For example, the computer 143 can monitor the rotation of the sample container 102 via the motor 703 to determine how far the sample container 102 has been rotated between each image used to form the expanded image 706. In this way, the horizontal pixels of the expanded image 706 can be associated with the rotation angle of the sample container 102, for example, as Figure 7G and Fig. 9 The rotation information may be stored in the memory 501 of the computer 143, for example.

[0072] refer to Fig. 9 In some embodiments, a horizontal window 902 (e.g., a segmentation slice) can be extracted from the segmentation mask 802 for analysis. The horizontal window 902 contains information about the start (at the first end 904) and end (at the second end 906) of the label 710, and the segmentation mask processing module 510 ( Figure 5 ) can associate the gap 908 between the first end 904 and the second end 906 with a particular amount of rotation of the sample container 102. The midpoint 910 between the first end 904 and the second end 906 of the label 710 can be determined based on the segmentation mask 802, along with the relative rotation required to rotate the sample container 102 so that the midpoint 910 (e.g., the center) of the gap 908 is aligned with the center 912 of the field of view of the sensor 408. In this manner, the optimal viewpoint for optically probing the sample 212 within the sample container 102 can be determined (e.g., the midpoint or center 910 of the gap 908), and the amount of rotation required to align the optimal viewpoint with the center of the field of view of the sensor 408 can be obtained.

[0073] In 612( Figure 6), method 600 includes rotating the sample container and / or the sensor to align the identified (eg, optimal) viewpoint with the center of the sensor's field of view. For example, viewpoint rotation module 512 ( Figure 5 ) may be used to rotate the sample container 102 so that a midpoint 910 of the label 710 (e.g., an identified and / or best viewpoint for characterizing the sample 212 and / or the sample container 102) and a center 912 of the field of view of the sensor 408 (at the center of the sample container 102) are aligned with each other before performing a quality inspection operation on the sample container 102 and / or the sample 212. Fig. 9 Alternatively or additionally, the sensor 408 may be rotated around the sample container 102 to align the identified (e.g., optimal) viewpoint with the center of the field of view of the sensor 408. In this manner, the sample container 102 may be properly and / or optimally oriented relative to the sensor 408.

[0074] After the identified viewpoint is aligned with the center of the field of view of the sensor 408, the sample container 102 and / or the sample 212 may be subjected to quality inspection and / or otherwise characterized. For example, the sample 212 may be optically analyzed for the presence and, optionally, the extent of interferents (e.g., H, I, and / or L) in the sample 212 (e.g., the serum or plasma portion 212SP thereof) using the identified viewpoint prior to analysis by one or more of the analyzers 106, 108, and / or 110. The sample container 102 may also be characterized and / or pre-screened for container material, label condition, or sample container orientation prior to analysis by one or more of the analyzers 106, 108, 110, etc.

[0075] Label characterization is not limited to segmentation of label 710. Other relevant characteristics may also be considered when determining the viewpoint, such as the size of the barcode, the number of labels on the sample container, and / or the condition of the barcode. Using the detected physical size of label 710 and prior knowledge of the geometry of the sample container, segmentation mask processing module 510 ( Figure 5 ) and computer 143 can infer the point at which label 710 is applied and how it wraps around the surface of sample container 102. Based on the endpoints of label 710 after wrapping, it can be determined whether there is an opening (e.g., a gap between the ends of the label), and if an opening does exist, the width of the opening can be determined. This information allows the best viewing point to be determined.

[0076] The number of labels on the sample container and their positions may also be important for sample characterization, as the label arrangement may obscure part or all of the analysis area. Labels may not be applied in exactly the same way every time, and may start and stop at different locations along the sample container, and may also be applied at different heights. This may cause labels to overlap each other and obscure critical areas of the sample container. Knowledge of the number of labels and the amount of overlap between them provides information about the size of the available analysis area and whether sample analysis is possible, and can be used to improve the accuracy of the results. To this end, the segmentation network 508 can be trained with expanded images of multiple labels with different arrangements.

[0077] The condition of labels may also play an important role in sample analysis. During the handling and processing of sample containers, interactions with labels on sample containers may cause labels to deviate from an ideal and / or newly applied state. For example, a machine handling a sample container may tear off a portion of a label thereon, or cause a label to peel from the surface of the sample container. In other cases, labels may bunch together when applied, preventing the label from lying flat on the sample container. This situation may affect the appearance of the label and may confuse a model designed to recognize only perfect labels. Incorporating information about the changing label condition into a model (e.g., segmentation network 508) makes the model robust to label deformations and other changes that may occur in real-world scenarios. For these reasons, the segmentation network 508 can be trained with expanded images with misplaced, torn, bunched, etc. labels.

[0078] The placement of the sample container may also play an important role in viewpoint detection. Variations in the shape of the sample container may cause it to appear tilted in the expanded image, shorter or longer than the average container, or thicker or thinner than the average container. These types of variations may occur due to misplacement by a robot or other handling device, or due to fitting the sample container into a holder that is not specifically designed for the sample container. The teaching model (e.g., segmentation network 508) is robust to differences in the appearance of the sample container, which protects it from abnormal placement while still providing accurate results. This increases the flexibility of the model because it will produce accurate results despite possible variations in real-world settings.

[0079] The binary segmentation of the pixels of the segmentation mask 802 between label and non-label pixels can be extended to multi-class segmentation, where other parts of the sample container 102 can be identified. Other parts of the sample container 102 that can be identified include, but are not limited to, fluid-air interfaces, bar codes, manufacturing labels, caps, and the like. The locations of these and other features derived from their segmentation can be used in conjunction with the midpoints described previously to determine the optimal viewpoint. For example, knowledge of other aspects of the sample container can be used to determine where to begin measuring the gap between the ends of the labels. In situations where different components may overlap, knowing where the other components are located on the sample container can help accurately determine the edges of the label. In some embodiments, the "coverage" component can be subtracted from the image to obtain the actual label boundary.

[0080] In some embodiments, a series of acquired images taken by the sensor 408 at different viewpoints can be used to construct a three-dimensional (3D) model of the sample container 102. The 3D model can provide an accurate representation of the container geometry and can be used to estimate additional properties, such as serum and red blood cell volumes. In this way, the quality inspection module 130 can obtain additional information about the fluid contents and geometry of the sample container 102 during the determination of the optimal viewpoint.

[0081] In one or more embodiments, the classification or regression model can use one or more original images directly, rather than using a segmentation network for viewpoint detection. This can be achieved, for example, by directly training the classification / regression network or by branching the segmentation network 508. In some embodiments, instead of generating an expanded image from a series of image captures, the classification network can be trained to directly predict the approximate amount of rotation required (e.g., aligning the midpoint of the label gap with the center of the field of view of the sensor used for image capture) by using one image from the series. This can remove the step of generating an expanded image. For example, the classification network can be trained to learn subtle differences in the curvature of the sample container from the captured image. This information may help determine how much the sample container should be rotated. Such a model can receive a captured image as input and return an indicator that relates to the category of the range of rotation required to identify the label gap and the center of the field of view of the imaging sensor. In some embodiments, and as further described below, the categories can be divided into groups of approximately 10 degrees (0-10, 10-20, etc.).

[0082] As another example of the classification model method, another network can be used to perform classification instead of using a segmentation network. For example, the 360-degree rotation of the sample container 102 can be divided into discrete rotation regions (e.g., each 20 to 30 degrees to obtain approximately 10-20 rotation regions). Given the original unfolded label, the network can be trained to predict which rotation region the best viewpoint is located in. If the best viewpoint is in the first 20-degree rotation region, the sample container 102 and / or the sensor 408 can be rotated by approximately 20 degrees. If the best viewpoint is in the second rotation region, the sample container 102 and / or the sensor 408 can be rotated by approximately 40 degrees, etc. The size of each rotation region can be adjusted based on the desired granularity (e.g., each rotation region can be greater than 30 degrees or less than 20 degrees). In some embodiments, a general convolutional neural network (CNN) can be used to perform classification. Example architectures include Inception, ResNet, ResNeXt, DenseNet, etc., although other CNN architectures can be used.

[0083] For the regression model, a reference point can be selected from the fully rotated image (e.g., the expanded image), and a label can be assigned to the image based on how much the imaged sample container has been rotated from the reference position. Using this information, the regression model can directly learn the relationship between the amount of rotation of the sample container 102 and / or the sensor 408 and the optimal viewpoint. The current viewpoint can then be compared to the optimal viewpoint to determine how much additional rotation is required for the sample container 102 and / or the sensor 408. Note that when there is a large gap between the current viewpoint (of the sensor 408) and the optimal viewpoint, the estimated additional rotation may not be very accurate. An iterative application can be employed until the viewpoint of the sensor 408 converges to the optimal viewpoint (e.g., until the optimal viewpoint is aligned with the observation center of the sensor 408).

[0084] The deep learning network can include a feature extraction portion followed by a task specific portion. The segmentation network 508 includes a task specific portion (also referred to as a segmentation portion) that determines the segmentation of the image (e.g., the classification of each pixel in the image, where pixels within the same label are considered to be "segments" of the image). The geometry of the segments in the image can be used to determine the ends of the labels. The feature extraction portion of the segmentation network 508 extracts the most relevant features of the image for segmentation or classification.

[0085] The feature extraction part and the segmentation part of the segmentation network 508 each include convolutional layers that process the image and its components. The feature extraction part extracts the most relevant features of the image, such as edges, shapes, etc., for segmentation or classification. These features are not necessarily interpretable, but can be used to generate the final result.

[0086] In some embodiments, the classification block can be included as a separate branch after the feature extraction portion of the segmentation network 508, and the classifier is trained using information obtained from the feature extraction portion of the segmentation network 508. The two tasks can be trained in parallel, or the segmentation block can be replaced with a classification block (as described below).

[0087] Figures 10A-10C An example embodiment of a training network that can be used with one or more embodiments provided herein is shown. Specifically, Fig. 10A An embodiment of a segmentation network 508 is shown in which an input image 1002 is processed by a feature extractor 1004 that extracts image features from the input image 1002. These image features are then fed to a segmentation portion (e.g., segmenter 1006) of the segmentation network 508, which classifies each pixel in the image and generates a segmented image 1008 (e.g., such as Figure 8 segmentation mask 802).

[0088] Fig. 10B 1002 is processed by a feature extractor 1004′ that extracts image features from the input image 1002. In some embodiments, the feature extractor 1004′ ( Fig. 10B ) can be similar to the segmented network 508 ( Fig. 10A ) employs a feature extractor 1004, while in other embodiments, the feature extractors 1004 and 1004' may be different (e.g., employing different types, numbers and / or arrangements of convolutional layers, using different training data sets, etc.).

[0089] The image features from the feature extractor 1004' are fed to a classification portion 1012 of a classification network 1010, which classifies the image features and identifies how far the sample container should be rotated. For example, the classification portion 1012 may provide a number 1014 indicating a predicted class that identifies the range of rotation required to align the label gap with the center of the field of view of the imaging sensor. In some embodiments, the classes may be divided into groups of approximately 10 degrees (0-10 degrees, 10-20 degrees, etc.) or another suitable group size (e.g., 5 degrees, 20 degrees, etc.), where the 360 ​​degree rotation of the sample container 102 is divided into discrete rotation regions (e.g., 20 to 30 degrees each to obtain approximately 10-20 rotation regions, etc.).

[0090] Fig. 10CAn embodiment of a segmentation and classification network 1016 is shown that includes both a segmenter 1006 and a classification portion 1012. For example, an input image 1002 is processed by a feature extractor 1004 that extracts image features from the input image 1002. These image features are then fed into both the segmenter 1006 and the classification portion 1012 of the segmentation network 508. For example, the feature extractor 1004, the segmenter 1006, and / or the classification portion 1012 may include any suitable number of convolutional layers.

[0091] The embodiments provided herein can use a single sensor to capture data. In addition, a single rotating platform can be used to orient the sample container in front of the sensor. These improvements greatly reduce the size and cost of the setup. The image capture device can also be made into modular components and used in other workflows with little setup overhead, as well as perform new analyses.

[0092] The embodiments provided herein ensure that the sample container is correctly oriented relative to the sensor. The system analyzes the sample container to detect the optimal viewpoint and can adjust (e.g., rotate) the sample container to the optimal viewpoint. This relieves the rest of the system of the responsibility for correctly placing / orienting the sample container for analysis and can improve the accuracy of the final analysis.

[0093] In addition, embodiments provided herein accommodate variations in the placement of labels applied to the outside of sample containers. Label placement can be done by a machine, in which case the label can be perfectly placed along the axis of the container, or by a person, who may introduce variations in alignment. For this case, the label may tilt and cover different parts of the sample container as it is wrapped around the sample container. The label may also not be in brand new condition when applied; there may be artifacts such as dirt, bunching of the label paper, tears, etc. The present system can be trained for these kinds of variations and still correctly predict the parts of the image that correspond to the label and the parts that do not. This robustness provides reliable results despite potential variations in label conditions.

[0094] Although the present disclosure is susceptible to various modifications and alternative forms, specific method and device embodiments have been shown by way of example in the drawings and are described in detail herein. However, it should be understood that the specific method and device disclosed herein are not intended to limit the present disclosure.

[0095] One or more of the methods described herein can be implemented in a computer program code as one or more computer program products, such as part of an application executable on a computer. Other systems, methods, computer program products, and data structures can also be provided. Each computer program product described herein can be carried by a computer-readable non-transitory medium (e.g., a DVD, a hard drive, a random access memory, etc.).

[0096] Therefore, while the present invention has been disclosed in connection with exemplary embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the invention as defined by the following claims.

Claims

1. A method for determining a viewpoint for optically inspecting a sample in a sample container, comprising: capturing image data of a sample container including a sample using a sensor, wherein a portion of the sample container includes a label having a first end and a second end; rotating at least one of the sample container and the sensor about a central axis of the sample container such that the sensor captures image data including at least the first end of the label and the second end of the label; generating an expanded image of the sample container using the captured image data; processing the expanded image to characterize the label and generate label characterization information; and The tag characterizing information is employed to identify a viewpoint through which to optically inspect a sample within a sample container. 2 . The method of claim 1 , wherein capturing image data with a sensor comprises capturing a plurality of images of the sample container at different viewpoints relative to the sample container.

3. The method of claim 2, further comprising using two or more of the plurality of images to determine one or more of a height, a width, and a diameter of the sample container. 4 . The method of claim 1 , wherein rotating at least one of the sample container and the sensor comprises rotating at least one of the sample container and the sensor 360 degrees relative to each other.

5. The method of claim 1, wherein using the captured image data to generate an expanded image of the label comprises stitching together multiple images of the sample container, each image depicting a different portion of the sample container.

6. The method of claim 5, wherein stitching together the plurality of images comprises: obtaining an image window from each of the plurality of images, each image window including only a portion of pixels of its corresponding image; as well as Stitch together image windows from multiple images. 7 . The method of claim 1 , wherein processing the unwrapped image to characterize the label comprises processing the unwrapped image through a trained segmentation network to generate a segmentation mask characterizing the label.

8. The method of claim 7, wherein the segmentation mask comprises a plurality of pixels, wherein each pixel is identified as a label pixel or a non-label pixel.

9. The method of claim 7, wherein employing the label representation information to identify the viewpoint comprises employing a segmentation mask to identify the viewpoint by: determining a center of a gap between a first end and a second end of the tag; and Determine how far to rotate the sample container so that the center of the gap between the first and second ends of the label is aligned with the center of the sensor's field of view.

10. The method of claim 1, further comprising inspecting the sample using the identified viewpoint.

11. The method of claim 10, further comprising characterizing the sample based on inspection of the sample using the identified viewpoint.

12. An apparatus adapted to determine a viewpoint for optically inspecting a sample in a sample container, comprising: a sensor configured to capture image data of a sample container including a sample, wherein a portion of the sample container includes a label having a first end and a second end; a rotation mechanism configured to rotate at least one of the sample container and the sensor about a central axis of the sample container such that the sensor captures image data including at least the first end of the label and the second end of the label; and A computer coupled to the sensor and the rotating mechanism, the computer comprising computer program code which, when executed by the computer, causes the computer to: directing the rotation mechanism to rotate at least one of the sample container and the sensor about a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; generating an expanded image of the sample container using the captured image data; processing the expanded image to characterize the label and generate label characterization information; and The tag characterizing information is employed to identify a viewpoint through which to optically inspect a sample within a sample container.

13. The device according to claim 12, wherein: The computer includes computer program code that causes the sensor to capture a plurality of images of the sample container at different viewpoints relative to the sample container.

14. The device according to claim 12, wherein: The computer includes computer program code that generates an expanded image by stitching together a plurality of images of the sample container, each image depicting a different portion of the sample container.

15. The device according to claim 14, wherein: The computer includes computer program code that generates an expanded image by: obtaining an image window from each of the plurality of images, each image window including only a portion of pixels of its corresponding image; as well as Stitch together image windows from multiple images.

16. The device according to claim 12, wherein: The computer includes computer program code that processes the unwrapped image to characterize a label by processing the unwrapped image through a trained segmentation network to generate a segmentation mask.

17. The device according to claim 16, wherein: The segmentation mask includes a plurality of pixels, wherein each pixel is identified as a label pixel or a non-label pixel.

18. The device according to claim 16, wherein: The computer comprises computer program code which employs the segmentation mask to: determining a center of a gap between a first end and a second end of the tag; and Determine how far to rotate the sample container to align the center of the gap between the first and second ends of the label with the center of the field of view of the sensor.

19. The device according to claim 12, wherein: The computer includes computer program code that employs the sensor to examine a sample using the identified viewpoint.

20. The device according to claim 19, wherein The computer includes computer program code that characterizes the sample based on inspection of the sample using the identified viewpoint.

21. A diagnostic analysis system comprising: track; a carrier movable on the track and configured to contain a sample container including a sample, wherein a portion of the sample container includes a label having a first end and a second end; a sensor configured to capture image data of a sample container; a rotation mechanism configured to rotate at least one of the sample container and the sensor about a central axis of the sample container such that the sensor captures image data including at least the first end of the label and the second end of the label; and A computer coupled to the sensor and the rotating mechanism, the computer comprising computer program code which, when executed by the computer, causes the computer to: directing the rotation mechanism to rotate at least one of the sample container and the sensor about a central axis of the sample container so that the sensor captures image data including at least the first end of the label and the second end of the label; generating an expanded image of the label using the captured image data; processing the expanded image to characterize the label and generate label characterization information; and The tag characterizing information is employed to identify a viewpoint through which to optically inspect a sample within a sample container.

Images