Image-based determination using intelligent monitoring structures

Abstract

The present invention relates to the use of intelligent monitoring structures and machine learning, etc. to correct errors in instruments, operations, etc.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 719,129, filed on August 16, 2018, and U.S. Provisional Patent Application No. 62 / 764,886, filed on August 16, 2018, the contents of which are incorporated herein by reference in their entirety. The entire disclosure of any publication or patent document mentioned herein is incorporated by reference in its entirety. Technical Field

[0003] Among other things, the present invention relates to devices and methods for performing biological and chemical assays, computational imaging, artificial intelligence, and machine learning. Background Art

[0004] In biological and chemical assays, including complete blood cell counts, accuracy is essential. However, instrumental and operational errors are unavoidable when using low-cost instruments and limited resources. The present invention provides, among other things, apparatus and methods for improving the accuracy of image-based biological / chemical assays in samples. Summary of the invention

[0005] The following brief summary is not intended to include all features and aspects of the invention.

[0006] One aspect of the invention is a method of improving imaging-based assays using a sample holding device with specially designed monitoring structures for assay accuracy, efficiency, error detection, monitoring, or any combination thereof.

[0007] Another aspect of the invention is a method of micro-selective image assay (MSIA) that can perform multiple assay applications on a single image of a sample captured by an imager.

[0008] Another aspect of the invention is a method of detecting monitoring structures and locating their centers from an image of a sample for assay, wherein the method combines machine learning with error correction using predetermined structural properties associated with a plurality of monitoring structures.

[0009] Another aspect of the invention is a method of estimating the true lateral dimension (TLD) in an image-based assay, whereby the actual size or morphological features of an analyte in a sample image can be determined.

[0010] Another aspect of the invention is a method of determining the actual area of ​​a region of interest in a sample image and the associated sample volume, by which image-based determinations can be performed on any selected sub-region to achieve flexibility and granularity.

[0011] Another aspect of the invention is a method of monitoring the quality of a sample holding device and the quality of sample preparation in an image-based assay from an image of a monitoring structure in the sample holding device.

[0012] Another aspect of the invention is a method of using a pre-designed monitoring structure in a sample holding device to adjust the operation of an imager in an image-based assay.

[0013] Another aspect of the present invention is a method of removing defects or foreign matter (eg, bubbles, dust, etc.) from an image of a sample captured by an imager in an image-based assay.

[0014] Another aspect of the present invention is a method of building a machine learning model for image-based assays using an intelligent monitoring architecture, wherein two methods based on original images of the sample and based on transformed sample images are described.

[0015] Another aspect of the invention is the use of the methods and algorithms described herein for image-based determination of actual usage of red blood cells in a complete blood count (CBC).

[0016] A method of using a device to improve an imaging-based assay, comprising:

[0017] (a) having a sample holder comprising a first plate and a second plate facing each other and a plurality of monitoring structures on a sample contact surface of one or both plates, wherein the monitoring structures have at least one pre-designed and predetermined parameter of geometry and / or optical properties, wherein the sample contact surface contacts the sample;

[0018] (b) sandwiching a sample containing or suspected of containing an analyte between corresponding sample contact surfaces of two plates to form a thin layer of 200 μm thick or thinner, wherein the sample on the sample contact surface is mixed and reacted with a reagent, and wherein the sandwiching, mixing or reaction is susceptible to error;

[0019] (c) imaging the sample together with a monitoring structure on a sample contact area using an imager, wherein the imager is susceptible to imperfections in the imaging, and wherein the imaging operation is susceptible to errors; and

[0020] (d) analyzing the image captured in step (c) using an algorithm to detect parameters associated with the analyte, wherein the analysis comprises comparing the image with the at least one pre-designed and predetermined parameter, and detecting and / or correcting the defects and errors using the at least one pre-designed and predetermined parameter. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Those skilled in the art will appreciate that the drawings described below are for illustrative purposes only. The drawings are not intended to limit the scope of the present teaching in any way. The drawings are not drawn entirely to scale. In the figures where experimental data points are given, the lines connecting the data points are only used to guide the observation of the data and have no other meaning.

[0022] Figures 1A and 1B show side views of a device for imaging-based assays. Figure 1A shows a solid surface with protruding monitoring labels. Figure 1B shows a solid surface with grooved monitoring labels. In an algorithm, the properties (e.g., spacing and distance) corresponding to the monitoring labels can be used to determine the properties of an analyte in a sample.

[0023] Figures 2A and 2B show side views of a device for imaging-based assays. Figure 2A shows how a monitoring marker (e.g., a protruding type) can be a structure separate from a spacer. Figure 2B shows how a monitoring marker can be the same structure as a spacer. In an algorithm, the properties corresponding to the monitoring marker can be used to determine the properties of an analyte in a sample.

[0024] Figure 2C shows a side view of a device for image-based determination. Figure 2C shows how the monitoring marker (e.g., groove-type) can be a structure separated from the spacer. In the algorithm, the characteristics corresponding to the monitoring marker can be used to determine the nature of the analyte in the sample.

[0025] Figures 2D and 2E show side views of a device for image-based assays. Figures 2D and 2E show how the monitoring markers can be structures separate from the spacer and disposed on two sample contacting areas of the device. In an algorithm, the properties corresponding to the monitoring markers can be used to determine the nature of the analyte in the sample.

[0026] 5 is an exemplary diagram and workflow for an algorithm using monitoring markers (eg, pillars) along with an imaging process and / or machine learning.

[0027] FIG6-A shows an analyte detection and localization workflow according to some embodiments of the present invention, which includes two stages, training and prediction.

[0028] FIG. 6-B illustrates a process of removing an item from an ordered list according to some embodiments of the present invention.

[0029] FIG. 7-A shows an embodiment of a QMAX card used in an image-based assay.

[0030] FIG8 shows a flow chart of an image-based assay using a sample holding device of a QMAX card.

[0031] FIG. 9 shows a flow chart of TLD / FoV estimation based on pillars or monitoring markers in an image-based assay.

[0032] FIG10 shows a flow chart for training a machine learning model for column or monitoring marker detection.

[0033] FIG11 shows a flowchart for generating a training database for an image transformation based machine learning model.

[0034] FIG12 shows a flowchart for training an image transformation model for image transformation based pillar or surveillance marker detection.

[0035] 13 is a sample image of an image-based assay with a column in the sample holding device.

[0036] FIG. 14 shows a pole detected in a transformed image for pole or monitoring marker detection.

[0037] FIG. 15 shows defects of bubbles and dust in the sample used for measurement.

[0038] 16 is three side views of a sample holding device of a QMAX card with posts or monitoring markers in an image-based assay using an imager.

[0039] FIG. 17 is an image of a sample with large bubbles.

[0040] FIG. 18 shows pillars detected in an image of a sample.

[0041] FIG. 19 is a graph of light intensity curves versus position.

[0042] Detailed Description of Exemplary Embodiments

[0043] The following detailed description illustrates some embodiments of the present invention by way of example and not limitation. The section headings and subheadings used herein are for organizational purposes only and should not be construed as limiting the subject matter described in any way. The content under a section heading and / or subtitle is not limited to the section heading and / or subtitle, but applies to the entire description of the present invention.

[0044] The citation of any publication for its disclosure prior to the filing date should not be construed as an admission that the present invention is not entitled to antedate such publication by virtue of prior invention. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed.

[0045] In an image-based assay for determining an analyte in a sample, an imager is used to create an image of the sample in a sample holder, and the image is used to determine a property of the analyte or the sample for use in the assay.

[0046] However, many factors may distort the image (i.e., an image that is different from the real sample or under non-ideal conditions). Image distortion can lead to inaccurate determination of analyte properties. For example, one fact is poor focus, because the biological sample itself does not have sharp edges that are preferred in focus. When the focus is poor, the object size will be different from the real object size, and other objects (e.g., blood cells) may become unrecognizable. Another example is that the lens may not be perfect, resulting in different degrees of distortion in different locations of the sample. Another example is that the sample holder is not in the same plane of the optical imaging system, resulting in good focus in one area and poor focus in another area.

[0047] The present invention relates to an apparatus and method for obtaining a "true" image from a distorted image in an image-based assay, thereby improving the accuracy of the assay.

[0048] One aspect of the present invention is an apparatus and method for using a monitoring marker having an optically observable flat surface that is parallel to adjacent surfaces.

[0049] Another aspect of the invention is an apparatus and method for using a QMAX card to form a uniform thickness layer of at least a portion of a sample in a sample holding area of ​​the QMAX card and using monitoring marks on the card to improve assay accuracy.

[0050] Another aspect of the invention is an apparatus and method for using monitoring markers in image-based assays along with computational imaging, artificial intelligence, and / or machine learning.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. Although any methods and materials similar or equivalent to those described herein can also be used to practice or test the present teachings, some exemplary methods and materials are now described.

[0052] The term "lateral dimension" refers to the linear dimension in the plane of the thin sample layer being imaged.

[0053] The terms "true lateral dimension (TLD)" and "field of view (FoV)" are interchangeable.

[0054] The term "microfeatures in a sample" may refer to analytes, microstructures, and / or microvariations of matter in a sample. Analytes refer to particles, cells, macromolecules such as proteins, nucleic acids, and other moieties. Microstructures may refer to microscale differences in different materials. Microvariations refer to microscale changes in local properties of a sample. Examples of microvariations are changes in local optical index and / or local mass. Examples of cells are blood cells, such as white blood cells, red blood cells, and platelets.

[0055] As used herein, the term "sample" refers to a material or material mixture containing one or more analytes or entities of interest. In certain embodiments, the sample is a body fluid sample from a subject. In some cases, a solid or semi-solid sample can be provided. The sample may include tissues and / or cells collected from a subject. The sample may be a biological sample. Examples of biological samples may include, but are not limited to, blood, serum, plasma, nasal swabs, nasopharyngeal washes, saliva, urine, gastric juice, spinal fluid, tears, feces, mucus, sweat, earwax, grease, glandular secretions, cerebrospinal fluid, tissue, semen, vaginal secretions, interstitial fluid from tumor tissue, eye fluid, spinal fluid, throat swabs, exhaled condensates (e.g., breath), hair, nails, skin, biopsy, placental fluid, amniotic fluid, umbilical cord blood, lymph, cavity fluid, sputum, pus, microbiota, meconium, breast milk and / or other excreta. The sample may include nasopharyngeal washes. Nasal swabs, throat swabs, fecal samples, hair, nails, earwax, breath and other solid, semi-solid or gas samples can be processed in extraction buffer before analysis (e.g., fixed or variable amount of time). If necessary, extraction buffer or its aliquots can then be processed similarly to other fluid samples. The example of the tissue sample of the subject may include but is not limited to connective tissue, muscle tissue, nerve tissue, epithelial tissue, cartilage, cancer sample or bone. In a specific embodiment, a sample can be obtained from a subject, such as a person, and can be processed before being used for the subject to measure. For example, before analysis, protein / nucleic acid can be extracted from a tissue sample before use, and its method is known. In a specific embodiment, the sample can be a clinical sample, for example, a sample collected from a patient. The sample can also be a sample of food, environment, etc. Some samples have a deformable but non-free flowing shape (e.g., sputum).

[0056] The term "analyte" refers to any substance suitable for testing in the present invention. Analytes include, but are not limited to, atoms, molecules (e.g., proteins, peptides, DNA, RNA, nucleic acids or other molecules), cells, tissues, viruses, bacteria, and nanoparticles with different shapes. Biomarkers are analytes.

[0057] As used herein, the terms "determine," "measure," and "evaluate," and "assay" are used interchangeably and include both quantitative and qualitative determination.

[0058] As used herein, the term "luminescent label" refers to a label that is capable of emitting light when under external excitation. This can be luminescent. Fluorescent labels (which include dye molecules or quantum dots) and luminescent labels (e.g., electroluminescent or chemiluminescent labels) are types of luminescent labels. External excitation is light (photons) for fluorescence, electric current for electroluminescence, and chemical reaction for chemiluminescence. External excitation can be a combination of the above.

[0059] The phrase "labeled analyte" refers to an analyte that is detectably labeled with a luminescent label such that the analyte can be detected by evaluating the presence of the label. The labeled analyte can be directly labeled (i.e., the analyte itself can be directly conjugated to the label, such as through a strong bond, such as a covalent bond or a non-covalent bond), or the labeled analyte can be indirectly labeled (i.e., the analyte is bound by a directly labeled second capture agent).

[0060] With respect to nucleic acids, the terms "hybridize" and "bind" are used interchangeably.

[0061] The term "hybridization" refers to the reaction of one or more polynucleotides to form a complex stabilized by hydrogen bonds between nucleotide residue bases. Hydrogen bonding can occur by Watson-Crick base pairing, Hoogstein binding, or in any other sequence-specific manner. The complex may comprise two chains forming a duplex structure, three or more chains forming a multi-stranded complex, a single self-hybridizing chain, or any combination of these.

[0062] As known to those skilled in the art, hybridization can be performed under various stringency conditions. Suitable hybridization conditions make the recognition interaction between the capture sequence and the target nucleic acid sufficiently specific and sufficiently stable. Conditions that increase the stringency of the hybridization reaction are well known and disclosed in the art. See, for example, Green et al. (2012), see below.

[0063] The terms "spacer" and "optical alignment mark" and "optical alignment mark" and "post" are interchangeable.

[0064] The term "assay" refers to a research (analytical) procedure in, but not limited to, laboratory, medicine, pharmacology, environmental biology, health care, and molecular biology - used to, but not limited to, qualitatively assess or quantitatively measure the presence, amount, concentration, or functional activity of a target entity (i.e., analyte). The analyte can be a drug, a biochemical, or a cell in an organism or an organic sample (e.g., human blood).

[0065] The term "image-based assay" refers to an assay procedure that utilizes an image of a sample captured by an imager, wherein the sample may be, but is not limited to, medical, biological, and chemical samples.

[0066] The term "imager" refers to any device that can take an image of an object. This includes, but is not limited to, a camera in a microscope, a smartphone, or special devices that can take images at various wavelengths.

[0067] The term "sample feature" refers to some property of a sample that represents a condition of potential interest. In certain embodiments, a sample feature is a feature that appears in an image of a sample and can be segmented and classified by a machine learning model or some other algorithm. Examples of sample features include, but are not limited to, the type of analyte in the sample, such as red blood cells, white blood cells, and tumor cells, and include analyte shape, count, size, volume, concentration, etc.

[0068] The term "defects in the sample" refers to foreign matter and artifacts that should not be present under ideal sample conditions or should not be considered in the sample. They can come from, but are not limited to, contaminants such as dust, bubbles, etc., and from peripheral objects, including structural objects in the sample, such as monitoring markers (e.g., columns) in the sample holding device. Defects in the sample can have significant size and occupy a significant amount of volume in the sample for determination, such as bubbles, where they can appear in different shapes, sizes, amounts and concentrations in the sample, and they are also sample-dependent from sample to sample.

[0069] The term "morphological characteristics" of an analyte refers to the appearance (eg, shape, color, size, etc.) and structure of the analyte.

[0070] The term "homography" refers to a class of collinear transformations caused by isomorphisms of projective spaces. It is known in the field of image processing and is applied in camera models to characterize image planes and corresponding physical planes in the real world.

[0071] The term "machine learning" refers to algorithms, systems, and devices in the field of artificial intelligence that typically use statistical techniques and artificial neural networks to provide computers with the ability to "learn" from data (i.e., gradually improve performance on a specific task) without being explicitly programmed.

[0072] The term "artificial neural network" refers to a hierarchically connected system inspired by biological networks that can "learn" to perform tasks by considering examples, typically without being programmed with any task-specific rules.

[0073] The term "convolutional neural network" refers to a class of multi-layer feed-forward artificial neural networks that are most commonly applied to analyzing visual images.

[0074] The term "deep learning" refers to a large class of machine learning methods in artificial intelligence (AI) that learn from data with a certain deep network structure.

[0075] The term "machine learning model" refers to a trained computational model built from a training process in machine learning of data. The trained machine learning model is applied by a computer in the inference phase, which gives the computer the ability to perform specific tasks (e.g., detecting and classifying objects). Examples of machine learning models include ResNet, DenseNet, etc., which are also called "deep learning models" due to the depth of their hierarchical network structure.

[0076] The term "image segmentation" refers to the image analysis process of dividing a digital image into multiple image patch segments (groups of pixels, usually with a set of bitmap masks covering the image segments surrounded by their segment boundary outlines). Image segmentation can be achieved by image segmentation algorithms in image processing, such as watershed, grabcut, mean shift, etc., and it can also be achieved by dedicated machine learning algorithms, such as MaskRCNN, etc.

[0077] A. Monitoring labels on solid surfaces

[0078] A1-1. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0079] (a) a solid surface comprising a sample contact area for contacting a sample comprising microfeatures; and

[0080] (b) one or more monitoring markers, wherein the monitoring markers:

[0081] i. Made of a different material from the sample;

[0082] ii. being located inside the sample during determination of the microstructure, wherein the sample forms a thin layer having a thickness of less than 200 um on the sample contact area;

[0083] iii. having a lateral linear dimension of about 1 um (micrometer) or more, and

[0084] iv. having at least one lateral linear dimension of 300um or less; and

[0085] During the determination process, the imager images at least one monitoring marker.

[0086] wherein is used during the determination of the analyte; and the geometric parameters (e.g., shape and size) of the monitoring marks and / or the spacing between the monitoring marks are (a) predetermined and known prior to the determination of the analyte, and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0087] A1-2. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0088] a solid surface comprising a sample contact area for contacting a sample comprising microfeatures; and

[0089] One or more monitoring marks, wherein each monitoring mark comprises a protrusion or groove from the solid surface, wherein:

[0090] i. The protrusion or groove comprises a flat surface substantially parallel to an adjacent surface, the adjacent surface being a portion of the solid surface adjacent to the protrusion or groove;

[0091] ii. The distance between the flat surface and the adjacent surface is about 200 micrometers (um) or less;

[0092] iii. a flat surface having (a) a linear dimension of at least about 1 um or more, and (b) at least one area having a linear dimension of 150 um or less;

[0093] iv. the planar surface of at least one monitoring mark is imaged by an imager used in determining microfeatures; and

[0094] v. The shape of the planar surface, the size of the planar surface, the distance between the planar surface and adjacent surfaces, and / or the spacing between monitoring marks are (a) predetermined and known prior to measurement of the microfeatures, and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0095] B. Monitoring mark on QMAX card

[0096] A2-1. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0097] a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0098] i. The first plate and the second plate are movable relative to each other into different configurations;

[0099] ii. Each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising microfeatures;

[0100] iii. one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate,

[0101] iv. The spacers have a substantially uniform height and a fixed spacing distance (ISD) equal to or less than 200 microns;

[0102] v. The monitoring mark is made of a different material from the sample;

[0103] vi. monitoring the marker being located inside the sample during determination of the microstructure, wherein the sample forms a thin layer having a thickness of less than 200 um on the sample contact area; and

[0104] vii. The monitoring mark has a lateral linear dimension of about 1 um (micrometer) or more and has at least one lateral linear dimension of 300 um or less;

[0105] During the determination process, the imager images at least one monitoring marker.

[0106] wherein the shape of the planar surface, the size of the planar surface, the distance between the planar surface and an adjacent surface, and / or the spacing between monitoring marks are (a) predetermined and known prior to the determination of the microfeatures, and (b) used as parameters in an algorithm for determining a property associated with the microfeatures;

[0107] One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both plates;

[0108] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying a compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is regulated by the plates and the spacers; and

[0109] The monitoring marker is (i) a different structure from the spacer, or (ii) the same structure as the spacer.

[0110] A2-2. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0111] a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0112] i. The first plate and the second plate are movable relative to each other into different configurations;

[0113] ii. Each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising microfeatures;

[0114] iii. one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate;

[0115] iv. The spacers have a substantially uniform height and a fixed spacing distance (ISD) equal to or less than 200 microns;

[0116] v. Each monitoring mark comprises a protrusion or groove on one or two sample contact areas;

[0117] vi. The protrusion or groove comprises a flat surface substantially parallel to an adjacent surface, the adjacent surface being a portion of the solid surface adjacent to the protrusion or groove;

[0118] vii. The distance between the flat surface and the adjacent surface is about 200 micrometers (um) or less;

[0119] viii. A planar surface having (a) a linear dimension of at least about 1 um or greater, and (b) at least one area having a linear dimension of 150 um or less;

[0120] ix. The planar surface of at least one monitoring mark is imaged by an imager used in determining microfeatures; and

[0121] x. The shape of the planar surface, the size of the planar surface, the distance between the planar surface and adjacent surfaces, and / or the spacing between monitoring marks are (a) predetermined and known prior to measurement of the microfeatures, and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0122] One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both plates;

[0123] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying a compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is regulated by the plates and the spacers; and

[0124] The monitoring marker is (i) a different structure from the spacer, or (ii) the same structure as the spacer.

[0125] A3. An apparatus for image-based determination, comprising:

[0126] The device of any preceding device embodiment, wherein the device has at least five monitoring marks, wherein at least three of the monitoring marks are not aligned on a linear line.

[0127] A4. An apparatus for determining an analyte in a sample using an imager, the system comprising:

[0128] (a) an apparatus as in any preceding apparatus embodiment; and

[0129] (b) An imager for measuring a sample containing an analyte.

[0130] A5. A system for performing an imaging-based assay, the system comprising:

[0131] (a) A device as in any preceding device embodiment;

[0132] (b) an imager for measuring a sample containing an analyte; and

[0133] (c) an algorithm for utilizing the monitoring markers of the device to determine a property associated with the analyte.

[0134] In some embodiments, the thickness of the thin layer is configured so that the analytes form a monolayer in the sample holder. The term "monolayer" means that in the thin sample layer inside the sample holder, there is substantially no overlap between two adjacent analytes in a direction perpendicular to the plane of the sample layer.

[0135] C. Monitoring markers with computational imaging artificial intelligence and / or machine learning

[0136] Another aspect of the invention is to utilize a column or monitoring tag of a sample holding device with computational imaging, artificial intelligence and / or machine learning. It has a process of forming an image based on measurements, using algorithms to process the image and map the objects in the image to their actual size in the real world. Machine learning (ML) is applied in the present invention to learn the salient features of objects in the sample captured by the ML model, constructed and trained from images of the sample taken by the imager. Intelligent decision logic is constructed and applied in the inference process of the present invention to detect and classify target objects in the sample based on the knowledge embedded in the ML model.

[0137] FIG9 shows a block diagram of a process 100 for TLD (true lateral dimension) / FoV (field of view) estimation in the present invention. In various implementations of process 100, actions may be removed, combined, or decomposed into sub-actions. The process starts at action module 101, where the process captures an image of a sample for determination, where the image is taken by an imager such as a QMAX card above the sample holding device, which has a column or monitoring mark and the structure described in the previous section of the monitoring mark on the QMAX card. The captured image from action module 101 is fed as input to action module 102, where the image for determination is processed and searched to detect and locate the center of the column or monitoring mark. In a typical implementation, process 102 has a pre-trained machine learning model M_column to detect the column or monitoring mark in the image for determination.

[0138] As described above, the columns or monitoring marks in the present invention have known shapes and sizes and are manufactured by a high-precision nanoimprint manufacturing process. In addition, these columns or monitoring marks are distributed in the sample holding device in a predefined periodic pattern and have known spacing between them. These features in the present invention become critical for process 100 because the columns or monitoring marks are small, about ~30 microns, surrounded by the sample in the sample holding device, and are subject to strong light scattering and diffraction between the particles in the sample. In addition, they are usually not in the focus of the imager because the imager focuses on the analyte in the sample for image-based determination, not on the structure of the sample holder. Therefore, even if a machine learning model is used for detection, errors will occur, in which some columns are not detected, detected incorrectly, or detected as incorrect or inaccurate center positions, making the direct use of the detected columns and center positions unreliable and introducing errors for TLD / FoV estimation.

[0139] As illustrated in FIG. 9 , action module 103 obtains the detected columns and their centers from action module 102 in process 100, and it performs post-error correction and estimation refinement based on the characteristics of the columns or monitoring marks described in the present invention. In particular, the detected columns should be aligned in a predefined known periodic pattern, and the distance between the centers of two adjacent columns should be subject to a known spacing, etc. The actions taken by action module 103 are to eliminate false detections, align the detected columns in a design pattern known to the manufacturer, use the periodicity of their distribution to discover others, and adjust the position of the column center according to the known periodicity and spacing between them. The combination of actions taken by action modules 102 and 103 in the present invention makes the column or monitoring mark detection robust to the accuracy of TLD / FoV estimation.

[0140] In action module 103 of process 100, it estimates a homography transformation based on the detected column centers from action modules 102 and 103. In the field of image processing, homography transformation is known, where the image captured by the imager on the sample is modeled by a perspective projection between the object in the image being captured and the actual object in the real world imaged on it. This perspective projection can be characterized by a homography transformation (also known as perspective transformation) described by the transformation matrix H described below:

[0141]

[0142] It is represented by x′=Hx, where H is a 3×3 non-singular homogeneous matrix that maps objects in the image taken by the imager to objects in the actual sample plan being imaged. According to this matrix, the length or area of ​​the object in the image taken by the imager is mapped to its actual size in the real world, and through this matrix, the TLD / FoV of the object and its actual size in the image can be determined.

[0143] However, in order to determine the transformation matrix H, it is necessary to know at least 4 pairs of points corresponding to the mapping of H from the image captured by the imager to the actual sample plan being imaged. And these 4 pairs of points are used as anchors, which combine the image captured by the imager and the actual sample plan imaged by the homography transformation between them, and they require at least 4 non-collinear column centers to make the H matrix non-singular. Through the actions in action module 102, and by applying the structural features of the columns or monitoring marks in action module 103, more than 4 non-collinear column center points can be detected, and their corresponding positions in the actual image plane are determined based on the structural features of the columns or monitoring marks in the sample holding device. In this way, the action module 104 of process 100 estimates the homography transformation matrix H for TLD / FoV estimation based on the detected column centers from action module 103. In addition, in action module 104, it uses the estimated transformation matrix H for TLD / FoV estimation in the subsequent image-based measurement process.

[0144] In the present invention, a dedicated machine learning model for column or monitoring marker detection is constructed from training data. In some embodiments of the present invention, column and monitoring marker detection is performed directly on the image used for determination. Figure 10 shows a block diagram of a process 200 for constructing a machine learning model for column or monitoring marker detection on the original input image used for determination. The process starts with an action module 201, wherein the process obtains a set of images for determination from an imager in a training database DB0. These images are collected by taking images of the sample in a sample holding device (e.g., a QMAX card). In an action module 202, each training image is obtained from DB0, and the column or monitoring marker in each image is marked. The marked images are saved in a second training database DB1 for machine learning model training. The training data set DB1 is specifically used to train a machine learning model to detect columns or monitoring markers, which is different from a typical training database for detecting analytes in samples.

[0145] In action module 203, a new training database DB1 is obtained from action module 202, and a machine learning model structure in the form of a deep neural network is selected to train the model for the training database DB1. In some embodiments of the present invention, a machine learning model of RetinaNet is used, and in some other embodiments, a machine learning model of Fast-RCNN is selected. Using the training database DB1 for column or monitoring marker detection, the machine learning model is trained using Tensorflow and PyTourch. Process 200 ends with action module 205, in which the machine learning model obtained from action module 204 is verified and saved for use in the determination application.

[0146] FIG. 11 is a block diagram of creating a specialized training database for a machine learning method based on image transformation for column and monitoring mark detection. Process 300 begins with action module 301, which acquires a training image database DB1A consisting of images taken by an imager for determination. In action module 302, images from DB1A are marked for columns or monitoring marks. The marked images in DB1A are input to action module 303, where the marked images are transformed by covering the marked columns or monitoring marks with a white mask along the contours of the marked columns or monitoring marks. The transformed images from action module 303 are input to action module 304, where the images are further transformed by applying a black mask to the areas in the image that are not covered by the white mask from action module 303. In the present invention, there are two advantages to using a white mask for columns and a black mask for areas not covered by the white mask. One is to maximize the contrast between the columns and other areas, and the second is to suppress noise in the background, so that subsequent detection of the columns or monitoring marks and their centers is more robust. In action module 304, the image transformed by 302 and 303 is verified and saved in a new target image training database DB2A to train the image transformation model in pillar or detection mark detection.

[0147] FIG. 12 shows a block diagram of a machine learning model based on image transformation for column and monitoring mark detection. Process 400 starts with action module 401. Action module 401 loads a training image database DB1A composed of images captured by an imager for measurement. Action module 402 loads a target image training database DB2A obtained by transforming the images in DB1A using the previously described process 300. In action module 403, it uses the training image database DB1A and the paired target image training database DB2A from action module 402 as training targets. It selects a machine learning framework to train a machine learning model, which transforms the original image captured by the imager for measurement into a new image of a white column or monitoring mask and a black background in the target image training database. In some embodiments of the present invention, a machine learning framework of pixel-to-pixel (PP) transformation is selected, while in some other embodiments of the present invention, a machine learning framework of CycleGAN is used. Action module 403 uses the training image database DB1A and the paired target image transformation database DB2A from process 300 to train the machine learning model TModel. In an embodiment of the present invention, the column and monitoring mark detection comprises the following actions:

[0148] 1. Take the image to be measured as input;

[0149] 2. Applying the image transformation model TModel to transform the input image to measure the image with a white mask covering the pillars or monitoring marks, and covering the remaining areas with a black mask as described herein;

[0150] 3. Detecting pillars or monitoring markers in the transformed image;

[0151] 4. Re-map the detected pillar or monitoring marker centers back to the original input image;

[0152] 5. Error correction of the detected position of the post or monitoring mark relative to the original input image using constraints from a pre-designed post or monitoring mark structure in the sample holding device;

[0153] 6. Determine the center of the detection post or monitoring marker and refine its position in the sample image based on the constraints of the known distributed structure in the sample holding device; and

[0154] 7. Estimate the holomorphic transformation on the image of the sample based on the detected pillars or monitoring marker centers to determine the TLD / FoV for subsequent image-based determination.

[0155] FIG. 17 is an actual image of a sample taken by an imager used to assay red blood cells. It is apparent that the noise level in the image of the sample is quite high and that the focus on the monitoring structures (e.g., columns or monitoring markers) in the sample holder is poor in the image of the sample because in image-based assays, the focus is on the analyte and not on the prereferal. This makes the detection and location of the columns or monitoring markers in the sample image challenging. FIG. 18 shows the detection results of the columns of monitoring markers from an image of a sample in a transformed image-based assay of red blood cells using the systems and methods described herein.

[0156] In some embodiments of the invention, the image used for the assay is divided into disjoint patches and a patch-dependent holomorphic transformation is estimated for each image patch if there are at least 4 detected non-collinear column centers in the patch. Otherwise, the TLD / FoV in the image-based assay is determined using the holomorphic transformation estimated from all detected column centers in the image.

[0157] Micro-selective image determination using region of interest selection, volume estimation, and defect removal

[0158] The TLD / FoV estimation method described herein opens up possibilities for other applications in image-based determinations. In some embodiments of the present invention, it is used to remove defects in image-based determinations, such as bubbles, dust, etc. Depending on the environment, the determination operation, and the type of sample used for determination, bubbles and dust may appear in the sample. In order to better determine the accuracy, these defects in the sample used for determination should be removed from the sample. However, once these defects occur and are captured in the closed space of the sample holding device, this may be extremely difficult. Moreover, in many cases, the device is pressure-sealed and should not be opened again.

[0159] Figure 15 depicts the presence of bubbles and dust in a sample. In some embodiments of the present invention, micro-selective image measurement is performed, comprising:

[0160] 1. Take the image of the sample to be measured as input;

[0161] 2. Estimate the TLD / FoV of the sample image from (1) to estimate the area, size and subsequently the sample volume;

[0162] 3. Detecting defects in the image of the sample, such as bubbles, dust, etc., by means of a trained machine learning model, so as to measure and segment these defects in the image of the sample;

[0163] 4. Estimate the total area of ​​the segmented defects in the sample image and calculate their actual area size by using the estimated TLD / FoV from (2), which utilizes the sample holding device posts and monitoring marks;

[0164] 5. Estimate the actual volume of the sample corresponding to the total surface area of ​​the detected defects in the image of the sample for determination based on the area estimate from (4) and the known height of the sample in the sample holding device;

[0165] 6. removing the surface area of ​​the detected defect in the image of the sample for determination and updating the total volume of the sample by subtracting the defect volume estimate from (5), the defect volume estimate corresponding to the total volume under the surface area of ​​the detected defect in the image of the sample; and

[0166] 7. Perform image-based determination on selected areas in the updated sample image using the updated sample volume from (6).

[0167] In some embodiments of the present invention, the area removed from the image of the sample is larger than the area of ​​the detected defect with a margin Δ, and thus, a larger volume is actually removed in the image-based assay based on the enlarged defect area. One benefit of this operation is to further reduce the negative impact of the defect on the assay result, because analytes can attach to defects and can affect the local uniformity of the analyte distribution.

[0168] As described herein, the present invention critically utilizes the structure of a sample holding device, such as a QMAX card, and the posts or monitoring markers therein. In particular, the image of the sample taken by the imager on the sample holding device (e.g., a QMAX card) is a pseudo 3D image, because the height of the sample in the sample holding device is known a priori and is uniform. Thus, the sample volume corresponding to the surface area of ​​the object in the image of the sample can be determined once its actual area size in the original sample plan is available.

[0169] In order to obtain an estimate of the actual area in the original sample plan, a structure of pillars or monitoring markers in the sample holding device is utilized to make more reliable TLD / FoV estimates. This leads to the paradigm of micro-selective image measurement in image-based measurement, by which the measurement process can select specific areas / volumes for measurement from the image of the sample taken by the imager.

[0170] For example, in some embodiments of the invention, it is possible to select a specific area where the analytes do not form clusters, and the sample volume of the selected area is determined by an image of the sample and a uniform height of the sample in the sample holding device.

[0171] In some embodiments, it selects regions in the image of the sample for better assay accuracy based on fewer defects, better signal-to-noise ratio, focus conditions, etc. The combination of machine learning-based pillar and monitoring mark detection for FoV estimation and machine learning-based defect detection and segmentation to reduce variations in the sample makes the methods described in the present invention flexible and resilient in image-based assays.

[0172] Monitoring assay operation using monitoring tags

[0173] In some embodiments, the thickness of the sample is configured to be thin so that the objects of interest (eg, cells) form a monolayer (ie, there is no significant overlap between the objects in a direction perpendicular to the sample layer).

[0174] Likewise, a method of determining manufacturing quality of a QMAX card using an imager comprises:

[0175] (a) obtaining a device as in any preceding embodiment, wherein the device comprises two movable plates, a spacer and one or more monitoring marks, wherein the monitoring marks are in the sample contacting area;

[0176] (b) obtaining an imager;

[0177] (c) depositing a sample in the sample contact area of ​​the device of (a) and forcing the two plates into a closed configuration;

[0178] (d) taking one or more images of the thin sample layer using an imager; and

[0179] (e) Determining the manufacturing quality of the QMAX card using images of the monitoring marks.

[0180] Additionally, a method of determining manufacturing quality of a QMAX card using an imager, the method comprising:

[0181] (a) obtaining the device of the previous embodiment, wherein the device has two movable plates, a spacer, and one or more monitoring marks, wherein the monitoring marks are in the sample contact area;

[0182] (b) obtaining an imager;

[0183] (c) depositing a sample in the sample contact area of ​​the device of (a) and forcing the two plates into a closed configuration;

[0184] (d) taking one or more images of the thin sample layer using an imager and monitoring the markers; and

[0185] (e) Determining the manufacturing quality of the QMAX card using images of the monitoring marks.

[0186] A method as in any preceding embodiment, wherein determining the manufacturing quality comprises measuring characteristics (eg, length, width, spacing, thick edge) of one or more monitoring marks, and comparing the measured characteristics to reference values ​​to determine the manufacturing quality of the QMAX card.

[0187] A method as in any of the foregoing embodiments, wherein determining the manufacturing quality comprises measuring a first characteristic (e.g., amount, length, width, spacing, thick edge) of one or more first monitoring marks, and comparing the measured first characteristic with a second characteristic (e.g., number, length, width, spacing, thick edge) of one or more second monitoring marks to determine the manufacturing quality of the QMAX card.

[0188] The method of any preceding embodiment, wherein the determining is performed during analysis of a sample using the device of any preceding embodiment.

[0189] One aspect of the present invention is to perform an assay using a QMAX card having two movable plates, and monitoring marks placed in a thin sample can be used to monitor the operating conditions of the QMAX card. The operating conditions may include whether the sample is loaded correctly, whether the two plates are closed correctly, and whether the gap between the two plates is the same or approximately the same as a predetermined value.

[0190] In some embodiments, for a QMAX card comprising two movable plates and having a predetermined gap between the two plates in a closed configuration, the operating condition of a QMAX card-based assay is monitored by taking an image of a monitoring mark in the closed configuration. For example, if the two plates are not closed correctly, the monitoring mark will appear differently in the image than if the two plates are closed correctly. A monitoring mark surrounded by a sample (correctly closed) will have a different appearance than a monitoring mark not surrounded by a sample (not correctly closed). Thus, it can provide information about sample loading conditions. A device for monitoring the operating condition of a device using monitoring marks, comprising a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0191] i. The first plate and the second plate are movable relative to each other into different configurations;

[0192] ii. Each of the first plate and the second plate comprises an inner surface, the inner surface comprising a sample contact area for contacting a sample to be analyzed;

[0193] iii. one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate;

[0194] iv. The monitoring mark has at least one of (a) a predetermined and known size, and (b) a size observable by an imager;

[0195] v. The monitoring mark is a microstructure having at least one lateral linear dimension of 300um or less; and

[0196] vi. The monitoring mark is located inside the sample;

[0197] One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both plates;

[0198] wherein the other configuration is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying an imprecise compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of very uniform thickness and is substantially stationary relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is adjusted by the plates and the spacers; and

[0199] Wherein, after force is applied to bring the two plates into a closed configuration, the monitoring mark is imaged to determine (i) whether the two plates have reached the intended closed configuration, thereby adjusting the sample thickness to an approximately predetermined thickness, and / or (ii) whether the sample has been loaded as desired.

[0200] In some embodiments, images of the monitoring mark are used to determine whether the two plates have reached a desired closed configuration, wherein the sample is adjusted to have a thickness of approximately a predetermined thickness.

[0201] In some embodiments, images of monitoring markers are used to determine whether samples have been loaded as desired.

[0202] In some embodiments, the monitoring mark is imaged to determine whether the two plates have reached the intended closed configuration in which the sample thickness is adjusted to a predetermined thickness, and to determine whether the sample has been loaded as desired.

[0203] In some embodiments, the spacer is used as a monitoring marker.

[0204] In some embodiments, a system includes an apparatus and a computing device and a non-transitory computer-readable medium having instructions that, when executed, perform a determination process in an assay.

[0205] In some embodiments, a non-transitory computer readable medium having instructions that, when executed, perform a method comprising using one or more images of a thin sample layer and a monitoring mark to determine (i) whether the two plates have reached an intended closed configuration, thereby adjusting the sample thickness to approximately a predetermined thickness, or (ii) whether the sample has been loaded as desired.

[0206] In some embodiments, a system includes a non-transitory computer readable medium having instructions that, when executed, perform any of the methods of the present disclosure.

[0207] W-1. A method for monitoring an operating condition of a device using a monitoring tag, the method comprising:

[0208] a) obtaining a device as in any of the preceding embodiments, wherein the device comprises two movable plates, a spacer and one or more monitoring marks, wherein the monitoring marks are in the sample contacting area;

[0209] b) obtaining an imager;

[0210] c) depositing a sample in a sample contact area of ​​the apparatus of (a) and forcing the two plates into a closed configuration;

[0211] d) taking one or more images of the thin sample layer using an imager and monitoring the markers; and

[0212] e) using the image of the monitoring mark to determine (i) whether the two plates have reached the desired closed configuration, thereby adjusting the sample thickness to approximately a predetermined thickness, or (ii) whether the sample has been loaded as desired.

[0213] Some examples

[0214] veneer

[0215] AA-1.1 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0216] (a) a solid surface comprising a sample contacting area for contacting a thin sample having a thickness of 200 μm or less and comprising or suspected of comprising microfeatures; and

[0217] (b) one or more markers, wherein the markers:

[0218] I. Having sharp edges that (i) have a predetermined and known shape and size and (ii) are observable by an imager imaging the microfeatures;

[0219] II. A microstructure having at least one lateral linear dimension of 300 um or less; and

[0220] III. Located inside the sample;

[0221] Wherein at least one of the labels is imaged by an imager during an assay.

[0222] AA-1.2 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0223] (a) a solid surface comprising a sample contacting area for contacting a thin sample having a thickness of 200 μm or less and comprising or suspected of comprising microfeatures; and

[0224] (b) one or more markers, wherein the markers:

[0225] i. Contains protrusions or grooves from the solid surface

[0226] ii. having a sharp edge that (i) has a predetermined and known shape and size and (ii) can be observed by an imager imaging the microfeature;

[0227] iii. A microstructure having at least one lateral linear dimension of 300um or less; and

[0228] iv. Located inside the sample;

[0229] Wherein at least one of the labels is imaged by an imager during an assay.

[0230] Two plates with a constant spacing

[0231] AA-2.1 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0232] A first plate, a second plate and one or more monitoring markers, wherein:

[0233] i. Each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising or suspected of comprising microfeatures;

[0234] ii. At least a portion of the sample is defined by the first and second plates as a thin layer having a substantially constant thickness of 200 um or less;

[0235] iii. The monitoring mark has a sharp edge that (a) has a predetermined and known shape and size and (b) can be observed by an imager imaging the microfeature;

[0236] iv. The monitoring mark is a microstructure with at least one lateral linear dimension of 300um or less; and

[0237] v. The monitoring mark is located inside the sample;

[0238] Wherein at least one of the labels is imaged by an imager during an assay.

[0239] Two removable plates

[0240] AA-3 A device for measuring microfeatures in a sample using an imager, the device comprising:

[0241] a. A first plate, a second plate, a spacer and one or more monitoring marks, wherein:

[0242] ii. the first plate and the second plate are movable relative to each other into different configurations;

[0243] iii. each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising or suspected of comprising microfeatures;

[0244] iv. one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate,

[0245] v. The monitoring mark has a sharp edge that (a) has a predetermined and known shape and size and (b) can be observed by an imager imaging the microfeature;

[0246] vi. The monitoring mark is a microstructure with at least one lateral linear dimension of 300um or less; and

[0247] vii. The monitoring mark is located inside the sample;

[0248] wherein at least one of the labels is imaged by an imager during the assay;

[0249] One of the configurations is an open configuration in which the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both plates;

[0250] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying an imprecise compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stationary relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is adjusted by the plates and the spacers; and

[0251] The monitoring marker is (i) a different structure from the spacer, or (ii) the same structure as the spacer.

[0252] Improved image capture using a sample holder with micro-markings

[0253] BB-1. An apparatus for improving image capture of microfeatures in a sample, the apparatus comprising:

[0254] (c) an apparatus as in any preceding apparatus embodiment; and

[0255] (d) an imager for measuring a sample containing or suspected of containing microfeatures;

[0256] Wherein the imager captures images, and wherein at least one image includes both a portion of the sample and the monitoring.

[0257] CB-2. A system for improving image capture of microfeatures in a sample, the system comprising:

[0258] (a) A device as in any preceding device embodiment;

[0259] (b) an imager for measuring a sample containing or suspected of containing microfeatures; and

[0260] (c) An algorithm that uses the markers as parameters in conjunction with an imaging processing method to adjust the imager settings for the next image.

[0261] C. Imaging Analysis Using a Sample Holder with Micro-labeling

[0262] CC-1 An apparatus for improving analysis of images of microfeatures in a sample, the apparatus comprising:

[0263] (a) A device as in any preceding device embodiment;

[0264] (b) a computing device for receiving an image of a marker and a sample containing or suspected of containing microfeatures; and

[0265] Therein, the computing device runs an algorithm which utilizes the markers as parameters together with an imaging processing method to improve the image quality in the image.

[0266] CC-2. A system for improving analysis of an image of microfeatures in a sample, the system comprising:

[0267] (a) A device as in any preceding device embodiment;

[0268] (b) an imager for determining whether a sample contains or is suspected of containing a microfeature by taking one or more images of the sample and the marking; and

[0269] (c) An algorithm that utilizes the marker as a parameter together with an imaging processing method to improve the image quality of at least one image captured in (c).

[0270] CC-3 A computer program product for determining microfeatures in a sample, the program comprising computer program code applied and adapted to at least one image:

[0271] (a) receiving an image of a sample and a monitoring marker, wherein the sample is loaded into an apparatus according to any preceding apparatus claim, wherein the image is captured by an imager; and

[0272] (b) processing and analyzing the image to calculate the amount of the microfeature, wherein the analysis uses a machine learning-based detection model and information provided by the image of the monitoring mark.

[0273] CC-4 A computing device for analyzing microfeatures in a sample, the computing device comprising a computing device that operates the algorithm in any embodiment of the present invention.

[0274] CC-5 A method, apparatus, computer program product, or system as in any preceding embodiment, wherein the improvement of image quality comprises at least one selected from the group consisting of: denoising, image normalization, image sharpening, image scaling, alignment, super-resolution, deblurring, and any combination thereof.

[0275] CC-6 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the imaging processing method comprises at least one selected from the group consisting of: histogram-based operations, mathematical-based operations, convolution-based operations, smoothing operations, derivative-based operations, morphology-based operations, shading correction, image enhancement and / or restoration, segmentation, feature extraction and / or matching, object detection and / or classification and / or localization, image understanding and any combination thereof.

[0276] CC-6.1 A method, apparatus, computer program product, or system as in any of the preceding embodiments, wherein the histogram-based operation comprises at least one selected from the group consisting of: contrast stretching, equalization, minimum filtering, median filtering, maximum filtering, and any combination thereof.

[0277] CC-6.2 A method, apparatus, computer program product, or system as in any of the foregoing embodiments, wherein the mathematics-based operation comprises at least one selected from the group consisting of: binary operations (e.g., NOT, OR, AND, XOR, and SUB), arithmetic-based operations (e.g., ADD, SUB, MUL, DIV, LOG, EXP, SQRT, TRIG, and INVERT), and any combination thereof.

[0278] CC-6.3 A method, apparatus, computer program product, or system as in any preceding embodiment, wherein the convolution-based operation comprises at least one selected from the group consisting of: an operation in the spatial domain, a Fourier transform, a DCT, an integer transform, an operation in the frequency domain, and any combination thereof.

[0279] CC-6.4 A method, apparatus, computer program product, or system as in any of the preceding embodiments, wherein the smoothing operation comprises at least one selected from the group consisting of linear filtering, uniform filtering, triangular filtering, Gaussian filtering, nonlinear filtering, median filtering, kuwahara filtering, and any combination thereof.

[0280] CC-6.5 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the derivative-based operation is selected from at least one of the group consisting of: first-order derivative operation, gradient filtering, basic derivative filtering, Prewitt gradient filtering, Sobel gradient filtering, alternative gradient filtering, Gaussian gradient filtering, second-order derivative filtering, basic second-order derivative filtering, frequency domain Laplacian, Gaussian second-order derivative filtering, alternative Laplacian filtering, gradient-directed second-order derivative (SDGD) filtering, third-order derivative filtering, higher-order derivative filtering (e.g., filtering greater than third-order derivative) and any combination thereof.

[0281] CC-6.6 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the morphological-based operation comprises at least one selected from the group consisting of dilation, erosion, Boolean convolution, opening and / or closing, hit and miss operations, contours, skeletons, propagation, gray value morphological processing, grayscale dilation, grayscale erosion, grayscale opening, grayscale closing, morphological smoothing, morphological gradient, morphological Laplacian and any combination thereof.

[0282] CC-6.7 A method, apparatus, computer program product, or system as in any preceding embodiment, wherein image enhancement and / or restoration comprises at least one selected from the group consisting of sharpening, unsharpening, noise suppression, distortion suppression, and any combination thereof.

[0283] CC-6.8 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the segmentation comprises at least one selected from the group consisting of: thresholding, fixed thresholding, histogram-derived thresholding, Isodata algorithm, background symmetry algorithm, triangle algorithm, edge finding, gradient-based process, zero-crossing-based process, PLUS-based process, binary mathematical morphology, salt and pepper filtering, separation of objects with holes, filling holes in objects, removal of boundary touching objects, exoskeleton, touching objects, gray value mathematical morphology, Top-hat transform, thresholding, local contrast stretching and any combination thereof.

[0284] CC-6.9 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the feature extraction and / or matching comprises at least one selected from the group consisting of independent component analysis, isometric mapping, kernel principal component analysis, latent semantic analysis, partial least squares, principal component analysis, multifactor dimensionality reduction, nonlinear dimensionality reduction, multilinear principal component analysis, multilinear subspace learning, semidefinite embedding, autoencoder and any group thereof.

[0285] A. Sample holder with micro markings

[0286] veneer

[0287] AA-1.1 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0288] (a) a solid surface comprising a sample contacting area for contacting a thin sample having a thickness of 200 μm or less and comprising or suspected of comprising microfeatures; and

[0289] (b) one or more markers, wherein the markers:

[0290] i. having a sharp edge that (i) has a predetermined and known shape and size and (ii) can be observed by an imager imaging the microfeature;

[0291] ii. a microstructure having at least one lateral linear dimension of 300um or less; and

[0292] iii. is located inside the sample; and

[0293] Wherein at least one of the labels is imaged by an imager during an assay.

[0294] AA-1.2 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0295] (a) a solid surface comprising a sample contacting area for contacting a thin sample having a thickness of 200 μm or less and comprising or suspected of comprising microfeatures; and

[0296] (b) one or more markers, wherein the markers:

[0297] i. Contains protrusions or grooves from the solid surface

[0298] ii. having a sharp edge that (i) has a predetermined and known shape and size and (ii) can be observed by an imager imaging the microfeature;

[0299] iii. A microstructure having at least one lateral linear dimension of 300um or less; and

[0300] iv. Located inside the sample;

[0301] Wherein at least one of the labels is imaged by an imager during an assay.

[0302] Two plates with a constant spacing

[0303] AA-2.1 An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0304] A first plate, a second plate and one or more monitoring markers, wherein:

[0305] i. Each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising or suspected of comprising microfeatures;

[0306] ii. At least a portion of the sample is defined by the first and second plates as a thin layer having a substantially constant thickness of 200 um or less;

[0307] iii. The monitoring mark has a sharp edge that (a) has a predetermined and known shape and size and (b) can be observed by an imager imaging the microfeature;

[0308] iv. The monitoring mark is a microstructure with at least one lateral linear dimension of 300um or less; and

[0309] v. The monitoring mark is located inside the sample;

[0310] Wherein at least one of the labels is imaged by an imager during an assay.

[0311] Two removable plates

[0312] AA-3 A device for measuring microfeatures in a sample using an imager, the device comprising:

[0313] a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0314] I. The first plate and the second plate can be moved relative to each other into different configurations;

[0315] i. Each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising or suspected of comprising microfeatures;

[0316] ii. one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate,

[0317] iii. The monitoring mark has a sharp edge that (a) has a predetermined and known shape and size and (b) can be observed by an imager imaging the microfeature;

[0318] iv. The monitoring mark is a microstructure with at least one lateral linear dimension of 300um or less; and

[0319] v. The monitoring mark is located inside the sample;

[0320] Wherein at least one of the labels is imaged by an imager during an assay.

[0321] One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both of the plates;

[0322] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying an imprecise compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stationary relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is adjusted by the plates and the spacers; and

[0323] Wherein the monitoring marker is (a) a different structure from the spacer, or (b) the same structure as the spacer.

[0324] B. Improved image capture using a sample holder with micro-markings

[0325] BB-1. An apparatus for improving image capture of microfeatures in a sample, the apparatus comprising:

[0326] (e) A device as in any preceding device embodiment; and

[0327] (f) an imager for measuring a sample containing or suspected of containing microfeatures; and

[0328] Wherein the imager captures images, wherein at least one image includes both a portion of the sample and the monitoring.

[0329] CB-2. A system for improving image capture of microfeatures in a sample, the system comprising:

[0330] i. A device as in any of the preceding device embodiments;

[0331] ii. an imager for determining whether a sample contains or is suspected of containing microfeatures; and

[0332] iii. A non-transitory computer readable medium having instructions which when executed use the flag as a parameter in conjunction with an imaging processing method to adjust settings of the imager for the next image.

[0333] C. Imaging Analysis Using a Sample Holder with Micro-labeling

[0334] CC-1 An apparatus for improving analysis of images of microfeatures in a sample, the apparatus comprising:

[0335] a) An apparatus as in any preceding apparatus embodiment; and

[0336] b) a computing device for receiving an image of a marker and a sample containing or suspected of containing a microfeature;

[0337] Therein, the computing device runs an algorithm which utilizes the markers as parameters together with an imaging processing method to improve the image quality in the image.

[0338] CC-2. A system for improving analysis of an image of microfeatures in a sample, the system comprising:

[0339] (a) A device as in any preceding device embodiment;

[0340] (b) an imager for determining whether a sample contains or is suspected of containing a microfeature by taking one or more images of the sample and the marking; and

[0341] (c) A non-transitory computer readable medium having instructions that, when executed, utilize the marker as a parameter in conjunction with an imaging processing method to improve the image quality of at least one image captured in (c).

[0342] CC-3 A computer program product for determining microfeatures in a sample, the program comprising computer program code means applied and adapted to at least one image:

[0343] (a) receiving an image of a sample and a monitoring marker, wherein the sample is loaded into an apparatus according to any preceding apparatus claim, and wherein the image is captured by an imager; and

[0344] (b) processing and analyzing the image to calculate the amount of the microfeature, wherein the analysis uses a machine learning-based detection model and information provided by the image of the monitoring mark.

[0345] CC-4 A computing device for analyzing microfeatures in a sample, the computing device comprising a computing device that operates the algorithm in any embodiment of the present invention.

[0346] CC-5 A method, apparatus, computer program product, or system as in any of the foregoing embodiments, wherein the improvement in image quality comprises at least one selected from the group consisting of: denoising, image normalization, image sharpening, image scaling, alignment (e.g., for face detection), super-resolution, deblurring, and any combination thereof.

[0347] CC-6 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the imaging processing method comprises at least one selected from the group consisting of: histogram-based operations, mathematical-based operations, convolution-based operations, smoothing operations, derivative-based operations, morphology-based operations, shading correction, image enhancement and / or restoration, segmentation, feature extraction and / or matching, object detection and / or classification and / or localization, image understanding and any combination thereof.

[0348] CC-6.1 A method, apparatus, computer program product, or system as in any of the preceding embodiments, wherein the histogram-based operation comprises at least one selected from the group consisting of: contrast stretching, equalization, minimum filtering, median filtering, maximum filtering, and any combination thereof.

[0349] CC-6.2 A method, apparatus, computer program product, or system as in any of the foregoing embodiments, wherein the mathematics-based operation comprises at least one selected from the group consisting of: binary operations (e.g., NOT, OR, AND, XOR, and SUB), arithmetic-based operations (e.g., ADD, SUB, MUL, DIV, LOG, EXP, SQRT, TRIG, and INVERT), and any combination thereof.

[0350] CC-6.3 A method, apparatus, computer program product, or system as in any preceding embodiment, wherein the convolution-based operation comprises at least one selected from the group consisting of: an operation in the spatial domain, a Fourier transform, a DCT, an integer transform, an operation in the frequency domain, and any combination thereof.

[0351] CC-6.4 A method, apparatus, computer program product, or system as in any of the preceding embodiments, wherein the smoothing operation comprises at least one selected from the group consisting of linear filtering, uniform filtering, triangular filtering, Gaussian filtering, nonlinear filtering, median filtering, kuwahara-filtering, and any combination thereof.

[0352] CC-6.5 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the derivative-based operation is selected from at least one of the group consisting of: first-order derivative operation, gradient filtering, basic derivative filtering, Prewitt gradient filtering, Sobel gradient filtering, alternative gradient filtering, Gaussian gradient filtering, second-order derivative filtering, basic second-order derivative filtering, frequency domain Laplacian, Gaussian second-order derivative filtering, alternative Laplacian filtering, gradient-directed second-order derivative (SDGD) filtering, third-order derivative filtering, higher-order derivative filtering (e.g., filtering greater than third-order derivative) and any combination thereof.

[0353] CC-6.6 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the morphological-based operation comprises at least one selected from the group consisting of dilation, erosion, Boolean convolution, opening and / or closing, hit and miss operations, contours, skeletons, propagation, gray value morphological processing, grayscale dilation, grayscale erosion, grayscale opening, grayscale closing, morphological smoothing, morphological gradient, morphological Laplacian and any combination thereof.

[0354] CC-6.7 A method, apparatus, computer program product, or system as in any preceding embodiment, wherein image enhancement and / or restoration comprises at least one selected from the group consisting of: sharpening, noise suppression, distortion suppression, and any combination thereof.

[0355] CC-6.8 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the segmentation comprises at least one selected from the group consisting of: thresholding, fixed thresholding, histogram-derived thresholding, ISO data algorithm, background symmetry algorithm, triangle algorithm, edge finding, gradient-based process, zero-crossing-based process, PLUS-based process, binary mathematical morphology, salt and pepper filtering, separation of objects with holes, filling holes in objects, removal of boundary touching objects, exoskeleton, touching objects, gray value mathematical morphology, Top-hat transform, thresholding, local contrast stretching and any combination thereof.

[0356] CC-6.9 A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the feature extraction and / or matching comprises at least one selected from the group consisting of independent component analysis, isometric mapping, kernel principal component analysis, latent semantic analysis, partial least squares, principal component analysis, multifactor dimensionality reduction, nonlinear dimensionality reduction, multilinear principal component analysis, multilinear subspace learning, semidefinite embedding, autoencoder and any group thereof.

[0357] T1. A method for determining a true lateral dimension (TLD) of a sample on a sample holder from a distorted image, the method comprising:

[0358] (a) obtaining a device as in any preceding embodiment, wherein the device comprises one or more monitoring markers in the sample contact area;

[0359] (b) obtaining an imager, computing hardware, and a non-transitory computer-readable medium containing an algorithm;

[0360] (c) depositing a thin sample layer comprising microfeatures in a sample contact region of the device of (a);

[0361] (d) taking one or more images of the thin sample layer and monitoring the markers using an imager, wherein the imager is located above the thin sample layer; and

[0362] (e) using an algorithm to determine the true lateral dimensions of the sample;

[0363] in

[0364] (i) the algorithm is a computer code executed on a computer system; and

[0365] (ii) The algorithm uses images of monitoring markers as parameters.

[0366] T2. A method, apparatus, computer program product or system as in any preceding embodiment, wherein each monitoring mark comprises a protrusion or groove from a solid surface.

[0367] T3. The method, apparatus, computer program product, or system of any preceding embodiment, wherein the microstructures do not have sharp edges.

[0368] T4. The method, apparatus, computer program product, or system of any preceding embodiment, wherein the sample comprises a sample selected from the group consisting of: a biological sample, a chemical sample, and a sample without sharp edges.

[0369] T5. A method, apparatus, computer program product or system as in any of the foregoing embodiments, wherein the monitoring marker is used together with an imaging processing method as a parameter in an algorithm that (i) adjusts an image, (ii) processes an image of a sample, (iii) determines a property associated with a microfeature, or (iv) any combination of the foregoing.

[0370] T6. The method, apparatus, computer program product, or system of any preceding embodiment, wherein the spacers have a substantially uniform height and a fixed separation distance (ISD) equal to or less than 200 microns;

[0371] AA. A method and apparatus for improving imaging of thin layers of samples.

[0372] AA1. A method for improving imaging of a thin layer of a sample, the method comprising:

[0373] (a) obtaining a labeled sample holder, wherein the sample comprises one or more monitoring labels in the sample contact area;

[0374] (b) obtaining an imager, computing hardware, and a non-transitory computer-readable medium containing an algorithm;

[0375] (c) depositing a thin sample layer comprising microfeatures in a sample contact region of the device of (a);

[0376] (d) taking one or more images of the thin sample layer and monitoring the markers using an imager, wherein the imager is located above the thin sample layer; and

[0377] (e) using an algorithm to determine the true lateral dimensions of the sample;

[0378] in:

[0379] (i) the algorithm is a computer code executed on a computer system; and

[0380] (ii) The algorithm uses images of monitoring markers as parameters.

[0381] A-1

[0382] T1. A method for determining a true lateral dimension (TLD) of a sample on a sample holder from a distorted image, the method comprising:

[0383] i. obtaining a device as in any of the preceding embodiments, wherein the device comprises one or more monitoring markers in the sample contact area;

[0384] ii. Obtaining an imager, computing hardware, and a non-transitory computer-readable medium containing an algorithm;

[0385] iii. depositing a thin sample layer comprising microfeatures in a sample contact region of the device of (a);

[0386] iv. taking one or more images of the thin sample layer and monitoring the markers using an imager, wherein the imager is located above the thin sample layer; and

[0387] v. Determine the true lateral dimensions of the sample using an algorithm;

[0388] in

[0389] (a) the algorithm is a computer code executed on a computer system; and

[0390] (b) The algorithm uses images of surveillance markers as parameters.

[0391] NN1. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0392] (a) a solid surface comprising a sample contact area for contacting a sample comprising microfeatures; and

[0393] (b) one or more monitoring markers, wherein the monitoring markers:

[0394] i. Made of a different material from the sample;

[0395] ii. being located inside the sample during determination of the microstructure, wherein the sample forms a thin layer having a thickness of less than 200 um on the sample contact area;

[0396] iii. having a lateral linear dimension of about 1 um (micrometer) or more, and

[0397] iv. having at least one lateral linear dimension of 300um or less; and

[0398] During the determination process, the imager images at least one monitoring marker.

[0399] wherein is used during the determination of the analyte; and the geometric parameters (e.g., shape and size) of the monitoring marks and / or the spacing between the monitoring marks are (a) predetermined and known prior to the determination of the analyte, and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0400] NN2. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0401] a solid surface comprising a sample contact area for contacting a sample comprising microfeatures; and

[0402] One or more monitoring marks, wherein each monitoring mark comprises a protrusion or groove from the solid surface, wherein:

[0403] (a) the protrusion or groove comprises a flat surface that is substantially parallel to an adjacent surface, which is a portion of the solid surface adjacent to the protrusion or groove;

[0404] (b) the distance between the planar surface and the adjacent surface is about 200 micrometers (um) or less;

[0405] (c) a planar surface having (a) a linear dimension of at least about 1 μm or more, and (b) at least one region having a linear dimension of 150 μm or less;

[0406] (d) the planar surface of at least one monitoring mark is imaged by an imager used in measuring the microfeatures; and

[0407] (e) The shape of the planar surface, the size of the planar surface, the distance between the planar surface and an adjacent surface, and / or the spacing between monitoring marks are (a) predetermined and known prior to measurement of the microfeatures and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0408] NN3. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0409] a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0410] (a) the first plate and the second plate are movable relative to each other into different configurations;

[0411] (b) each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising microfeatures;

[0412] (c) one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate,

[0413] (d) the spacers have a substantially uniform height and a fixed spacing distance (ISD) equal to or less than 200 microns;

[0414] (e) The monitoring mark is made of a different material from the sample;

[0415] (f) monitoring the marker being located inside the sample during determination of the microstructure, wherein the sample forms a thin layer having a thickness of less than 200 μm on the sample contact area; and

[0416] (g) the monitoring mark has a lateral linear dimension of about 1 um (micrometer) or more and has at least one lateral linear dimension of 300 um or less;

[0417] During the determination process, the imager images at least one monitoring marker.

[0418] wherein the shape of the planar surface, the size of the planar surface, the distance between the planar surface and an adjacent surface, and / or the spacing between monitoring marks are (a) predetermined and known prior to the determination of the microfeatures, and (b) used as parameters in an algorithm for determining a property associated with the microfeatures;

[0419] One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not regulated by a spacer, and the sample is deposited on one or both of the plates;

[0420] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying an imprecise compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stationary relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is adjusted by the plates and the spacers; and

[0421] The monitoring marker is (i) a different structure from the spacer, or (ii) the same structure as the spacer.

[0422] NN4. An apparatus for determining microfeatures in a sample using an imager, the apparatus comprising:

[0423] a first plate, a second plate, a spacer, and one or more monitoring marks, wherein:

[0424] (a) the first plate and the second plate are movable relative to each other into different configurations;

[0425] (b) each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample comprising microfeatures;

[0426] (c) one or both of the first plate and the second plate comprises a spacer permanently fixed to the inner surface of the respective plate,

[0427] (d) the spacers have a substantially uniform height and a fixed spacing distance (ISD) equal to or less than 200 microns;

[0428] (e) each monitoring mark comprises a protrusion or a groove on one or both sample contact areas;

[0429] (f) the protrusion or groove comprises a flat surface that is substantially parallel to an adjacent surface, which is a portion of the solid surface adjacent to the protrusion or groove;

[0430] (g) the distance between the planar surface and the adjacent surface is about 200 micrometers (um) or less;

[0431] (h) a planar surface having (a) a linear dimension of at least about 1 μm or more, and (b) at least one region having a linear dimension of 150 μm or less;

[0432] (i) a planar surface of at least one monitoring mark is imaged by an imager used in measuring microfeatures; and

[0433] (j) The shape of the planar surface, the size of the planar surface, the distance between the planar surface and an adjacent surface, and / or the spacing between monitoring marks are (a) predetermined and known prior to measurement of the microfeatures and (b) used as parameters in an algorithm for determining properties associated with the microfeatures.

[0434] a. One of the configurations is an open configuration in which: the two plates are partially or completely separated, the spacing between the plates is not adjusted by a spacer, and the sample is deposited on one or both of the plates;

[0435] wherein the other of the configurations is a closed configuration, the closed configuration being configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying an imprecise compressive force on the force-bearing area; and in the closed configuration: at least a portion of the sample is compressed by the two plates into a layer of substantially uniform thickness and is substantially stationary relative to the plates, wherein the uniform thickness of the layer is defined by the sample contact area of ​​the two plates and is adjusted by the plates and the spacers; and

[0436] The monitoring marker is (i) a different structure from the spacer, or (ii) the same structure as the spacer.

[0437] NN5. An apparatus for imaging-based assays, comprising:

[0438] The device of any preceding device embodiment, wherein the device has at least four monitoring marks that are not aligned on a linear line.

[0439] NN6. An apparatus for determining microfeatures in a sample using an imager, the system comprising:

[0440] (a) an apparatus as in any preceding apparatus embodiment; and

[0441] (b) Imager for measuring samples containing microfeatures.

[0442] NN7. A system for performing an imaging-based assay, the system comprising:

[0443] (a) A device as in any preceding device embodiment;

[0444] (b) an imager for measuring a sample comprising microfeatures; and

[0445] (c) A non-transitory computer-readable medium containing instructions that, when executed, utilize monitoring signatures of the device to determine a characteristic associated with the micro-feature.

[0446] NN8. A system for determining microfeatures in a sample using an imager, the system comprising:

[0447] (a) A device as in any preceding device embodiment;

[0448] (b) an imager for measuring a sample comprising microfeatures; and

[0449] (c) a non-transitory computer-readable medium comprising instructions that, when executed, utilize monitoring signatures of the device to determine a characteristic associated with the micro-feature, wherein the instructions comprise machine learning.

[0450] NN9. A method for measuring microfeatures in a sample using an imager, comprising:

[0451] (a) obtaining a device, apparatus or system as described in any of the preceding embodiments;

[0452] (b) obtaining a sample and depositing the sample on a sample contact area in the device, apparatus, or system of (a), wherein the sample comprises microfeatures; and

[0453] (c) Determination of microfeatures.

[0454] NN10. A method for measuring microfeatures in a sample using an imager, comprising:

[0455] (a) obtaining a device, apparatus or system as described in any of the preceding embodiments;

[0456] (b) obtaining a sample and depositing the sample on a sample contact area in the device, apparatus, or system of (a), wherein the sample comprises microfeatures;

[0457] (c) determining the microfeatures, wherein the determining comprises the step of using machine learning.

[0458] T1. A method for determining a true lateral dimension (TLD) of a sample on a sample holder from a distorted image, the method comprising:

[0459] (a) obtaining a device as in any preceding embodiment, wherein the device comprises one or more monitoring markers in the sample contact area;

[0460] (b) obtaining an imager, computing hardware, and a non-transitory computer-readable medium containing an algorithm;

[0461] (c) depositing a thin sample layer comprising microfeatures in a sample contact region of the device of (a);

[0462] (d) taking one or more images of the thin sample layer and monitoring the markers using an imager, wherein the imager is located above the thin sample layer; and

[0463] (e) using an algorithm to determine the true lateral dimensions of the sample;

[0464] in

[0465] (i) the algorithm is a computer code executed on a computer system; and

[0466] (ii) The algorithm uses images of monitoring markers as parameters.

[0467] T2. A method for determining a true lateral dimension (TLD) of a sample on a sample holder from a distorted image, the method comprising:

[0468] (a) obtaining a device as in any preceding embodiment, wherein the device comprises one or more monitoring markers in the sample contact area;

[0469] (b) obtaining an imager, computing hardware, and a non-transitory computer-readable medium containing an algorithm;

[0470] (c) depositing a thin sample layer comprising microfeatures in a sample contact region of the device of (a);

[0471] (d) taking one or more images of the thin sample layer and monitoring the markers using an imager, wherein the imager is located above the thin sample layer; and

[0472] (e) using the algorithm to determine the real lateral dimensions and coordinates of the imaged sample in the real world by physical measurements (e.g., micrometers);

[0473] in

[0474] (i) the algorithm is a computer code executed on a computer system; and

[0475] (ii) The algorithm uses images of monitoring markers as parameters.

[0476] T3. A device, method or system as in any preceding embodiment, wherein the microfeatures and monitoring marks from the sample are disposed within a sample holding device.

[0477] T4. An apparatus, method or system as in any preceding embodiment, wherein the determining comprises detecting and locating the monitoring marker in an image of the sample taken by the imager.

[0478] T5. An apparatus, method or system as in any preceding embodiment, wherein the determining comprises generating a monitoring mark grid based on the monitoring marks detected from an image of the sample taken by the imager.

[0479] T6. An apparatus, method or system as in any preceding embodiment, wherein the determining comprises computing a homography transformation from the generated monitoring marker grid.

[0480] T7. An apparatus, method or system as in any preceding embodiment, wherein the determining comprises estimating the TLD from the homography transform, and determining the area, size and concentration of microfeatures detected in the image-based determination.

[0481] T8. A method, apparatus or system as in any preceding embodiment, wherein the TLD estimation is based on a region in a sample image captured by the imager, comprising:

[0482] (a) obtaining a sample;

[0483] (b) loading the sample into a sample holding device, such as a QMAX device, wherein monitoring markers are present, wherein the monitoring markers are not immersed in the sample and are located in the device where they can be imaged from the top by an imager in the image-based assay;

[0484] (c) capturing an image of the sample in the sample loading device including the microfeatures and monitoring marks;

[0485] (d) detecting monitoring marks in a sample image captured by an imager;

[0486] (e) dividing the sample image into non-overlapping regions;

[0487] (f) generating a region-based marker grid for each non-overlapping region where more than 4 non-collinear monitoring markers are detected in the local region;

[0488] (g) generating a marking grid for all other regions not in (f) based on the monitoring marks detected from the sample image captured by the imager;

[0489] (h) computes a region-specific homography transform for each region in (f) based on its own region-based labeling grid generated by (f);

[0490] (i) Compute the homography transformation for all other regions not in (f) based on the labeled grid generated in (g);

[0491] (j) Based on the region-based homography transformation in (g), the region-based TLD of each region is estimated in (f);

[0492] (k) estimating TLDs of other regions not included in (f) based on the homography transformation of (i); and

[0493] (l) Application of the estimated TLDs from (j) and (k) to determine the area and concentration of microfeatures imaged in each image partition in an image-based assay.

[0494] T9. A method, device or system as in any preceding embodiment, wherein the monitoring marks in the sample holding device are distributed according to a periodic pattern with a defined pitch period.

[0495] T10. The method, device or system of any preceding embodiment, wherein the monitoring marker is detected and used as a detectable anchor to calibrate and improve measurement accuracy in image-based assays.

[0496] T11. A method, device or system as in any of the preceding embodiments, wherein detection of monitoring marks in a sample image captured by an imager utilizes the periodicity of the distribution of monitoring marks in a sample holding device for error correction and / or reliability of detection.

[0497] T12. A method, device or system as in any of the foregoing embodiments, wherein the detection, identification, area and / or shape contour estimation of the monitoring mark in the image-based measurement is performed by machine learning (ML), the machine learning having an ML-based monitoring mark detection model and a device constructed or trained from images captured by an imager on the device in the image-based measurement.

[0498] T13. A method, device or system as in any of the preceding embodiments, wherein the detection, identification, area and / or shape contour estimation of the monitoring marker in the image-based assay is by image processing or image processing combined with machine learning.

[0499] T14. The method, apparatus or system of any preceding embodiment, wherein the detected monitoring marker is applied to TLD estimation in an image-based assay to calibrate the system and / or improve measurement accuracy in an imaging-based assay.

[0500] T15. A method, apparatus or system as in any preceding embodiment, wherein the detected monitoring markers are applied in image-based assays without limitation to microfeature size, volume and / or concentration estimation to calibrate the system and / or improve measurement accuracy.

[0501] T16. A method, device or system as in any of the foregoing embodiments, wherein detection of monitoring marks and / or TLD estimation is applied to fault detection in image-based assays, including but not limited to detecting defects in a sample holding device, misalignment of the sample holding device in an imager and / or focusing failure of the imager.

[0502] T17. A method, apparatus or system as in any preceding embodiment, wherein the monitoring marker is detected as an anchor for use in a system to estimate the area of ​​an object in an image-based measurement, comprising:

[0503] i. loading the sample into a sample holding device having a monitoring marker located in the device in an image-based assay;

[0504] ii. capturing an image of the sample in the sample holding device including micro-features and monitoring marks; and

[0505] iii. Detect monitoring marks in an image of the sample taken by an imager on a sample holding device, determine the TLD and calculate an area estimate in an image-based assay to determine the size of the imaged object from pixels in the image to its physical size in microns in the real world.

[0506] T18. A method, apparatus or system as in any preceding embodiment, wherein the system comprises:

[0507] i. Detect monitoring marks in digital images;

[0508] ii. Generate monitoring mark grid;

[0509] iii. Computing the image transformation based on the monitoring marker grid; and

[0510] iv. Estimating the area of ​​objects in an image of a sample and their physical size in the real world in image-based assays.

[0511] T19. A method, apparatus or system as in any of the foregoing embodiments, wherein a monitoring marker grid generated from detected monitoring markers is used to calculate a homography transformation to estimate the TLD, the area of ​​an object in an image of a sample taken by an imager, and the physical size of the object in the real world.

[0512] T20. A method, apparatus or system as in any preceding embodiment, wherein the method comprises:

[0513] i. dividing an image of a sample captured by an imager in an image-based assay into non-overlapping regions;

[0514] ii. Detect local monitoring markers in the image;

[0515] iii. If more than 4 non-collinear monitoring markers are detected in the region, a region-based marker grid is generated for the region;

[0516] iv. generating a marking grid for all other regions based on the monitoring marks detected in the image of the sample taken by the imager;

[0517] v. computing a region-based homography from the region-based labeled grid generated for each region in (iii);

[0518] vi. Compute the homography transform for all other regions not in (iii) based on the label grid generated in (iv); and

[0519] vii. Estimate the TLD of each region based on the homography transformation generated from (v) and (vi), and then determine the area of ​​objects in the image of the sample and their real-world size in an image-based assay for each region.

[0520] T21. The method, device or system of any preceding embodiment, wherein the assay is a medical, diagnostic, chemical or biological test.

[0521] T22. The method, apparatus or system of any preceding embodiment, wherein the microfeatures are cells.

[0522] T23. The method, device or system of any preceding embodiment, wherein the microfeatures are blood cells.

[0523] T24. A method, device or system as in any preceding embodiment, wherein the microfeature is a protein, a peptide, a DNA, an RNA, a nucleic acid, a small molecule, a cell or a nanoparticle.

[0524] T25. A method, device or system as in any preceding embodiment, wherein the microcomponent comprises a tag.

[0525] T26. A method, apparatus or system as in any preceding embodiment, wherein the algorithm comprises a computer program product comprising computer program code means adapted to:

[0526] (a) receiving an image of a sample, wherein the sample is loaded into a QMAX device and the image is captured by an imager connected to the QMAX device, wherein the image includes both the sample and a monitoring marker; and

[0527] (b) analyzing the image using a detection model and generating a two-dimensional data array of the image, wherein the two-dimensional data array includes probability data of the micro-feature at each position in the image, the detection model being established through a training process, the training process comprising:

[0528] i. feeding the annotated dataset into the convolutional neural network, wherein the annotated dataset comes from a sample of the same type and for the same micro-features as the test sample; and

[0529] ii. Training and building detection models through convolution; and

[0530] (c) Analyze the two-dimensional data array to detect local signal peaks by:

[0531] i. Signal list process, or

[0532] ii. Local search process; and

[0533] (d) Calculate the amount of micro-features based on the local signal peak information.

[0534] T27. A method, apparatus or system as in any preceding embodiment, wherein the algorithm comprises a computer program product comprising computer program code means for applying and being adapted in at least one image:

[0535] (a) an inferred pattern between an object in the sample and a contour map of pixels of said object in an image of the sample taken by an imager on the sample holding device,

[0536] (b) digitally reconstructing an image of at least one object detected from the inference module in the image of the sample and generating a contour mask surrounding the object identified by the inference module, wherein the object is in focus,

[0537] (c) identifying at least a portion of the sample image for at least one object in a selected portion of the sample image, and

[0538] (d) calculating at least one feature of the object from the at least one portion to identify the object in a selected portion of an image of the sample taken by the imager,

[0539] (e) calculating the count of the detection object and its concentration in the selected portion of the image of the sample,

[0540] When the program is executed on a computing device, or in a computing cloud connected via a network.

[0541] T28. A method, apparatus or system as in any preceding embodiment, wherein the algorithm comprises a computer program product comprising computer program code means adapted to:

[0542] (a) receiving an image of a sample, wherein the sample is loaded into a QMAX device and the image is captured by an imager connected to the QMAX device, wherein the image includes both the sample and a monitoring marker; and

[0543] (b) analyzing the image to calculate the amount of the microfeature, wherein the analyzing uses a machine learning-based detection model and information provided by the monitored labeled image.

[0544] The method, apparatus or system of any preceding embodiment further comprises a computer readable storage medium or memory storage unit comprising a computer program of any preceding embodiment.

[0545] The method, device or system as in any of the foregoing embodiments further comprises a computing device or a mobile device comprising a computing device as in any of the foregoing embodiments.

[0546] The method, apparatus or system of any preceding embodiment further comprises a computing device or mobile device comprising a computer program product of any preceding embodiment.

[0547] The method, apparatus or system of any of the foregoing embodiments further comprises a computing device or mobile device comprising a computer-readable storage medium or storage unit of any of the foregoing embodiments.

[0548] A system for analyzing a sample, comprising:

[0549] a first plate, a second plate, a surface amplification layer, and a capture agent, wherein

[0550] (a) the first plate and the second plate are movable relative to each other into different configurations and have sample contact areas on their respective inner surfaces for contacting a sample containing a target analyte;

[0551] (b) Surface amplification layer on a sample contact area,

[0552] (c) a capture agent is immobilized on the surface amplification layer, wherein the capture agent specifically binds to the target analyte,

[0553] wherein the surface amplification layer amplifies an optical signal from the target analyte or a label attached to the target analyte when the surface amplification layer is adjacent to the surface amplification layer, the optical signal being much stronger than when the surface amplification layer is micrometers or more away,

[0554] One of the configurations is an open configuration in which the average spacing between the inner surfaces of the two plates is at least 200 um; and

[0555] Another configuration is a closed configuration, wherein at least a portion of the sample is located between the two plates and the average spacing between the inner surfaces of the plates is less than 200 um.

[0556] A system for analyzing a sample, comprising:

[0557] a first plate, a second plate, a surface amplification layer, and a capture agent, wherein

[0558] (a) the first plate and the second plate are movable relative to each other into different configurations and have sample contact areas on their respective inner surfaces for contacting a sample containing a target analyte;

[0559] (b) Surface amplification layer on a sample contact area,

[0560] (c) a capture agent is immobilized on the surface amplification layer, wherein the capture agent specifically binds to the target analyte,

[0561] wherein the surface amplification layer amplifies an optical signal from a label attached to the target analyte when the surface amplification layer is adjacent to the surface amplification layer, the optical signal being much stronger than when the surface amplification layer is micrometers or more away,

[0562] One of the configurations is an open configuration in which the average spacing between the inner surfaces of the two plates is at least 200 um;

[0563] wherein another configuration is a closed configuration, wherein at least a portion of the sample is located between the two plates and the average spacing between the inner surfaces of the plates is less than 200 um;

[0564] Wherein the thickness of the sample in the closed configuration, the concentration of the label dissolved in the sample in the closed configuration, and the magnification factor of the surface magnification layer are configured to make any label directly or indirectly bound to the capture agent in the closed configuration visible without washing away unbound label.

[0565] An apparatus comprising a device as in any preceding embodiment and a reader for reading the device.

[0566] A homogeneous assay using a device as in any of the preceding embodiments, wherein the thickness of the sample in the closed configuration, the concentration of the label, and the magnification factor of the amplification surface are configured to visualize label bound to the amplification surface without washing away unbound label.

[0567] A method as in any preceding embodiment, wherein the method is performed by:

[0568] Obtaining a device as in any of the preceding embodiments;

[0569] Depositing samples on one or both plates while the plates are in an open configuration;

[0570] closing the panel to a closed configuration; and

[0571] The sample contact area is read by a reading device to produce an image of the signal.

[0572]

[00136] An apparatus or method as in any preceding embodiment, wherein the label bound to the magnifying surface is visible in less than 60 seconds.

[0573] A device or method as in any preceding embodiment, wherein the method is a homogeneous assay, wherein the signal is read without using a washing step to remove any biological material or label not bound to the amplification surface.

[0574] An apparatus or method as in any preceding embodiment, wherein the tags bound to the magnification surface are read by a pixelated reading method.

[0575] An apparatus or method as in any preceding embodiment, wherein the tags bound to the amplification surface are read by a lumped reading method.

[0576] The device or method of any preceding embodiment, wherein the assay has a detection sensitivity of 0.1 nM or less.

[0577] A device or method as in any preceding embodiment, wherein biological material or labels not bound to the amplification surface are removed by sponging prior to reading.

[0578] A device or method as in any preceding embodiment, wherein the signal amplification layer comprises D2PA.

[0579] In the device or method of any of the aforementioned embodiments, the signal amplifying layer comprises a metal material layer.

[0580] The device or method of any preceding embodiment, wherein the signal amplifying layer comprises a continuous metal film made of a material selected from the group consisting of gold, silver, copper, aluminum, alloys thereof, and combinations thereof.

[0581] An apparatus or method as in any preceding embodiment, wherein the different metal layers locally enhance or act as reflectors, or both, to enhance the optical signal.

[0582] A device or method as in any preceding embodiment, wherein the signal amplification layer comprises a metal material layer and a dielectric material on top of the metal material layer, wherein the capture agent is on the dielectric material.

[0583] The device or method as in any of the preceding embodiments, wherein the metal material layer is a uniform metal layer, a nanostructured metal layer, or a combination thereof.

[0584] An apparatus or method as in any preceding embodiment, wherein the signal is amplified by plasmon enhancement.

[0585] The device or method of any preceding embodiment, wherein determining comprises detecting the label by Raman scattering.

[0586] The device or method of any preceding embodiment, wherein the capture agent is an antibody.

[0587] The device or method of any preceding embodiment, wherein the capture agent is a polynucleotide.

[0588]

[00136] The device or method of any preceding embodiment, wherein the device further comprises a spacer secured to one of the plates, wherein the spacer adjusts the spacing between the first plate and the second plate in the closed configuration.

[0589]

[00136] A device or method as in any preceding embodiment, wherein the amplification factor of the surface amplification layer is adjusted to render visible an optical signal from a single tag directly or indirectly bound to the capture agent.

[0590] The device or method of any preceding embodiment, wherein the amplification factor of the surface amplification layer is adjusted to make optical signals from single tags directly or indirectly bound to the capture agent visible, wherein visible single tags bound to the capture agent are individually counted.

[0591]

[00136] An apparatus or method as in any preceding embodiment, wherein the spacing between the first plate and the second plate in the closed configuration is configured to provide a saturation binding time of the target analyte to the capture agent of 300 seconds or less.

[0592]

[00136] An apparatus or method as in any preceding embodiment, wherein the spacing between the first plate and the second plate in the closed configuration is configured to provide a saturation binding time of the target analyte to the capture agent of 60 seconds or less.

[0593] An apparatus or method as in any preceding embodiment, wherein the amplification factor of the surface amplification layer is adjusted to make the light signal from a single tag visible.

[0594] The device or method of any preceding embodiment, wherein the capture agent is a nucleic acid.

[0595] The device or method of any preceding embodiment, wherein the capture agent is a protein.

[0596] The device or method of any preceding embodiment, wherein the capture agent is an antibody.

[0597]

[00136] A device or method as in any preceding embodiment, wherein the sample contact area of ​​the second plate has a reagent storage site, and wherein the storage site is approximately above the binding site on the first plate in the closed configuration.

[0598]

[00136] A device or method as in any preceding embodiment, wherein the reagent storage site comprises a detection agent that binds to the target analyte.

[0599] The device or method of any preceding embodiment, wherein the detection agent comprises a label.

[0600] The device or method of any preceding embodiment, wherein both the capture agent and the detection agent are bound to the target analyte to form a sandwich comprising the tag.

[0601] In the device or method of any of the aforementioned embodiments, the signal amplifying layer comprises a metal material layer.

[0602] A device or method as in any preceding embodiment, wherein the signal amplification layer comprises a metal material layer and a dielectric material on top of the metal material layer, wherein the capture agent is on the dielectric material.

[0603] The device or method as in any of the preceding embodiments, wherein the metal material layer is a uniform metal layer, a nanostructured metal layer, or a combination thereof.

[0604] A device or method as in any of the foregoing embodiments, wherein the amplification layer comprises a metal material layer and a dielectric material on top of the metal material layer, wherein the capture agent is on the dielectric material, and the dielectric material layer has a thickness of 0.5nm, 1nm, 5nm, 10nm, 20nm, 50nm, 00nm, 200nm, 500nm, 1000nm, 2um, 3um, 5um, 10um, 20um, 30um, 50um, 100um, 200um, 500um, or in a range between any two values.

[0605] An apparatus or method as in any preceding embodiment, wherein the method further comprises quantifying the signal in a region of the image to provide an estimate of the amount of one or more analytes in the sample.

[0606]

[00136] An apparatus or method as in any preceding embodiment, wherein the method comprises identifying and counting individual binding events between analytes and capture agents in the image region, thereby providing an estimate of the amount of one or more analytes in the sample.

[0607] A device or method as in any of the foregoing embodiments, wherein the identification and counting steps include: (1) determining the local intensity of the background signal; (2) determining the local signal intensity of one label, two labels, three labels, four or more labels; and (3) determining the total number of labels in the imaging area.

[0608] As in any of the above embodiments, the identification and counting steps include: (1) determining the local spectrum of the background signal; (2) determining the local signal spectrum of one tag, two tags, three tags, four or more tags; and (3) determining the total number of tags in the imaging area.

[0609] An apparatus or method as in any of the foregoing embodiments, wherein the identification and counting steps include: (1) determining the local Raman characteristics of the background signal; (2) determining the local signal Raman characteristics of one tag, two tags, three tags, four or more tags; and (3) determining the total number of tags in the imaging area.

[0610] An apparatus or method as in any preceding embodiment, wherein the identifying and counting steps comprise determining one or more of the local intensity, spectral, and Raman signatures.

[0611] An apparatus or method as in any preceding embodiment, wherein the method comprises quantifying the aggregate signal in a region of the image, thereby providing an estimate of the amount of one or more analytes in the sample.

[0612]

[00136] An apparatus or method as in any preceding embodiment, wherein the sample contact area of ​​the second plate has a reagent storage site, and wherein in the closed configuration the storage site is approximately above a binding site on the first plate.

[0613] The device or method of any preceding embodiment, wherein the method further comprises the step of labeling the target analyte with a detection agent.

[0614] The device or method of any preceding embodiment, wherein the detection agent comprises a label.

[0615] The device or method of any preceding embodiment, wherein the capture agent and the detection agent are both bound to the target analyte to form a sandwich.

[0616] A device or method as in any preceding embodiment, wherein the method further comprises measuring the volume of the sample in the area imaged by the reader.

[0617] The device or method of any preceding embodiment, wherein the target analyte is a protein, a peptide, a DNA, an RNA, a nucleic acid, a small molecule, a cell, or a nanoparticle.

[0618] An apparatus or method as in any preceding embodiment, wherein the image shows the location, local intensity, and local spectrum of the signal.

[0619] The device or method of any preceding embodiment, wherein the signal is a luminescent signal selected from the group consisting of: fluorescent, electroluminescent, chemiluminescent, and electrochemiluminescent signals.

[0620] An apparatus or method as in any preceding embodiment, wherein the signal is a Raman scattered signal.

[0621]

[00136] A device or method as in any preceding embodiment, wherein the signal is a force due to a local electrical interaction, a local mechanical interaction, a local biological interaction, or a local optical interaction between the plate and the reading device.

[0622] The method or apparatus of any preceding embodiment, wherein the spacer has a columnar shape and a nearly uniform cross-section.

[0623] The method or apparatus of any preceding embodiment, wherein the separation distance (SD) is equal to or less than about 120 um (micrometers).

[0624] The method or apparatus of any preceding embodiment, wherein the separation distance (SD) is equal to or less than about 100 um (micrometers).

[0625] A method or apparatus as in any preceding embodiment, wherein the fourth power of the separation distance (ISD) divided by the thickness (h) of the flexible board and the Young's modulus (E) (ISD 4 / (hE)) is 5×10 6 um 3 / GPa or less.

[0626] A method or apparatus as in any preceding embodiment, wherein the fourth power of the separation distance (ISD) divided by the thickness (h) of the flexible board and the Young's modulus (E) (ISD 4 / (hE)) is 5×10 5 um 3 / GPa or less.

[0627] A method or apparatus as in any of the foregoing embodiments, wherein the spacers have a columnar shape, a substantially flat top surface, a predetermined substantially uniform height and a predetermined constant spacing distance, wherein the spacing distance is at least about 2 times greater than the size of the analyte, wherein the Young's modulus of the spacers multiplied by the packing factor of the spacers is equal to or greater than 2 MPa, wherein the packing factor is the ratio of the spacer contact area to the total plate area, and wherein for each spacer, the ratio of the lateral dimension of the spacer to its height is at least 1 (1).

[0628] A method or apparatus as in any preceding embodiment, wherein the spacers have a columnar shape, a substantially flat top surface, a predetermined substantially uniform height and a predetermined constant separation distance, the separation distance being at least about 2 times greater than a size of the analyte, wherein a Young's modulus of the spacers multiplied by a packing factor of the spacers is equal to or greater than 2 MPa, wherein the packing factor is a ratio of the spacer contact area to the total plate area, and wherein for each spacer, a ratio of a lateral dimension of the spacer to its height is at least one (1), wherein the separation distance (ISD) to the fourth power divided by the thickness (h) of the flexible plate and the Young's modulus (E) (ISD 4 / (hE)) is 5x10 6 um 3 / GPa or less.

[0629] The method or apparatus of any of the preceding embodiments, wherein a ratio of the spacing distance of the spacers to an average width of the spacers is 2 or greater, and a filling factor of the spacers multiplied by a Young's modulus of the spacers is 2 MPa or greater.

[0630] The method or device of any preceding embodiment, wherein the analyte is a protein, a peptide, a nucleic acid, a synthetic compound, or an inorganic compound.

[0631] A method or device as in any preceding embodiment, wherein the sample is a biological sample selected from amniotic fluid, aqueous humor, vitreous humor, blood (e.g., whole blood, fractionated blood, plasma or serum), breast milk, cerebrospinal fluid (CSF), cerumen (ear wax), chyle, chyme, endolymph, perilymph, feces, breath, gastric acid, gastric juice, lymph, mucus (including nasal drainage and sputum), pericardial fluid, peritoneal fluid, pleural fluid, pus, rheumatic fluid, saliva, exhaled condensate, sebum, semen, sputum, sweat, synovial fluid, tears, vomitus and urine.

[0632] The method or apparatus of any preceding embodiment, wherein the spacer has a columnar shape, and a ratio of the width to the height of the column is equal to or greater than 1.

[0633] A method or apparatus as in any preceding embodiment, wherein the sample deposited on one or both of the plates has an unknown volume.

[0634] The method or apparatus of any preceding embodiment, wherein the spacers have a columnar shape, and the columns have a substantially uniform cross-section.

[0635] The method or apparatus of any preceding embodiment, wherein the sample is used to detect, purify and quantify compounds or biomolecules associated with a stage of a certain disease.

[0636] The method or apparatus of any preceding embodiment, wherein the sample is associated with infectious and parasitic diseases, injuries, cardiovascular diseases, cancer, psychiatric disorders, neuropsychiatric disorders, pulmonary diseases, renal diseases, and other and organic diseases.

[0637] The method or apparatus of any preceding embodiment, wherein the sample is involved in the detection, purification and quantification of microorganisms.

[0638] The method or apparatus of any preceding embodiment, wherein the sample involves viruses, fungi and bacteria from the environment (eg, water, soil or a biological sample).

[0639] A method or apparatus as in any preceding embodiment, wherein the sample involves detection or quantification of compounds or biological samples (eg, toxic waste, anthrax) that pose a threat to food safety or national security.

[0640] The method or apparatus of any preceding embodiment, wherein the sample is related to quantification of a vital parameter in medical or physiological monitoring.

[0641] The method or device of any preceding embodiment, wherein the sample is related to glucose, blood, oxygen level, total blood cell count.

[0642] The method or apparatus of any preceding embodiment, wherein the sample is associated with detection and quantification of specific DNA or RNA from a biological sample.

[0643] A method or apparatus as in any preceding embodiment, wherein the sample is associated with sequencing and comparison of genetic sequences of chromosomal and mitochondrial DNA for genomic analysis.

[0644] A method or apparatus as in any preceding embodiment, wherein the sample involves detecting a reaction product, such as during drug synthesis or purification.

[0645] A method or device as in any preceding embodiment, wherein the sample is cells, tissues, body fluids and feces.

[0646] The method or apparatus of any preceding embodiment, wherein the sample is a sample in the fields of human, veterinary, agricultural, food, environmental and pharmaceutical testing.

[0647] The method or device of any preceding embodiment, wherein the sample is a biological sample selected from hair, nail, ear wax, breath, connective tissue, muscle tissue, neural tissue, epithelial tissue, cartilage, cancer sample or bone.

[0648] A method or apparatus as in any of the preceding embodiments, wherein the spacing distance is in a range of 5 um to 120 um.

[0649] A method or apparatus as in any preceding embodiment, wherein the spacing distance is in the range of 120 um to 200 um.

[0650] A method or apparatus as in any preceding embodiment, wherein the flexible board has a thickness in the range of 20 um to 250 um and a Young's modulus in the range of 0.1 to 5 GPa.

[0651] A method or apparatus as in any preceding embodiment, wherein for the flexible board, a thickness of the flexible board multiplied by a Young's modulus of the flexible board is in a range of 60 to 750 GPa-um.

[0652] A method or apparatus as in any preceding embodiment, wherein the uniform thickness of the sample layer is at least 1 mm 2 It is uniform over the lateral area.

[0653] A method or apparatus as in any preceding embodiment, wherein the uniform thickness of the sample layer is at least 3 mm 2 It is uniform over the lateral area.

[0654] A method or apparatus as in any preceding embodiment, wherein the uniform thickness of the sample layer is at least 5 mm 2 It is uniform over the lateral area.

[0655] A method or apparatus as in any preceding embodiment, wherein the uniform thickness of the sample layer is at least 10 mm 2 It is uniform over the lateral area.

[0656] A method or apparatus as in any preceding embodiment, wherein the uniform thickness of the sample layer is at least 20 mm 2 It is uniform over the lateral area.

[0657] A method or apparatus as in any preceding embodiment, wherein the uniform thickness sample layer is 20 mm 2 Up to 100mm 2 The lateral area within the range is uniform.

[0658] A method or apparatus as in any preceding embodiment, wherein the uniform thickness sample layer has a thickness uniformity of up to + / - 5% or better.

[0659] A method or apparatus as in any preceding embodiment, wherein the uniform thickness sample layer has a thickness uniformity of up to + / - 10% or better.

[0660] A method or apparatus as in any preceding embodiment, wherein the uniform thickness sample layer has a thickness uniformity of up to + / - 20% or better.

[0661] A method or apparatus as in any preceding embodiment, wherein the uniform thickness sample layer has a thickness uniformity of up to + / - 30% or better.

[0662] A method, apparatus, computer program product or system as in any preceding embodiment having five or more monitoring marks, wherein at least three of the monitoring marks are not linear.

[0663] The method, apparatus, computer program product, or system of any preceding embodiment, wherein each of the panels comprises a force receiving region on its respective outer surface for applying an imprecise compressive force that urges the panels together;

[0664] The method, apparatus, computer program product, or system of any preceding embodiment, wherein one or both of the plates are flexible;

[0665] A method, apparatus, computer program product, or system as in any preceding embodiment, wherein the fourth power of the separation distance (IDS) divided by the thickness (h) and Young's modulus (E) of the flexible board (ISD4 / (hE)) is 5x106 um3 / GPa or less.

[0666] The method, apparatus, computer program product, or system of any preceding embodiment, wherein at least one of the spacers is within the sample contacting region;

[0667] A method, apparatus, computer program product or system as in any preceding embodiment, wherein the fat specification of analyte

[0668] A method, apparatus, computer program product, or system as in any preceding embodiment, wherein the fat specification of algorithms

[0669] A method, apparatus, computer program product or system as in any preceding embodiment, wherein the fat imprinting force and the hand pressure (fat specification of algorithms)

[0670] A device, system or method as in any of the foregoing embodiments, wherein the algorithm is stored on a non-transitory computer-readable medium, and wherein the algorithm comprises instructions that, when executed, perform a method of determining a property corresponding to the analyte using a monitoring marker of the device.

[0671] Some examples of tags

[0672] In the present invention, in some embodiments, the mark has the same shape as the spacer.

[0673] In some embodiments, marking is periodic or aperiodic.

[0674] In some embodiments, the distance between two marks is predetermined and known, but the absolute coordinates on the board are unknown.

[0675] In some embodiments, the marking has a predetermined and known shape.

[0676] In some embodiments, the markings are configured to have a distribution in the plate such that regardless of the position of the plate, the markings are always present in the field of view of the imaging optics.

[0677] In some embodiments, the markings are configured to have a distribution in the plate such that regardless of the position of the plate, there are always markings in the field of view of the imaging optics and the number of markings is sufficient for local optical information.

[0678] In some embodiments, the mark is used to control the optical properties of a local area of ​​the sample, and the area size is 1um^2, 5um^2, 10um^2, 20um^2, 50um^2, 100um^2, 200um^2, 500um^2, 1000um^2, 2000um^2, 5000um^2, 10000um^2, 100000um^2, 500000um^2, or in a range between any two values.

[0679] Use of "finite imaging optics"

[0680] In the present invention, in some embodiments, the optical system used to image the assay has a "finite imaging optics". Some embodiments of finite imaging optics include, but are not limited to:

[0681] 1. A finite imaging optical system, comprising:

[0682] Imaging lens;

[0683] Imaging sensors;

[0684] wherein the imaging sensor is part of a camera of a smartphone;

[0685] wherein at least one of the imaging lenses is part of a camera of a smartphone;

[0686] 2. A finite imaging optical system as in any of the preceding embodiments, wherein: the physical optical resolution is less than 1um, 2um, 3um, 5um, 10um, 50um, or within a range between any two values.

[0687] 3. A finite imaging optical system as in any of the preceding embodiments, wherein: the physical optical resolution is less than 1um, 2um, 3um, 5um, 10um, 50um, or within a range between any two values.

[0688] 4. A finite imaging optical system as in any preceding embodiment, wherein: the preferred physical optical resolution is between about 1 um and 3 um;

[0689] 5. A finite imaging optical system as in any of the preceding embodiments, wherein: the numerical aperture is less than 0.1, 0.15, 0.2, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, or in a range between any two values.

[0690] 6. A finite imaging optical system as in any preceding embodiment, wherein: the preferred numerical aperture is between 0.2 and 0.25.

[0691] 7. A finite imaging optical system as in any preceding embodiment, wherein: the working distance is 0.2 mm, 0.5 mm, 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or in a range between any two values.

[0692] 8. A finite imaging optical system as in any preceding embodiment, wherein: the working distance is 0.2 mm, 0.5 mm, 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or in a range between any two values.

[0693] 9. A finite imaging optical system as in any of the preceding embodiments, wherein: the preferred working distance is between 0.5 mm and 1 mm.

[0694] 10. A finite imaging optical system as in any preceding embodiment, wherein: the focal depth is 100 nm, 500 nm, 1 um, 2 um, 10 um, 100 um, 1 mm, or in a range between any two values.

[0695] 11. A finite imaging optical system as in any preceding embodiment, wherein: the depth of focus is 100 nm, 500 nm, 1 um, 2 um, 10 um, 100 um, 1 mm, or in a range between any two values.

[0696] 12. The finite imaging optical system of any preceding embodiment, wherein: the image sensor is part of a smartphone camera module.

[0697] 13. The finite imaging optical system of any preceding embodiment, wherein: the diagonal length of the image sensor is less than 1 inch, 1 / 2 inch, 1 / 3 inch, 1 / 4 inch, or within a range between any two values;

[0698] 14. The finite imaging optical system of any preceding embodiment, wherein: the imaging lens comprises at least two lenses, and one lens is part of a camera module of a smartphone.

[0699] 15. The finite imaging optical system of any preceding embodiment, wherein: at least one outer lens is paired with an inner lens of a smartphone.

[0700] 16. A finite imaging optical system as in any of the preceding embodiments, wherein: the optical axis of the outer lens is aligned with the inner lens of the smartphone, and the alignment tolerance is less than 0.1 mm, 0.2 mm, 0.5 mm, 1 mm, or within a range between any two values.

[0701] 17. A finite imaging optical system as in any of the preceding embodiments, wherein: the height of the outer lens is less than 2 mm, 5 mm, 10 mm, 15 mm, 20 mm, or within a range between any two values.

[0702] 18. A finite imaging optical system as in any preceding embodiment, wherein: the preferred height of the outer lens is between 3 mm and 8 mm.

[0703] 19. A finite imaging optical system as in any preceding embodiment, wherein: the preferred height of the outer lens is between 3 mm and 8 mm.

[0704] 20. A finite imaging optical system as in any preceding embodiment, wherein: the diameter of the outer lens is less than 2 mm, 4 mm, 8 mm, 10 mm, 15 mm, 20 mm, or in a range between any two values.

[0705] 21. A finite imaging optical system as in any of the preceding embodiments, wherein: the physical optical magnification per unit is less than 0.1X, 0.5X, 1X, 2X, 4X, 5X, 10X, or within a range between any two values.

[0706] 22. A finite imaging optical system as in any of the preceding embodiments, wherein: the preferred physical optical magnification is less than 0.1X, 0.5X, 1X, 2X, 4X, 5X, 10X, or within a range between any two values.

[0707] Use of "Limited Sample Operation"

[0708] In the present invention, in some embodiments, the sample positioning system used to image the assay has "limited sample manipulation". Some examples of limited sample manipulation include, but are not limited to:

[0709] Description of Limited Sample Operating System:

[0710] 1. Limited sample operating system, including:

[0711] Sample holder;

[0712] The sample holder has a socket for receiving a sample card.

[0713] 2. A finite sample manipulation system as in any of the preceding embodiments, wherein: the accuracy of positioning the sample in the direction along the optical axis is worse than 0.1um, 1um, 10um, 100um, 1mm, or within a range between any two values.

[0714] 3. A finite sample manipulation system as in any preceding embodiment, wherein: the preferred accuracy of positioning the sample in the direction along the optical axis is between 50um and 200um.

[0715] 4. A finite sample manipulation system as in any preceding embodiment, wherein: the accuracy of positioning the sample in a plane perpendicular to the optical axis is worse than 0.01um, 0.1um, 1um, 10um, 100um, 1mm, or within a range between any two values.

[0716] 5. A finite sample manipulation system as in any preceding embodiment, wherein: the preferred accuracy of positioning the sample in a plane perpendicular to the optical axis is between 100 um and 1 mm.

[0717] 6. A limited sample handling system as in any of the preceding embodiments, wherein the horizontal error of positioning the sample card is better than 0.01 degrees, 0.1 degrees, 0.5 degrees, 1 degree, 10 degrees, or within a range between any two values.

[0718] 7. A limited sample handling system as in any preceding embodiment, wherein a preferred horizontal error for positioning the sample card is between 0.5 degrees and 10 degrees.

[0719] 8. A limited sample handling system as in any preceding embodiment, wherein a preferred horizontal error for positioning the sample card is between 0.5 degrees and 10 degrees.

[0720] Detection of sharp edges of markings

[0721] The term "sharp optical edge" refers to the edge or boundary of an object in an image captured by a particular optical system having the following properties: at the edge or boundary of the object in the image, the light intensity (including but not limited to R, G, B, gray, hue, brightness) changes sharply with respect to position. For example, quantitatively, as shown in FIG. 19 , at the boundary, the distance (X) where the normalized intensity decreases from 90% to 10% 90％ ) should be less than 5% of the length of the object in the image by such an optical system. In other words, boundary gradient = 90% / X 90％ >1800% / object length. An example of a sharp edge is the edge of a columnar spacer with a flat top and nearly 90 degree sidewalls. An example of an object without sharp edges is a sphere.

[0722] Example of TLD and volume estimation using monitoring tags

[0723] FIG7-A shows an embodiment of a sample holding device QMAX device and its monitoring markers, columns, used in some embodiments of the present invention. The columns in the QMAX device make the gap between the two parallel plates of the sample holding device uniform. The gap is narrow and related to the size of the analyte, where the analyte forms a monolayer in the gap. In addition, the monitoring markers in the QMAX device are a special form of columns, so they are not immersed in the sample and can be imaged together with the sample by the imager in an image-based assay.

[0724] Example of TLD (True Lateral Dimension) estimation using monitoring markers

[0725] In some embodiments of the present invention, it is used for TLD and true volume estimation. Monitoring markers (columns) are used as detectable anchors. However, it is difficult to detect monitoring markers with an accuracy suitable for TLD estimation in image-based determinations. This is because these monitoring markers are infiltrated and surrounded by analytes in the sample holding device. They are distorted and blurred in the image due to distortion of the lens, light diffraction of micro-objects in the sample, micro-level defects, misalignment of focus, noise in the sample image, etc. If the imager is a camera from a commercial device (e.g., a camera from a smart phone), it becomes even more difficult because these cameras are not calibrated by dedicated hardware once they leave manufacturing.

[0726] In the present invention, detection and localization of monitoring markers as detectable anchors for TLD estimation are formulated in a machine learning framework, and a dedicated machine learning model is built / trained to detect them in microscopic imaging. In addition, in some embodiments of the present invention, the distribution of monitoring markers is intentionally set to be periodic and distributed in a predetermined pattern. This makes the method of the present invention more robust and reliable.

[0727] The embodiments of the present invention include:

[0728] (1) loading a sample into a sample holding device, such as a QMAX device, wherein a monitoring marker having a known configuration is present in the device that is not immersed in the sample and can be imaged by an imager;

[0729] (2) capturing an image of the sample in the sample holding device including the analyte and the monitoring marker;

[0730] (3) building and training a machine learning (ML) model to detect monitoring markers in images of samples;

[0731] (4) detecting and locating a monitoring marker in a sample holding device from an image of the sample using the ML detection model from (3);

[0732] (5) generating a marker grid according to the monitoring markers detected in (4);

[0733] (6) calculating homography transformation based on the generated monitoring marker grid;

[0734] (7) estimating and preserving the true lateral size of the sample image according to the homography transformation from (6); and

[0735] (8) Apply the estimated TLD from (7) in subsequent image-based assays to determine the area, size, volume, and concentration of the analyte.

[0736] In some embodiments of the present invention, region-based TLD estimation and calibration is employed in image-based determination. It comprises:

[0737] (1) loading the sample into a sample holding device, such as a QMAX device, wherein a monitoring marker is present in the device - not immersed in the sample and capable of being imaged by an imager in the image-based assay;

[0738] (2) capturing an image of the sample in the sample holding device including the analyte and the monitoring marker;

[0739] (3) building and training a machine learning (ML) model for detecting monitoring markers from images of the sample taken by the imager;

[0740] (4) dividing the sample image captured by the imager into non-overlapping areas;

[0741] (5) using the ML model of (3) to detect and locate the monitoring marker from an image of the sample taken by the imager;

[0742] (6) generating a region-based marker grid for each region where more than four non-collinear monitoring markers are detected in the local region;

[0743] (7) generating a marker grid for all areas not in (6) based on the monitoring markers detected from the image of the sample taken by the imager;

[0744] (8) computing a region-specific homography transform for each region in (6) based on its own region-based labeling grid generated in (6);

[0745] (9) Compute homography transformations for all other regions based on the labeled grid generated in (7);

[0746] (10) estimating the region-based TLD for each region in (6) based on the region-based homography transformation generated in (8);

[0747] (11) estimating TLDs of other regions based on the homography transformation from (9); and

[0748] (12) In subsequent image-based determinations, the estimated TLDs from (10) and (11) are saved and applied to the partitions.

[0749] When the monitoring marks are distributed in a predefined periodic pattern, such as in a QMAX device, they appear and are distributed periodically at a certain pitch, as a result, the detection of the monitoring marks in the above process becomes more robust and reliable. This is because, if the detected position and configuration do not follow the predefined periodic pattern, all monitoring marks can be identified and determined from only a small number of detected monitoring marks by utilizing the periodicity, and detection errors can be corrected and eliminated.

[0750] The terms "monitoring mark", "monitor mark" and "marker" are interchangeable in the description of the present invention.

[0751] The terms "imager" and "camera" are used interchangeably in the description of the present invention.

[0752] The term "denoising" refers to the process of removing noise from a received signal. An example is removing noise from an image of a sample, as images from an imager / camera can pick up noise from a variety of sources, including but not limited to white noise, salt and pepper noise, Gaussian noise, etc. Denoising methods include but are not limited to: linear and nonlinear filtering, wavelet transforms, statistical methods, deep learning, etc.

[0753] The term "image normalization" refers to algorithms, methods, and devices that change the range of pixel intensity values ​​in a processed image. For example, it includes but is not limited to increasing contrast by histogram stretching, subtracting the average pixel value from each image, etc.

[0754] In some embodiments of the invention, methods and algorithms are designed to utilize monitoring markers in, for example, a sample holding device (e.g., a QMAX device). This includes, but is not limited to, the estimation and adjustment of the following parameters in the imaging device:

[0755] 1. Shutter speed,

[0756] 2.ISO,

[0757] 3. Focus (lens position),

[0758] 4. Exposure compensation,

[0759] 5. White balance: temperature, color, and

[0760] 6. Scaling (scale factor).

[0761] Examples of image processing / analysis algorithms used with markers

[0762] In some embodiments of the present invention, the monitoring markers of the present invention are used to apply and enhance image processing / analysis. They include but are not limited to the following image processing algorithms and methods:

[0763] 1. Histogram-based operations include but are not limited to:

[0764] a. Contrast stretch;

[0765] b. Balance;

[0766] c.Minimum filtering;

[0767] d. Median filtering; and

[0768] e. Maximum filtering.

[0769] 2. Mathematical operations include but are not limited to:

[0770] a. Binary operations: NOT, OR, AND, XOR, SUB, etc., and

[0771] b. Arithmetic-based operations: ADD, SUB, MUL, DIV, LOG, EXP, SQRT, TRIG, INVERT, etc.

[0772] 3. Convolution-based operations in the spatial and frequency domains include but are not limited to Fourier transform, DCT, integer transform, wavelet transform, etc.

[0773] 4. Smoothing operations include but are not limited to:

[0774] a. Linear filtering: uniform filtering, triangle filtering, Gaussian filtering, etc., and

[0775] b. Nonlinear filtering: median filtering, Kuwahara filtering, etc.

[0776] 5. Operations based on derivatives include but are not limited to:

[0777] a. First-order derivative: gradient filtering, basic derivative filtering, Prewitt gradient filtering, Sobel gradient filtering, alternative gradient filtering, Gaussian gradient filtering, etc.;

[0778] b. Second-order derivatives: basic second-order derivative filtering, frequency domain Laplacian, Gaussian second-order derivative filtering, alternative Laplacian filtering, gradient-directed second-order derivative (SDGD) filtering, etc., and

[0779] c. Other filtering with higher order derivatives, etc.

[0780] 6. Morphology-based operations include but are not limited to:

[0781] a. Expansion and erosion;

[0782] b. Boolean convolution;

[0783] c.Open and close;

[0784] d. Hit and miss operations;

[0785] e. Segmentation and contouring;

[0786] f. Skeleton;

[0787] g. dissemination;

[0788] h. Gray value morphological processing: grayscale expansion, grayscale erosion, grayscale opening, grayscale closing, etc.; and i. Morphological smoothing, morphological gradient, morphological Laplacian, etc.

[0789] Other Examples of Image Processing / Analysis Techniques

[0790] In some embodiments of the invention, image processing / analysis algorithms are used with and enhanced by monitoring markers. They include but are not limited to the following:

[0791] 1. Image enhancement and restoration including but not limited to

[0792] a. Sharpening and unsharpening,

[0793] b. Noise suppression, and

[0794] c. Distortion suppression.

[0795] 2. Image segmentation includes but is not limited to:

[0796] a.Threshold - fixed threshold, histogram export threshold, Isodata algorithm, background symmetry algorithm, triangle algorithm, etc.;

[0797] b. Edge finding - gradient-based process, zero-crossing-based process, PLUS-based process, etc.;

[0798] c. Binary mathematical morphology - salt and pepper filtering, separation of objects with holes, filling holes in objects, removal of boundary touching objects, exoskeleton, touching objects, etc.; and

[0799] d. Gray value mathematical morphology - top-hat transformation, adaptive thresholding, local contrast stretching, etc.

[0800] 3. Feature extraction and matching include but are not limited to:

[0801] a. Independent component analysis;

[0802] b. Isometric mapping;

[0803] c. Principal component analysis and kernel principal component analysis;

[0804] d. Latent semantic analysis;

[0805] e. Least squares and partial least squares;

[0806] f. Multi-factor dimensionality reduction and nonlinear dimensionality reduction;

[0807] g.Multilinear principal component analysis;

[0808] h. Multilinear subspace learning;

[0809] i. Semidefinite embedding; and

[0810] j. Autoencoder / decoder.

[0811] 4. Object detection, classification and localization

[0812] 5. Image Understanding

[0813] Example E1. Improvement of microscopic imaging using monitoring markers

[0814] The present invention uses monitoring marks to improve focus in microscopic imaging. In particular, monitoring marks with sharp edges will provide detectable (visible features) for the focus evaluation algorithm to analyze the focus condition of a specific focus setting, especially in low light environments and microscopic imaging. In the embodiment of image-based determination, the focus evaluation algorithm is a core part in the automatic focus implementation.

[0815] For some diagnostic applications (e.g., colorimetry, absorption-based hemoglobin testing, and CBC for samples with very low cell concentrations), the detectable features provided by the analyte in the image of the sample are often insufficient for the focus assessment algorithm to run accurately and smoothly. Monitoring markers with sharp edges, such as those in the QMAX device, provides additional detectable features to the focus assessment routine to achieve the accuracy and reliability required in image-based assays.

[0816] For some diagnostic applications, the analytes in the sample are not evenly distributed. Relying solely on the features provided by the analytes tends to produce some unfair focus settings, which give high weight to some local high concentration areas in focus, while low analyte concentration areas are off target. In some embodiments of the present invention, this effect is controlled by focusing adjustments based on information from monitoring markers that have strong edges and are evenly distributed in a precisely processed periodic pattern.

[0817] In addition, each imager has an imaging resolution limited in part by the number of pixels in its imager sensor, which varies from one to several million pixels. For some microscopic imaging applications, the analytes have small or microscopic sizes in the sample, such as the size of a platelet in human blood, which is about 1.4 um. When the target detection procedure requires a certain number of pixels, the limited resolution in the image sensor, in addition to the available size of the FOV, places a significant constraint on the device's capabilities in image-based assays.

[0818] Single Image Super Resolution (SISR) is a technique that uses image processing and / or machine learning techniques to upsample the original source image to a higher resolution and remove as much blur as possible caused by interpolation, so that object detection routines can also be run on the newly generated image. This will significantly reduce the above constraints and enable some applications that would not otherwise be possible. Monitoring markers with known shapes and structures (such as those in QMAX cards) can be used as local references to evaluate SISR algorithms to avoid the over-sharpening effect produced by most prior art algorithms.

[0819] In some embodiments of the invention, image fusion is performed to overcome the physical SNR (signal-to-noise ratio) limitations in image-based assays.

[0820] The signal-to-noise ratio measures the quality of an image of a sample taken by an imager in microscopic imaging. There are practical limitations to imaging devices due to cost, technology, manufacturing, etc. In some cases, such as in mobile healthcare, applications require a higher SNR than conventional imaging devices can provide. In some embodiments of the present invention, multiple images (with the same and / or different imaging settings, such as embodiments for 3D fusion to merge multiple images focused at different focal depths into one super-focused image) are taken and processed to generate an output image with a higher SNR, thereby enabling these applications.

[0821] However, images captured by an imager or multiple imagers tend to have some disadvantages and defects caused by physical limitations and implementation constraints. In image-based assays, this situation becomes serious in microscopic imaging of samples because the analytes in the sample have tiny sizes and often have no obvious edge features. In some embodiments of the present invention, monitoring markers in a sample holding device (e.g., a QMAX card) are used for an enhanced solution.

[0822] One such embodiment is dealing with distortion in images of a sample taken by an imager. If the distortion parameters are known, the situation is relatively simple (most manufacturers give curves / tables for their lenses to describe proportional distortion, other distortions can be measured in well-defined experiments). However, when the distortion parameters are unknown (it can vary with focus position or even with the sample), using monitoring marks, the new algorithm can use regularly or even periodically placed monitoring marks of a sample holding device (e.g., a QMAX card) to iteratively estimate the distortion parameters without the need for a single coordinate reference.

[0823] Other Examples

[0824] In the present invention, in some embodiments, the sample holding device has a flat surface with some special monitoring marks for analyzing micro features in image-based assays. Some exemplary embodiments are listed below:

[0825] Estimation of true lateral dimension (TLD) of a microscopic image of a sample in an image-based assay. The true lateral dimension (TLD) determines the physical size of the imaged analyte in the real world and also determines the image coordinates of the sample in the real world that are relevant to the concentration estimation in the image-based assay. Monitoring markers can be used as detectable anchors to determine the TLD and improve the accuracy of the image-based assay. In an embodiment of the present invention, a machine learning model is used to detect the monitoring markers, from which the TLD / FoV of the image of the sample is derived. In addition, if the monitoring markers have a periodic distribution pattern on the flat surface of the sample holding device, the detection of the monitoring markers and the TLD / FoV estimation based on each sample can become more reliable and robust in the image-based assay.

[0826] The analyte is analyzed using the measured response of the analyte compound at a specific wavelength or wavelengths of light to predict the analyte concentration. Monitoring markers that are not immersed in the sample can be used to determine the light absorption corresponding to the background of zero concentration - to determine the analyte concentration by light absorption, and this method is used for the HgB test in the present invention. In addition, each monitoring marker can be used as an independent detector of background absorption to make the concentration estimate robust and reliable.

[0827] Focusing on microscopic images for image-based assays. Evenly distributed monitor marks can be used to improve focus accuracy. (a) It can be used to provide a minimum amount of visual features for samples with no / fewer than necessary features for reliable focusing. This can be performed in low light due to the edge content of the monitor marks. (b) It can be used to provide visual features when features in the sample are not evenly distributed to make focus decisions fair. (c) It can provide a reference for local lighting conditions that have no / less / different impact on the content of the sample to adjust weights in the focus assessment algorithm.

[0828] The monitoring marks can be used as a reference for detecting and / or correcting image defects caused by, but not limited to, unevenly distributed illumination, various types of image distortion, noise, and imperfect image preprocessing operations. For example, as shown in FIG. 18 , when a straight line in the 3D world is mapped into a curve in the image of the sample, the position of the mark can be used to detect and / or correct scale distortion. The scale distribution parameters of the entire image can be estimated based on the position changes of the monitoring marks of the sample holding device described herein. And by performing linear tests on horizontal / vertical lines in the reproduced image based on the distortion removal of the assumed scale distortion parameters, the values ​​of the scale distortion parameters can be iteratively estimated.

[0829] Examples of machine learning (ML) calculations

[0830] One way to use machine learning is to use a trained machine learning model in the inference process of processing to perform detection of analytes in an image of a sample and calculate a bounding box covering their location. Another way to use machine learning methods to detect and locate analytes in a sample image is to build and train a detection and segmentation model that involves annotating the analytes in the sample image at the pixel level. In this method, in an image-based assay, a tight binary pixel mask covering the analytes in the sample image can be used to detect and locate the analytes in the sample image.

[0831] When measuring hemoglobin in human blood, an image is taken at a given narrowband wavelength, and then the average energy through the analyte area and the reference area is analyzed. Based on the known absorption rate of the analyte at a given wavelength and the height of the analyte sample area, the concentration can be estimated. However, this measurement is noisy. To eliminate the noise, multiple images can be taken using different wavelengths of light and machine learning regression can be used to achieve a more accurate and robust estimate. Machine learning-based inference takes multiple input images of the sample, collected at different wavelengths, and outputs a single concentration value.

[0832] E. Machine Learning Algorithms

[0833] E-1. QMAX device for measurement and imaging

[0834] According to the present invention, a device for detecting and locating biological analytes is disclosed, comprising a QMAX device, an imager and a computing unit. A biological sample is suspected to be present on the QMAX device. The count and location of the analyte contained in the sample are obtained through the present disclosure.

[0835] According to the present invention, an imager captures an image of a biological sample. The image is submitted to a computing unit. The computing unit can be physically connected directly to the imager, connected via a network, or directly via image transmission.

[0836] E-2. Workflow

[0837] The disclosed analyte detection and localization employs machine learning deep learning. Machine learning algorithms are algorithms that are capable of learning from data. A more rigorous definition of machine learning is "a computer program is said to learn from experience E about some class of tasks T and performance measure P if its performance at tasks in T improves with experience E as measured by P." It explores the study and construction of algorithms that can learn and predict data - such algorithms overcome following strictly static program instructions by making data-driven predictions or decisions by building models based on sample inputs.

[0838] Deep learning is a specific type of machine learning based on a set of algorithms that try to model high-level abstractions in data. In a simple case, there might be two groups of neurons: neurons that receive input signals and neurons that send output signals. When the input layer receives input, it passes a modified version of the input to the next layer. In a deep network, there are many layers between the input and output (and these layers are not made of neurons, but it can help to think of it that way), allowing the algorithm to use multiple processing layers consisting of multiple linear and nonlinear transformations.

[0839] The disclosed analyte detection and localization workflow includes two stages, training and prediction, as shown in Figure 6-A. We describe the training and prediction stages in the following paragraphs.

[0840] train

[0841] During the training phase, the annotated training data is fed into a convolutional neural network. A convolutional neural network is a type of neural network specialized for processing data with a known lattice-like topology. Examples include time series data, which can be thought of as a 1D grid sampled at regular time intervals, and image data, which can be thought of as a 2D grid of pixels. Convolutional networks have been very successful in real-world applications. The name "convolutional neural network" indicates that the network employs a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is simply a neural network that uses convolution to replace the general matrix multiplication in at least one of its layers.

[0842] The training data is annotated for the analyte to be detected. The annotation indicates whether the analyte is present in the training data. The annotation can be in the form of a bounding box that completely contains the analyte or the center position of the analyte. In the latter case, the center position is further converted into a circle covering the analyte.

[0843] When the training data is large, it poses two challenges: annotation (usually done manually) is time-consuming, and training is computationally expensive. To overcome these challenges, the training data can be split into patches of small size, and then these patches or parts of these patches can be annotated and trained.

[0844] The annotated training data is fed to a convolutional neural network for model training. The output is a model that can be used to make pixel-level predictions on images. We use the Caffe library with a fully convolutional network (FCN). Other convolutional neural network architectures such as TensorFlow can also be used.

[0845] The training phase produces a model that will be used in the prediction phase. This model can be reused in the prediction phase on input images. Therefore, the compute unit only needs to access the produced model. It does not need to access the training data, nor does it have to run the training phase on the compute unit.

[0846] predict

[0847] In the prediction phase, a detection component is applied to the input image, followed by a localization component. The output of the prediction phase is the counts of analytes contained in the sample, as well as the location of each analyte.

[0848] In the detection component, the input image is fed to the convolutional neural network along with the model generated from the training phase. The output of the detection stage is a pixel-level prediction in the form of a heat map. The heat map can have the same size as the input image, or it can be a scaled-down version of the input image. Each pixel in the heat map has a value from 0 to 1, which can be considered as the probability (confidence) of whether the pixel belongs to the analyte. The higher the value, the greater the chance that it belongs to the analyte.

[0849] The heat map is the input of the localization component. We disclose an algorithm to locate the center of the analyte. The main idea is to iteratively detect local peaks based on the heat map. After finding the peak, the local area around the peak but with smaller values ​​is calculated. We remove this area from the heat map and find the next peak from the remaining pixels. The process is repeated, only to remove all pixels from the heat map.

[0850] One embodiment of the localization algorithm is to sort the heat map values ​​from the highest value to the lowest value into a one-dimensional ordered list. Then pick the pixel with the highest value and remove it from the list along with its neighbors. Repeat this process to pick the pixel with the highest value in the list until all pixels have been removed from the list.

[0851] Algorithm global search (heat map)

[0852] Input:

[0853] heatmap

[0854] Output:

[0855] loci

[0856] loci←{}

[0857] sort(heatmap)

[0858] while(heatmap is not empty){

[0859] s←pop(heatmap)

[0860] D←{disk center as s with radius R}

[0861] heatmap=heatmap\D / / remove D from the heatmap

[0862] add s to loci

[0863] }

[0864] After sorting, the heatmap is a one-dimensional ordered list where the heatmap values ​​are sorted from high to low. Each heatmap value is associated with its corresponding pixel coordinate. The first item in the heatmap is the item with the highest value, which is the output of the pop(heatmap) function. A disk is created where the center is the pixel coordinate of the disk with the highest heatmap value. All heatmap values ​​whose pixel coordinates are within the disk are then removed from the heatmap. The algorithm repeatedly pops the highest value in the current heatmap, removing disks around it, until the item is removed from the heatmap.

[0865] In the ordered list heatmap, each item is aware of the continuation item and the following item. When removing an item from an ordered list, we make the following changes, as shown in Figure 6-B:

[0866] Assume that the removed item is x r , whose ongoing project is x p , whose subsequent item is x f .

[0867] For project x p , redefine its successor item to be the successor item in the removed item. Therefore, x p The subsequent project is now x f .

[0868] For removing item x r , undefines its proceeding and successor items, removing it from the ordered list.

[0869] For subsequent projects x f , redefine its ongoing project as the ongoing project in the removed project. Therefore, x f The ongoing project is now x p .

[0870] After all items are removed from the ordered list, the localization algorithm ends. The number of elements in the set locus will be the count of analytes and the position information is the pixel coordinates of each s in the set locus.

[0871] Another embodiment searches for local peaks that are not necessarily the local peak with the highest heatmap value. To detect each local peak, we start from a random starting point and search for the local maximum. After finding a peak, calculate the local area around the peak but with smaller values. We remove this area from the heatmap and find the next peak from the remaining pixels. Repeat the process, only removing all pixels from the heatmap.

[0872] algorithm local search(s, heatmap)

[0873] Input:

[0874] s:starting location(x,y)

[0875] heatmap

[0876] Output:

[0877] s:location of local peak.

[0878] We only consider pixels of value>0.

[0879] Algorithm Cover(s,heatmap)

[0880] Input:

[0881] s:location of local peak.

[0882] heatmap:

[0883] Output:

[0884] cover:a set of pixels covered by peak:

[0885] This is a breadth-first search algorithm starting from s with a changed visit point condition: neighbors p of the current position q are added to the cover only if heatmap[p]>0 and heatmap[p]<=heatmap[q]. Thus, every pixel in the cover has a non-descending path to the local peak s.

[0886] Algorithm positioning (heat map)

[0887] Input:

[0888] heatmap

[0889] Output:

[0890] loci

[0891] loci←{}

[0892] pixels←{all pixels from heatmap}

[0893] while pixels is not empty

[0894] s←any pixel from pixels

[0895] s←LocalSearch(s,heatmap) / / s is now local peak

[0896] probe local region of radius R surrounding s for better local peak

[0897] r←Cover(s,heatmap)

[0898] pixels←pixels\r / / remove all pixels in cover

[0899] add s to loci.

[0900] E-3. Examples of the present invention

[0901] EA1. A deep learning method for data analysis, comprising:

[0902] (f) receiving an image of a test sample, wherein the sample is loaded into a QMAX device and the image is captured by an imager coupled to the QMAX device, wherein the image includes a detectable signal from an analyte in the test sample;

[0903] (g) analyzing the image using a detection model and generating a two-dimensional data array of the image, wherein the two-dimensional data array includes probability data of the analyte at each position in the image, the detection model being established through a training process, the training process comprising:

[0904] iii. feeding the annotated dataset into a convolutional neural network, wherein the annotated dataset is from a sample of the same type and for the same analyte as the test sample; and

[0905] iv. Training and building detection models through convolution; and

[0906] (h) Analyzing the two-dimensional data array to detect local signal peaks by:

[0907] iii. Signal list process, or

[0908] iv. Local search process; and

[0909] (i) Calculate the amount of analyte based on the local signal peak information.

[0910] EB1. A system for data analysis, comprising:

[0911] QMAX device, imager and computing unit, wherein:

[0912] (a) the QMAX apparatus is configured to compress at least a portion of the test sample into a layer of highly uniform thickness;

[0913] (b) an imager configured to generate an image of the sample at the layer of uniform thickness, wherein the image includes a detectable signal from an analyte in the test sample;

[0914] (c) The computing unit is configured to:

[0915] i. receiving an image from an imager;

[0916] ii. Analyzing the image using a detection model and generating a two-dimensional data array of the image, wherein the two-dimensional data array includes probability data of the analyte at each position in the image, the detection model is established through a training process, the training process comprising:

[0917] feeding the annotated dataset into a convolutional neural network, wherein the annotated dataset is from a sample of the same type and for the same analyte as the test sample; and

[0918] Training and building detection models through convolution; and

[0919] iii. (C) analyzing the two-dimensional data array using a signal list process or a local search process to detect local signal peaks; and

[0920] iv. Calculate the amount of analyte based on the local signal peak information.

[0921] EA2. The method of embodiment EA1, wherein the signal list process comprises:

[0922] i. Iteratively detecting local peaks from a two-dimensional data array to establish a signal list, calculating the local area around the detected local peak, and sequentially removing the detected peak and local area data to the signal list; and

[0923] ii. Repeatedly remove the highest signal from the signal list and remove signals around the highest signal in sequence to detect local signal peaks.

[0924] EA3. The method of any embodiment EA, wherein the local search process comprises:

[0925] i. Find the local maximum in a two-dimensional data array starting from a random point;

[0926] ii. Calculate the local area around the peak but with smaller value;

[0927] iii. removing the local maximum and surrounding smaller values ​​from the two-dimensional data array; and

[0928] iv. Repeat steps i-iii to detect local signal peaks.

[0929] EA4. The method of any preceding EA embodiment, wherein the annotated data set is partitioned prior to annotation.

[0930] EB2. The system of embodiment EB1, wherein the imager comprises a camera.

[0931] EB3. A system as in embodiment EB2, wherein the camera is part of the mobile communication device.

[0932] EB4. A system as in any preceding EB embodiment, wherein the computing unit is part of a mobile communication device.

[0933] Example of identifying error risks to improve measurement reliability

[0934] In some embodiments, a method for improving the reliability of an assay is provided, the method comprising:

[0935] (a) Image the sample on a QMAX card;

[0936] (b) analyzing error risk factors; and

[0937] (c) rejecting the measurement result of the card report card if the error risk factor is above a threshold;

[0938] The error risk factor is one of the following factors or any combination thereof. These factors are, but are not limited to, (1) edge of blood, (2) air bubbles in blood, (3) too small blood volume or too large blood volume, (4) blood cells under the spacer, (5) aggregated blood cells, (6) lysed blood cells, (7) overexposed image of the sample, (8) underexposed image of the sample, (8) poor focus of the sample, (9) optical system error, such as wrong lever position, (10) card not closed, (11) wrong card, which is a card without spacer, (12) dust in the card, (13) oil in the card, (14) dirt outside the focus plane of the card, (15) the card is not in the correct position inside the reader, (16) the card is empty, (17) manufacturing error in the card, (18) wrong card for other applications, (19) dried blood, (20) expired card, (21) large changes in blood cell distribution, (22) no blood sample or no target blood sample and others.

[0939] In some embodiments, the error risk analyzer is capable of detecting, distinguishing, classifying, modifying and / or correcting the following situations in biological and chemical applications in the device: (1) at the edge of the sample, (2) bubbles in the sample, (3) too small sample volume or too large sample volume, (4) sample under the spacer, (5) aggregated sample, (6) lysed sample, (7) overexposed image of the sample, (8) underexposed image of the sample, (8) poor sample focus, (9) optical system errors, such as wrong lever, (10) card not closed, (11) wrong card for a card without a spacer (12) dust in the card, (13) oil in the card, (14) dirt outside the focal plane of the card, (15) card not in the correct position inside the reader, (16) empty card, (17) manufacturing error in the card, (18) wrong card for other applications, (19) dry sample, (20) expired card, (21) large changes in blood cell distribution, (22) wrong sample and others.

[0940] Wherein the threshold is determined from group testing.

[0941] Wherein the threshold is determined from machine learning.

[0942] Wherein the monitoring marker is used as a comparison to identify the error risk factor.

[0943] The monitoring marker is used as a comparison to evaluate a threshold value of the error risk factor.

[0944] More Examples

[0945] Example A1 is a method for improving an imaging-based assay using an apparatus. The method may include receiving a sample image of a sample holder, the sample holder comprising a plurality of monitoring structures integrated on a contact surface of at least one plate of the sample holder, wherein the plurality of monitoring structures are placed according to a pattern, and wherein the contact surface contacts a sample containing a plurality of analytes; detecting the plurality of monitoring structures in the sample image using a machine learning model; optionally performing error correction on the detected plurality of monitoring structures using predetermined structural characteristics associated with the plurality of monitoring structures; determining a true lateral dimension value associated with the sample image based on the detected plurality of monitoring structures; determining a homographic transformation between the positions of the detected plurality of monitoring structures in the sample image and a predetermined distribution pattern of the plurality of monitoring structures in an actual image plane associated with the sample holder based on the true lateral dimension value; transforming the sample image containing a plurality of analytes into a corresponding perspective view in the actual image plane based on the homographic transformation; and calculating at least one morphological characteristic associated with at least one of the plurality of analytes in the sample image.

[0946] In Example A2, the method of Example A1 may further include detecting a center of each of the plurality of monitoring structures; and determining a homography transformation based on the detected centers of the plurality of monitoring structures including at least 4 non-collinear points.

[0947] In Example A3, the method of Example A1 may further provide dividing the sample image into non-overlapping image patches, each of which contains at least 4 non-collinear detection centers of the multiple monitoring structures, wherein a patch-specific homography transformation is estimated and applied to compensate the image patch, and for other image patches, a global homography transformation is estimated and applied based on the detected centers of the multiple monitoring structures in the entire image.

[0948] In Example A4, the method of Example A1 can also provide that the monitoring structure is a column or a monitoring mark. In some implementations, the column can be a nanostructure integrated with one or two contact surfaces of at least one plate substantially perpendicular to the sample holder. In other implementations, these monitoring marks are marked areas on the one or two contact surfaces. These marked areas can have different optical properties (e.g., transparency) from unmarked areas on the one or two contact surfaces. For example, the marked area can be a surface area coated or engraved with a thin layer of nanomaterial, while the unmarked area is not covered by any nanomaterial.

[0949] In Example A5, the method of Example A1 may further provide: training a machine learning model using a labeled training image set, and wherein the machine learning model may be a RetinerNet or a convolutional neural network (CNN) whose parameters are trained using the labeled training image set.

[0950] In Example A6, the method of Example A1 may also provide that the predetermined structural characteristic includes at least one of a periodicity, a shape, or a size associated with a plurality of monitoring structures. The monitoring structures may be arranged in an organized pattern. The periodicity may refer to the number of monitoring structures within a measuring unit (e.g., a linear measurement such as inches or millimeters, or an area measurement such as square inches or square millimeters). The shape may refer to the geometric construction of each monitoring structure. The monitoring structure may be a cylindrical structure having a cross-section that is a triangle, a rectangle, a square, a circle, a polygon, or any suitable two-dimensional shape. The size may refer to the cross-sectional area of ​​the monitoring structure. In one implementation, the monitoring structures may have substantially the same shape and size. In another implementation, the monitoring structures may have a variety of shapes and sizes, and the positions of different monitoring structures are predetermined during the manufacture of the sample holder.

[0951] In Example A7, the method of Example A1 can further provide that the at least one morphological characteristic comprises at least one of a size or a length of one of the plurality of analytes. The size can be an area measurement. The length can be a linear measurement along an axis. For example, for a circle, the length can be a diameter; for a rectangle, the length can be a height, a width, or a diagonal length.

[0952] Example A8 is a method in an image-based assay, micro-selective image assay (MSIA). The method includes capturing an image of a sample for measurement in a sample holding device, wherein the sample holding device may be a sample holder as described in Example A1, and a sample in the sample holding device comprising a known uniform height and an analyte forms a monolayer in an area of ​​interest; estimating the TLD or FoV of the image of the sample to determine estimates of area, size, and volume in the image-based measurement; detecting defects including bubbles or dust in the image of the sample by a trained machine learning model to measure and segment these defects in the image of the sample; estimating the total area of ​​the segmented defects in the sample image using the estimated TLD / FoV from (b) and calculating their actual area size; estimating the actual volume of the sample in the image of the sample for measurement corresponding to the total surface area of ​​the detected defects based on the area estimate from (d) and the known height of the sample in the sample holding device; subtracting the defect volume estimate from (e) corresponding to the total volume under the surface area of ​​the detected defects, removing the surface area of ​​the detected defects in the image of the sample for measurement, and updating the total volume of the remaining sample; and performing an image-based measurement on a selected area of ​​the updated sample image from (f) and the updated sample volume for micro-selective image measurement.

[0953] In Example A9, the method of Example A8 can further provide micro-selective image measurement (MSIA) using the monitoring structure in the sample holding device of A4 to estimate the TLD / FoV of the image of the sample and map the image of the sample to its actual size through the embodiment of A1.

[0954] In Example A10, the method of Example A8 can further provide micro-selective image assay (MSIA) based on other selection criteria, including: a) the distribution of defects in the sample, including bubbles and dust; b) the location of columns and monitoring marks and other artifacts in the image of the sample for assay; and c) the distribution and conditions of analytes in the image of the sample for assay, including conditions for analyte clustering and focusing conditions.

[0955] In Example A11, the method of Example A8 can further provide micro-selective image assay (MSIA) to perform a multi-target assay from an image of the sample, wherein the multi-target assay is based on region / zone selection having areas defined by different reagents or sample heights for multiple assay applications from a single image of the sample.

[0956] In Example A12, the method of either Example A8 or A11 may further provide training a machine learning model on labeled training image samples captured by the imager for defect detection and segmentation in image-based assays.

[0957] In Example A13, the method of either Example A8 or A11 may further provide training a machine learning model to detect analytes in a sample image, and determine the size of the detected analytes using the method of Example A1 and the structure of the sample holding device described in Example A8.

[0958] Example A14 is a method for monitoring an image-based assay using a column or using a column in an image of a sample for an assay or an image of a monitoring mark to determine the quality of the image holding device and the quality of sample preparation, the method comprising detecting areas corresponding to missing columns or broken columns in the sample holding device, which affects the effective sample volume for the assay; and detecting bubbles and dust in the image of the sample, indicating flaws in the assay operation or defects in the sample holding device.

[0959] In Example A15, the method of Example A14 may further include performing an assay using the image of the column or monitoring marker in the image of the sample to detect and adjust operation of the imager in the image-based assay, including: focusing, contrast stretching, iso adjustment, and filtering.

[0960] Example BA-1 is an intelligent measurement monitoring method, which includes receiving, by a processing device, an image encoding first information of a biological sample deposited in a sample card and second information of multiple monitoring markers; executing, by the processing device, a first machine learning model on the image to determine measurements of geometric features associated with multiple monitoring markers; determining, by the processing device, a change between the measurements of the geometric features and a reference true value of the geometric features provided with the sample card; correcting, by the processing device, the image encoding the first information and the second information based on the change; and using, by the processing device, the corrected image to determine biological characteristics of the biological sample.

[0961] In Example BA-2, the method of Example BA-1 can further provide that the sample card comprises a first plate, a plurality of columns integrated substantially perpendicular to a surface of the first plate, and a second plate capable of surrounding the first plate to form a thin layer in which the biological sample is deposited.

[0962] In Example BA-3, the method of any of Examples BA-1 or BA-2 may further provide that a plurality of monitoring markers correspond to a plurality of columns.

[0963] In Example BA-4, the method of Example BA-3 may also provide that at least two of the multiple columns are separated by a true lateral dimension (TLD), and wherein executing a first machine learning model on the image by the processing device to determine measurements of geometric features associated with multiple monitoring markers includes executing the first machine learning model on the image by the processing device to determine the TLD.

[0964] Example BB-1 is an imaging system comprising a sample card comprising a first plate, a plurality of columns integrated substantially perpendicular to a surface of the first plate, and a second plate capable of surrounding the first plate to form a thin layer in which the biological sample is deposited; and a computing device comprising: a processing device communicatively coupled to an optical sensor to receive from the optical sensor an image encoding first information of the biological sample deposited in the sample card and second information of a plurality of monitoring markers, using a first machine learning model on the image to determine measurements of geometric features associated with the plurality of monitoring markers, determining changes between the measurements of the geometric features and a reference true value of the geometric features provided with the sample card, correcting the image encoding the first information and the second information based on the changes, and determining biological properties of the biological sample based on the corrected image.

[0965] Example DA-1 is a method for measuring the volume of a sample in a thin layer sample card, the method comprising receiving, by a processing device of an image system, an image of a sample card containing a sample and a monitoring standard, wherein the monitoring standard comprises a plurality of columns vertically integrated with a first plate of the sample, and each of the plurality of columns has substantially the same height (H); determining, by the processing device, a plurality of non-sample sub-regions using a machine learning model, wherein the plurality of non-sample sub-regions correspond to at least one of columns, bubbles, or impurity elements; calculating, by the processing device, an area occupied by the sample by removing the plurality of non-sample sub-regions from the image; calculating, by the processing device, the volume of the sample based on the calculated area and height (H); and determining, by the processing device, a biological characteristic of the sample based on the volume.

[0966] A1: A method of using a device to improve an imaging-based assay, comprising:

[0967] a) receiving a sample image of a sample holder, the sample holder comprising a plurality of monitoring structures integrated on a contact surface of at least one plate of the sample holder, wherein the plurality of monitoring structures are placed according to a pattern, and wherein the contact surface contacts a sample comprising a plurality of analytes;

[0968] b) detecting the plurality of monitoring structures in the sample image using a machine learning model;

[0969] c) performing error correction on the detected plurality of monitoring structures using predetermined structural characteristics associated with the plurality of monitoring structures;

[0970] d) determining a true lateral dimension value associated with the sample image based on the detected plurality of monitoring structures;

[0971] e) determining a homographic transformation between the detected positions of the plurality of monitoring structures in the sample image and a predetermined distribution pattern of the plurality of monitoring structures in an actual image plane associated with the sample holder based on the true lateral dimension value;

[0972] f) transforming the sample image containing the plurality of analytes into a corresponding perspective image in an actual image plane based on a homography transformation; and

[0973] g) calculating at least one morphological characteristic associated with at least one of the plurality of analytes in the sample image.

[0974] A2: The method of A1, further comprising:

[0975] detecting a center of each of the plurality of monitoring structures; and

[0976] A homography transformation is determined based on detected centers of a plurality of monitored structures comprising at least 4 non-collinear points.

[0977] A3: A method as in A1, wherein the sample image is divided into non-overlapping image patches, each of which contains at least 4 non-collinear detection centers of the multiple monitoring structures, wherein a patch-specific homography transformation is estimated and applied to compensate the image patch, and for other image patches, a global homography transformation is estimated and applied based on the detected centers of the multiple monitoring structures in the entire image.

[0978] D1: The method of A1, wherein the monitoring structure is a column or a monitoring tag.

[0979] D2: A method as in A1, wherein the machine learning model is trained using a labeled training image set, and wherein the machine learning model is a RetinerNet whose parameters are trained using the labeled training image set.

[0980] D3: The method of A1, wherein the predetermined structural characteristic comprises at least one of a periodicity, a shape, or a size associated with the plurality of monitoring structures.

[0981] D4: The method of A1, wherein the at least one morphological characteristic comprises at least one of a size or a length of one of the plurality of analytes.

[0982] A4: A method for microselective image assay (MSIA) in an image-based assay, comprising:

[0983] a) capturing an image of a sample for assay in the sample holding device, wherein the sample holding device is as in A1(a) and the sample in the sample holding device comprising a known uniform height and an analyte forms a monolayer in a region of interest;

[0984] b) estimating TLD / FoV of the image of the sample according to (a) for use in subsequent estimation of area, size and volume in image-based determination;

[0985] c) detecting defects including bubbles or dust in the image of the sample by means of a trained machine learning model for inspection and segmentation of the defects in the image of the sample;

[0986] d) Estimate the total area of ​​the segmented defects in the sample image using the estimated TLD / FoV from (b) and calculate their actual area size;

[0987] e) estimating the actual volume of the sample corresponding to the total surface area of ​​the detected defects in the image of the sample for determination based on the area estimate from (d) and the known height of the sample in the sample holding device;

[0988] f) subtracting the defect volume estimate from (e) corresponding to the total volume beneath the surface area of ​​the detected defect, removing the surface area of ​​the detected defect in the image of the sample used for the assay, and updating the total volume of the remaining sample; and

[0989] g) Performing image-based assays on the selected region of the updated sample image from (f) and the updated sample volume for micro-selective image assays.

[0990] A5: A method as in A4, wherein micro-selective image assay (MSIA) utilizes the monitoring structure in the sample holding device of A4 to estimate the TLD / FoV of the sample image and the sample image is mapped to its actual size by an embodiment of A1.

[0991] A6: A method as in A4, wherein the microselectivity image assay (MSIA) is based on other selectivity criteria including:

[0992] a) Distribution of defects in the sample, including bubbles and dust;

[0993] b) the location of the pillars and monitoring marks and other artifacts in the image of the sample used for determination;

[0994] c) The distribution and conditions of the analytes in the image of the sample used for the assay, including the conditions for analyte clustering and focusing conditions.

[0995] A7: A method as in A4, wherein a microselective image assay (MSIA) performs a multi-target assay from an image of the sample, wherein the multi-target assay is based on region / zone selection having areas defined by different reagents or sample heights for multiple assay applications from a single image of the sample.

[0996] A8: A method as in any of A4 or A7, wherein the machine learning model is trained on labeled training image samples captured by the imager for defect detection and segmentation in image-based assays.

[0997] A9: A method as in any of A4 or A7, wherein a machine learning model is trained to detect analytes in sample images, and the size of the detected analytes is determined using the structure of the sample holding device described in A1 and A4.

[0998] A10: A method of monitoring an image-based assay using a column or monitoring marker to determine the quality of an image holding device and the quality of sample preparation, the column or monitoring marker using an image of the column or monitoring marker in an image of a sample for an assay, the method comprising:

[0999] a) detecting missing or broken columns in the sample holding device that affect the effective sample volume for the assay;

[1000] b) detecting objects or analytes on the column or monitoring labels, which affect the height of the sample in the sample holding device and thus the effective volume of the sample for the assay; and

[1001] c) Detection of bubbles and dust in the sample image, indicating flaws in the measurement operation or defects in the sample holding device.

[1002] A12: As in the method of A10, the marked image is subjected to an assay to detect and adjust the operation of the imager in the image-based assay, including: focusing, contrast stretching, iso adjustment and filtering.

[1003] definition

[1004] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. Although any methods and materials similar or equivalent to those described herein can also be used to practice or test the present teachings, some exemplary methods and materials are now described.

[1005] "QMAX" (Q: quantification; M: amplification; A: addition of reagents; X: acceleration; also known as self-calibrated compressed open flow (SCOF)) devices, assays, methods, kits and systems are described in: U.S. Provisional Patent Application No. 62 / 202,989 filed on August 10, 2015, U.S. Provisional Patent Application No. 62 / 218,455 filed on September 14, 2015, U.S. Provisional Patent Application No. 62 / 293,188 filed on February 9, 2016, U.S. Provisional Patent Application No. 62 / 305,123 filed on March 8, 2016, and U.S. Provisional Patent Application No. 62 / 369,181 filed on July 31, 2016. U.S. Provisional Patent Application No. 62 / 394,753, filed September 15, 2016, PCT Application (designating the United States) No. PCT / US2016 / 045437, filed August 10, 2016, PCT Application (designating the United States) No. PCT / US2016 / 051775, filed September 14, 2016, PCT Application (designating the United States) No. PCT / US2016 / 051794, filed September 15, 2016, and PCT Application (designating the United States) No. PCT / US2016 / 054025, filed September 27, 2016, all of which are hereby incorporated by reference in their entirety and for all purposes.

[1006] As used herein, the term "sample" refers to a material or material mixture containing one or more analytes or entities of interest. In certain embodiments, a sample can be obtained from a biological sample, such as a cell, tissue, body fluid, and feces. Body fluids of interest include, but are not limited to, amniotic fluid, aqueous humor, vitreous humor, blood (e.g., whole blood, fractionated blood, plasma, serum, etc.), breast milk, cerebrospinal fluid (CSF), cerumen (ear wax), chyle, chyme, endolymph, perilymph, feces, gastric acid, gastric juice, lymph, mucus (including nasal drainage and sputum), pericardial fluid, peritoneal fluid, pleural fluid, pus, inflammatory secretions, saliva, sebum (skin oil), semen, sputum, sweat, synovial fluid, tears, vomitus, urine, and exhaled condensate. In a specific embodiment, a sample can be obtained from a subject, such as a human, and can be processed before being used for subject determination. For example, before analysis, protein / nucleic acid can be extracted from a tissue sample before use, and its method is known. In a specific embodiment, a sample can be a clinical sample, for example, a sample collected from a patient.

[1007] The term "analyte" refers to molecules (e.g., proteins, peptides, DNA, RNA, nucleic acids or other molecules), cells, tissues, viruses and nanoparticles having different shapes. In some embodiments, "analyte" as used herein is any substance suitable for testing in the present method.

[1008] As used herein, "diagnostic sample" refers to any biological sample derived from a subject as a body byproduct, such as a body fluid. The diagnostic sample can be obtained directly from the subject in liquid form, or can be obtained from the subject by first placing the body byproduct in a solution, such as a buffer. Exemplary diagnostic samples include, but are not limited to, saliva, serum, blood, sputum, urine, sweat, tears, semen, feces, breath, biopsy, mucus, etc.

[1009] As used herein, "environmental sample" refers to any sample obtained from the environment. Environmental samples can include liquid samples from rivers, lakes, ponds, oceans, glaciers, icebergs, rain, snow, sewage, reservoirs, tap water, drinking water, etc.; solid samples from soil, compost, sand, rock, concrete, wood, brick, sewage, etc.; and gas samples from air, underwater thermal outlets, industrial waste gas, vehicle exhaust, etc. Typically, before analyzing the sample with the method of the present invention, the sample in non-liquid form is converted into liquid form.

[1010] As used herein, "food sample" refers to any sample suitable for animal consumption (e.g., human consumption). Food samples can include raw materials, cooked foods, plant and animal sources of food, pre-processed foods, and partially or completely processed foods, etc. Typically, samples in non-liquid form are converted to liquid form before the samples are analyzed using the methods of the present invention.

[1011] As used herein, the term "diagnosis" refers to the use of methods or analytes for identifying, predicting the outcome of, and / or predicting the response to treatment of a disease or condition of interest. Diagnosis can include predicting the likelihood or predisposition to having a disease or condition, estimating the severity of a disease or condition, determining the risk of progression of a disease or condition, assessing the clinical response to treatment, and / or predicting the response to treatment.

[1012] As used herein, a "biomarker" is any molecule or compound found in a sample of interest and known to be diagnostic of or associated with the presence or predisposition to a disease or condition of interest in a subject from which the sample was derived. Biomarkers include, but are not limited to, polypeptides or complexes thereof (e.g., antigens, antibodies), nucleic acids (e.g., DNA, miRNA, mRNA), drug metabolites, lipids, carbohydrates, hormones, vitamins, and the like known to be associated with a disease or condition of interest.

[1013] As used herein, "condition" with respect to diagnosing a health condition refers to a physiological state of the mind or body that is distinguished from other physiological states. In some cases, a health condition may not be diagnosed as a disease. Exemplary health conditions of interest include, but are not limited to, nutritional health; aging; exposure to environmental toxins, pesticides, herbicides, synthetic hormone analogs; pregnancy; menopause; menopause; sleep; stress; prediabetes; exercise; fatigue; chemical balance; etc.

[1014] It must be noted that, as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise, such as when the word "single" is used. For example, reference to an "analyte" includes a single analyte and multiple analytes, reference to a "capture agent" includes a single capture agent and multiple capture agents, reference to a "detection agent" includes a single detection agent and multiple detection agents, and reference to a "reagent" includes a single reagent and multiple reagents.

[1015] As used herein, the terms "adapted" and "configured" mean that an element, component, or other subject matter is designed and / or intended to perform a given function. Thus, the use of the terms "adapted" and "configured" should not be interpreted to mean that a given element, component, or other subject matter is simply "capable of" performing a given function. Similarly, a subject matter stated as being configured to perform a particular function may additionally or alternatively be described as being operable to perform that function.

[1016] As used herein, the phrases "at least one" and "one or more" with respect to a list of more than one entity refer to any one or more entities in the list of entities, and are not limited to at least one of each and every entity specifically listed in the list of entities. For example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently, "at least one of A and / or B") may refer to A alone, B alone, or A and B in combination.

[1017] As used herein, the term "and / or" placed between a first entity and a second entity means one of (1) the first entity, (2) the second entity, and (3) the first entity and the second entity. Multiple entities listed using "and / or" should be interpreted in the same manner, i.e., "one or more" of the entities so combined. In addition to the entities specifically identified by the "and / or" clause, other entities may optionally be present, whether related or unrelated to those entities specifically identified.

[1018] When numerical ranges are mentioned herein, the present invention includes embodiments in which the endpoints are included, embodiments in which both endpoints are excluded, and embodiments in which one endpoint is included but the other is excluded. Both endpoints should be assumed to be included unless otherwise stated. Furthermore, unless otherwise stated or apparent to one of ordinary skill in the art from context and understanding.

Claims

1. A system for improving analysis of images of microfeatures in a sample, the system comprising: (a) An apparatus for measuring micro features in a sample using an imager, having a solid surface comprising a sample contact area for contacting a sample comprising microfeatures; and One or more monitoring markers, wherein the monitoring markers: i. Made of a different material from the sample; ii. being located inside the sample during determination of the microstructure, wherein the sample forms a thin layer having a thickness of less than 200 um on the sample contact area; iii. Having a lateral linear dimension of 1 um or greater, and iv. having at least one lateral linear dimension of 300um or less; as well as v. The monitoring mark and the spacer are different structures, wherein the spacer is a structure between the first plate and the second plate, used to adjust the spacing between the plates, each monitoring mark comprises a protrusion or groove from the sample contact area of ​​the plate, wherein the protrusion or groove comprises a flat surface substantially parallel to an adjacent surface, and the adjacent surface is a portion of the sample contact area adjacent to the protrusion or groove; and wherein during the determination, the imager images at least one monitoring marker; which is used during the determination of said analyte; and the geometrical parameters of the monitoring marker are: predetermined and known prior to the determination of the analyte and used as parameters in an algorithm for determining a property associated with the microfeature; (b) an imager for determining whether a sample contains or is suspected of containing a microfeature by taking one or more images of the sample and monitoring the marker; and (c) a non-transitory computer readable medium containing an algorithm which, when executed, utilizes the monitoring markers as parameters in conjunction with an imaging processing method to improve the image quality of at least one of the images.

2. The system of claim 1, further comprising: A second computer-readable medium storing a machine learning model, wherein the machine learning model is configured to transform an image captured by the optical imaging system into a transformed image, and, after correcting the image of at least a portion of the sample by a correction algorithm, performing training of the machine learning model and / or image transformation.

3. A method for measuring microfeatures in a sample using an imager, comprising: Providing the system of claim 1; The sample and the monitoring markers are imaged using an imager, and an algorithm stored in a non-transitory computer readable medium is used to improve the image quality of at least one of the images using the monitoring markers as parameters in conjunction with an imaging processing method.

4. The method of claim 3, further comprising: performing an image transformation on an image of at least a portion of the sample specimen using a machine learning model; and using the transformed image to measure the analyte; The training of the machine learning model and / or the image transformation is performed after the images of at least a portion of the samples are corrected by a correction algorithm.

5. The system of claim 1, wherein the algorithm corrects local image distortions of an image of at least a portion of the sample.

6. The system according to claim 1, wherein: The algorithm corrects for local illumination variations in an image of at least a portion of the sample.

7. The system of claim 1, wherein the algorithm is a machine learning model.

8. The system of claim 1, wherein the algorithm is computer vision.

9. The system of claim 1, wherein the algorithm is a combination of machine learning and computer vision.

10. The system of claim 1, wherein the monitoring mark is a periodic structure having a predetermined period. The system according to claim 1 , wherein each monitoring mark is a column with a flat top surface.

12. The system according to claim 1, wherein the distance between the monitoring marks is equal to or less than 120 um.

13. The system of claim 1, wherein the distance between monitoring marks is in the range of 120 um to 200 um.

14. The system according to claim 1, wherein the monitoring mark is a cylindrical structure with a cross-section of a triangle, a rectangle, a square, a circle or a polygon.

15. The system of claim 1, wherein the analyte is a protein, a peptide, DNA, RNA, a nucleic acid, a small molecule, a cell, or a nanoparticle.

16. The system of claim 1, wherein the analyte is a blood cell.

17. The system of claim 1, wherein the algorithm comprises a homography transformation.

18. The system of claim 1, wherein the monitoring marker comprises at least 4 non-collinear columns.

19. The system of claim 1, wherein each monitoring marker is a pillar having a flat top surface with an area of ​​10,000 um^2 or less.

20. The system of claim 1, wherein each monitoring structure has sharp edges.

21. The system according to claim 1, wherein the geometric parameters of the monitoring mark include the shape and size of the monitoring mark and / or the center distance between the monitoring marks.

22. The method of claim 3, wherein the algorithm corrects local image distortions of an image of at least a portion of the sample.

23. The method according to claim 3, wherein: The algorithm corrects for local illumination variations in an image of at least a portion of the sample.

24. The method of claim 3, wherein the algorithm is a machine learning model.

25. The method of claim 3, wherein the algorithm is computer vision.

26. The method of claim 3 wherein the algorithm is a combination of machine learning and computer vision.

27. The method of claim 3 wherein the monitoring mark is a periodic structure having a predetermined period.

28. The method of claim 3, wherein each monitoring mark is a column with a flat top surface.

29. The method according to claim 3, wherein the distance between the monitoring marks is equal to or less than 120 um.

30. The method of claim 3 wherein the distance between monitoring marks is in the range of 120 um to 200 um.

31. The method according to claim 3, wherein the monitoring mark is a cylindrical structure with a cross-section of a triangle, rectangle, square, circle or polygon.

32. The method of claim 3 wherein the analyte is a protein, peptide, DNA, RNA, nucleic acid, small molecule, cell or nanoparticle.

33. The method of claim 3 wherein the analyte is a blood cell.

34. The method of claim 3 wherein the algorithm comprises a homography transformation.

35. The method of claim 3 wherein the monitoring marker comprises at least 4 non-collinear columns.

36. The method of claim 3 wherein each monitoring mark is a pillar having a flat top surface with an area of ​​10,000 um^2 or less.

37. The method of claim 3 wherein each monitoring structure has sharp edges.

38. The method of claim 3 wherein monitoring geometric parameters of the markers comprises monitoring the shape and size of the markers and / or monitoring the center distances between the markers.

Images