Method and system for particle classification - Patents.com

Abstract

Methods and systems for particle classification are provided herein. Multiple images of a fluid / sample, such as a pharmaceutical, drug, or medication, may be analyzed using one or more machine learning models. The one or more machine learning models may include, for example, a convolutional neural network. The one or more machine learning models may identify and classify one or more sub-visible particles in each image. The one or more sub-visible particles may include, for example, cells, pathogens, protein aggregates, silicone oil droplets, fibers, bubbles, glass particles, combinations thereof, and / or the like.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 289,489, filed December 14, 2021, and U.S. Provisional Patent Application No. 63 / 341,775, filed May 13, 2022, both of which are incorporated by reference in their entireties. [Background technology]

[0002] Monitoring of subvisible particles (SVPs) in pharmaceutical products is important due to the immunogenicity of certain SVP types. Because certain SVPs may represent aggregates of thousands to millions of molecules, the presence of SVPs in pharmaceutical products may adversely affect clinical performance as well as patient safety. Existing methods and systems do not efficiently identify certain types and sizes of SVPs in pharmaceutical products. These and other considerations are discussed herein. Summary of the Invention

[0003] It should be understood that both the following general description and the following detailed description are exemplary and explanatory only, and not limiting. Methods and systems for particle classification are described herein. These methods and systems can be used to detect and classify sub-visible particles ("SVPs") that may be present in a range of pharmaceutical products (e.g., drugs, medications, antibody formulations, etc.). For example, one or more images of a sample of the pharmaceutical product can be captured by a flow imaging microscope ("FIM") system. The FIM system can detect any SVPs present in the sample. The FIM system can generate metadata indicating the location of each detected SVP in the one or more images. Additionally or alternatively, another system or device can detect the SVPs (e.g., using a segmentation algorithm).

[0004] The one or more images may be analyzed using a machine learning model (e.g., a classifier). The machine learning model may include a convolutional neural network (CNN). The CNN may receive as input the one or more images (and optionally metadata) and classify each SVP in the one or more images. The CNN may classify each SVP according to size, type, etc. Additional advantages will be set forth in part in the description that follows or may be learned by practice. These advantages will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. [Brief description of the drawings]

[0005] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments of the disclosed methods and configurations and, together with the description, serve to explain the principles of the disclosed methods and configurations. [Figure 1] 1 illustrates an exemplary system. [Diagram 2] An example of a sub-visible particle is shown. [Figure 3A] 1 shows an example image. [Figure 3B] 1 shows an example image. [Figure 4A] 1 shows an example image. [Figure 4B] 1 shows an example image. [Figure 5A] 1 shows an example image. [Figure 5B] 1 shows an example image. [Figure 6] 1 illustrates an exemplary system. [Figure 7] 1 illustrates an exemplary training workflow. [Figure 8] 1 illustrates an exemplary convolutional neural network. [Figure 9] 1 illustrates an exemplary classification workflow. [Figure 10] 1 shows an exemplary table. [Figure 11] 1 shows an exemplary table. [Figure 12]1 shows an example image and feature map. [Figure 13] 1 shows an exemplary graph. [Figure 14A] 1 shows an exemplary table. [Figure 14B] 1 shows an exemplary table. [Figure 14C] 1 shows an exemplary table. [Figure 14D] 1 shows an exemplary table. [Figure 15] 1 shows an exemplary graph. [Figure 16] 1 shows an exemplary graph. [Figure 17] 1 illustrates an exemplary system. [Figure 18] 1 shows a flow chart of an exemplary method. [Figure 19] 1 shows a flow chart of an exemplary method. [Figure 20] 1 shows a flow chart of an exemplary method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context otherwise dictates. Ranges may be expressed herein as from "about" one particular value, and / or to "about" another particular value. When such a range is expressed, the alternative includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, it will be understood that the particular value forms the alternative by use of the antecedent "about." It will be further understood that the endpoints of each of these ranges are significant in relation to the other endpoint, and independently of the other endpoint.

[0007] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes cases where the event or circumstance occurs and cases where the event or circumstance does not occur.

[0008] Throughout this description and claims, the word "comprise" and variations of this word, such as "comprising" and "comprises," mean "including, but not limited to," and are not intended to exclude, for example, other components, integers, or steps. "Exemplary" means "one example of," and is not intended to convey an indication of a preferred or ideal configuration. "Etc." is not used in a limiting sense, but rather for descriptive purposes.

[0009] Where combinations, subsets, interactions, groups, etc. of components are disclosed, it is understood that each is specifically contemplated and described herein, although specific reference to each of the various individual and collective combinations and permutations of these components may not be explicitly described. This applies to all parts of this application, including but not limited to steps in the described methods. Thus, where there are various additional steps that may be performed, it is understood that each of these additional steps may be performed in any particular configuration or combination of configurations of the described methods.

[0010] As will be appreciated by those skilled in the art, hardware, software, or a combination of software and hardware may be implemented. Additionally, a computer program product on a computer-readable storage medium (e.g., non-transitory) having processor-executable instructions (e.g., computer software) embodied in the storage medium may be implemented. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, optical storage devices, magnetic storage devices, memory resistors, non-volatile random access memory (NVRAM), flash memory, or combinations thereof.

[0011] Throughout this application, reference is made to block diagrams and flow charts. It will be understood that each block of the block diagrams and flow charts, and combinations of blocks in the block diagrams and flow charts, respectively, can be implemented by processor-executable instructions. These processor-executable instructions can be loaded into a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, such that the processor-executable instructions executing on the computer or other programmable data processing apparatus create a device for implementing the function(s) specified in the flow chart block or blocks.

[0012] These processor-executable instructions may also be stored in a computer-readable memory that may direct a computer or other programmable data processing apparatus to function in a particular manner, such that the processor-executable instructions stored in the computer-readable memory create an article of manufacture that includes processor-executable instructions for implementing a function identified in one or more blocks of the flowchart. The processor-executable instructions may also be loaded into a computer or other programmable data processing apparatus and a series of operational steps performed on the computer or other programmable apparatus to create a computer-implemented process, such that the processor-executable instructions executing on the computer or other programmable apparatus provide steps for implementing a function identified in one or more blocks of the flowchart.

[0013] The blocks of the block diagrams and flowcharts support combinations of devices for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowcharts, and combinations of blocks of the block diagrams and flowcharts, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or a combination of special purpose hardware and computer instructions.

[0014] Methods and systems for particle classification are described herein. The methods and systems may combine high-throughput flow imaging techniques and machine learning for a variety of medical and pharmaceutical applications. Such applications include detection and classification of contaminants and / or unexpected particles in images of pharmaceutical (e.g., pharmaceutical, drug, antibody formulations, etc.) fluids / samples. The images may be received from a flow imaging microscope ("FIM") system. The images may be analyzed using one or more machine learning models, such as one or more convolutional neural networks (CNNs).

[0015] The image (hereinafter, "FIM image") may contain subvisible particles ("SVPs") of various sizes and types that may not be observable alone (e.g., without the aid of a microscope or other instrument). Exemplary SVPs include, but are not limited to, cells, pathogens, protein aggregates, silicone oil droplets, fibers, bubbles, glass particles, combinations thereof, and / or the like. The presence of SVPs in a pharmaceutical product may be undesirable for efficacy, safety, standardization, and the like. Thus, the present method and system may detect and classify SVPs that may be present in a pharmaceutical product.

[0016] Each SVP in each FIM image may be classified using one or more machine learning models. The FIM images are associated with metadata indicating the location of each unclassified particle in each FIM image, which may include non-SVPs. The metadata may have been generated by the FIM system that captured the FIM image. Additionally or alternatively, another system or device may detect particles in the FIM images (e.g., using a segmentation algorithm). For example, the location of each particle in each FIM image may be determined using one or more image segmentation algorithms.

[0017] Each unclassified particle in the FIM image may be initially classified as an SVP (or a particular SVP type) or a non-SVP by one or more machine learning models. Additionally or alternatively, the initial classification and / or location of each SVP and / or each non-SVP in each FIM image may be indicated by metadata. Non-SVPs in the FIM image may be ignored, while SVPs in the FIM image may be further classified according to size and / or type. For example, each of the SVPs may be classified based on its equivalent circular diameter (ECD) size. The SVPs may be further classified by one or more machine learning models according to a plurality of features for each of a plurality of SVP types. For example, one or more machine learning models may be trained to classify at least one SVP based on one or more features of the FIM image that indicate a particular type of SVP (e.g., features indicative of a protein particle).

[0018] The one or more machine learning models may include a CNN. The CNN may comprise at least one filter that may be applied to each FIM image. The at least one filter may include a size of, for example, 3×3 pixels. In some embodiments, each FIM image for training and / or testing may be pre-processed before being provided to the CNN. For example, each FIM image may be resized to a uniform size, such as 64×64 pixels. Other embodiments are possible. At least one filter may be applied to each resized FIM image by the CNN.

[0019] The CNN may include hyperparameters and at least one activation function for each hidden layer. The hyperparameters may include, for example, batch size, dropout rate, number of epochs, kernel size, stride, padding, etc. The at least one activation function may include, for example, a rectified linear unit activation function or a hyperbolic tangent activation function. Other implementations are possible.

[0020] The CNN may be trained using a training dataset. The training dataset may include a plurality of FIM images and corresponding metadata. This corresponding metadata may indicate an initial classification (e.g., SVP or non-SVP, SVP type A or SVP type B, etc.) and / or a location of each particle in each FIM image. It should be understood that in some embodiments, the CNN may be trained without corresponding metadata (e.g., using only a plurality of FIM images). The location of at least one particle in each FIM image may also be determined by one or more image segmentation algorithms (e.g., agglomerative clustering, water flow transformation, etc.) used by the CNN and / or another model. The plurality of FIM images in the training dataset may include multiple SVP types and / or sizes. In some embodiments, each FIM image of the training dataset may include only one SVP, and / or only one SVP type, and / or only one SVP size. In other embodiments, each FIM image of the training dataset may include multiple SVPs and / or multiple SVP types and / or multiple SVP sizes. At least one filter may be applied by the CNN to each FIM image of the training dataset to determine a plurality of features corresponding to each of a plurality of SVP types. Prior to training on the training dataset, at least one parameter of the CNN may be initialized with a value of the at least one parameter associated with another CNN trained on another training dataset (e.g., "transfer learning").

[0021] The CNN may be tested using a test dataset. The test dataset may include a number of FIM images and corresponding metadata. This corresponding metadata may indicate an initial classification (e.g., SVP or non-SVP, SVP type A or SVP type B, etc.) and / or a location of each particle in each FIM image. It should be understood that in some embodiments, the CNN may be tested without corresponding metadata (e.g., using only a number of FIM images). The location of at least one particle in each FIM image may also be determined by one or more image segmentation algorithms (e.g., agglomerative clustering, water flow transformation, etc.) used by the CNN and / or another model. The number of FIM images in the test dataset may include a number of SVP types and / or sizes, which may or may not be different from the types and / or sizes in the training dataset. In some embodiments, each FIM image in the test dataset may include only one SVP, and / or only one SVP type, and / or only one SVP size. In other embodiments, each FIM image of the test data set may include multiple SVPs and / or multiple SVP types and / or multiple SVP sizes.

[0022] The CNN may classify each SVP in each FIM image in the test data set. For example, at least one filter may be applied to each FIM image in the test data set at each layer of the CNN. The CNN may classify at least one SVP in each FIM image in the test data set based on one or more features indicative of a type of at least one SVP. The classification may include a binary classification such as "protein or not protein," "silicon or not silicon," "protein or fiber," or a multi-class classification such as "silicon, or air bubble, or glass piece." The classification may include a score or confidence level that at least one SVP is of a particular type (e.g., 90% confidence that at least one SVP is a protein particle). Other implementations are possible.

[0023] Referring now to FIG. 1, an exemplary system 100 for particle classification is shown. The system 100 may include a computing device 102 that may be part of a flow imaging microscope ("FIM") system. The computing device 102 may include an acquisition module 104 and an image processing module 106. The acquisition module 104 may include an imaging device, such as a camera and / or microscope configured for flow imaging microscopy, a still image camera, a video camera, an infrared camera, an optical sensor, combinations thereof, and / or the like. The computing device 102 may communicate with a client device 112 and / or a server 110 via a network 106. In some embodiments, the client device 112 and the server 110 may be a single device (not shown). The network 106 may be configured to use various network paths, protocols, devices, and / or the like for communication. The network 106 may have multiple communication links connecting each of the devices shown in the system 100. Network 106 may be a fiber optic network, a coaxial cable network, a hybrid fiber coaxial network, a wireless network, a satellite system, a direct broadcast system, an Ethernet network, a High Definition Multimedia Interface network, a Universal Serial Bus (USB) network, or any combination thereof.

[0024] The client device 112 may comprise a computing device, a mobile device, a smart device, a combination thereof, and / or the like. The server 110 may comprise one or more computing devices, including a storage module 110A (e.g., a storage medium), an image processing module 110B, and a machine learning module 110C. The client device 112 may comprise an application module 113 configured to communicate with and / or control aspects of the server 110. For example, the application module 113 may be configured to cause the server 110 (and its modules) to perform the steps and processes described herein.

[0025] The system 100 may be configured for classification of sub-visible particles ("SVPs"). As used herein, the term "particle" refers to any portion of a substance or material, such as protein or silicon aggregates, dust, mold spores, cells, cell clusters, fibers, small clumps of matter, organisms, tissues, biological matter, minerals, air bubbles, glass fragments, combinations thereof, and / or the like. The computing device 102 may capture images of the sample / fluid containing SVPs via the collection module 104 and image processing module 106 using high throughput flow imaging techniques. For example, the computing device 102 may be configured to capture images of the sample / fluid that may be pumped through a flow cell of a FIM system (not shown). In some embodiments, the collection module 104 may include a digital microscope that may capture images including particles within the sample / fluid. The collection module 104 may capture successive images / frames of each sample / fluid as it flows through a flow cell (not shown).

[0026] The sample / fluid may include a pharmaceutical product (e.g., a drug, a drug, an antibody formulation, etc.). Such an image (hereinafter, a "FIM image") may include various types of SVPs that may not be observable alone (e.g., without the aid of a microscope or other instrument). Exemplary SVPs include, but are not limited to, cells, pathogens, protein aggregates, silicon oil, fibers, bubbles, glass particles, combinations thereof, and / or the like. The presence of SVPs in a pharmaceutical product may be undesirable for efficacy, safety, standardization, etc. Thus, the system 100 may be configured to detect and classify SVPs in the FIM image.

[0027] The FIM images captured by the collection module 104 of the computing device 102 may be appended with metadata (or separate metadata may be generated) by the image processing module 106. For example, the image processing module 106 may append (or separately generate) metadata to the FIM images indicating the number of particles in each FIM image that potentially contain non-SVPs, and / or an indication of each location of each particle in each FIM image. The FIM images and metadata may be transmitted over the network 108 to the server 110 for storage. For example, the computing device 102 may transmit the FIM images and metadata to the storage module 110A of the server 110 for storage. As described further herein, the client device 112 may have the machine learning module 110C of the server 110 classify SVPs in the FIM images, for example, based on the FIM images themselves and the associated metadata.

[0028] Referring now to FIG. 2, exemplary images 202A, 202B, 204A, 204B, 206A, 206B depicting protein and silicon SVPs of various sizes are shown. SVPs can be classified based on their equivalent circular diameter (ECD) size. The ECD of an SVP may be measured, for example, in micrometers (μm). The ECD of an SVP depicted in the FIM images described herein can range, for example, from approximately 1.125 μm to greater than 130 μm. Exemplary ECD sizes can include very small ECD (<2 μm) (not shown), small ECD (>=2 μm and <10 μm) (202A, 202B), medium ECD (>=10 μm and <=25 μm) (204A, 204B), and large ECD (>=25 μm) (206A, 206B). Other example sizes are possible.

[0029] As described further herein, the client device 112 may have the machine learning module 110C of the server 110 classify SVPs in the FIM images, for example, based on the FIM images themselves and associated metadata. It should be understood that in some embodiments, the machine learning module 110C may classify SVPs in the FIM images without corresponding metadata (e.g., using only the FIM images). For purposes of explanation, the description herein describes the imaging processing module 110B of the server 110 as a module that receives and analyzes FIM images, but it should be understood that the image processing module 106 may similarly receive and process any of the FIM images captured by the computing device 102.

[0030] Machine learning module 110C may include one or more machine learning models, artificial intelligence models, combinations thereof, and / or the like. For example, machine learning module 110C may use or include a convolutional neural network, an image classification model, a segmentation model, a statistical algorithm, and the like. In some embodiments, machine learning module 110C (or another module or device of system 100) may classify any particles in the FIM image that are not indicated by associated metadata. For example, particles in the FIM image may be initially classified by machine learning module 110C as being SVPs (or a particular SVP type) or non-SVPs. Additionally or alternatively, the initial classification and / or location of each SVP and / or each non-SVP in each FIM image may be indicated by associated metadata. Once detected / classified, non-SVPs in the FIM image may be ignored, while SVPs in the FIM image may be further classified / identified by type (e.g., protein or silicon). For example, as described further herein, the machine learning module 110C may be configured (e.g., trained) to classify SVPs in the FIM images based on features indicative of the type of each SVP. The classification may be a binary classification such as "protein or not protein," "silicon or not silicon," "protein or fiber," or a multi-class classification such as "silicon, or air bubbles, or glass pieces." The classification may include a score or confidence level that the corresponding SVP is of a particular type (e.g., 90% confidence that the SVP is a protein particle). Other implementations are possible.

[0031] Exemplary FIM images are shown in Figures 3A-4B. Figures 3A and 3B show exemplary FIM images 302 and 304. Each of the FIM images 302 and 304 includes protein SVPs 302A and 304A, respectively, that can be classified by the machine learning module 110C using methods described further herein. Figures 4A and 4B show exemplary FIM images 402 and 404. The FIM image 404 may not include any detectable SVPs. However, the FIM image 402 may include one or more silicon SVPs 406A, which can be classified by the machine learning module 110C using methods described further herein.

[0032] The machine learning module 110C may be configured to detect SVPs in the FIM image using at least one filter, as described further herein. The at least one filter may be considered a "sliding window" that displays / analyzes each FIM image one portion at a time. FIG. 5A illustrates an example FIM image 502 and a sliding window 504 (e.g., at least one filter). The machine learning module 110C may use the sliding window 504 to classify SVPs that may be present in each portion of the FIM image 502 across the FIM image 502. For example, the machine learning module 110C may have the image processing module 110B analyze the FIM image 502 using the sliding window 504 and one or more segmentation algorithms / techniques to detect multiple SVPs within multiple regions of the FIM image 502. As another example, the machine learning module 110C may use metadata associated with the FIM image 502 to have the image processing module 110B analyze a particular portion of the FIM image 502 using the sliding window 504. A particular portion of the FIM image 502 may (or may not) be indicated by metadata and may correspond to the location of an SVP.

[0033] An exemplary result of image processing module 110B analyzing FIM image 502 using sliding window 504 is shown in FIG. 5B as output image 506. At least one filter may be considered a "sliding window" that displays / analyzes each FIM image one portion at a time. For example, sliding window 504 may start at a corner of FIM image 502 and output an indication of any SVPs within the region of FIM image 502. Classification may include an indication of the presence and / or type of SVP. Sliding window 504 may "loop" or "walk" over each portion of FIM image 502 and may indicate regions 508 where SVPs are present.

[0034] In some embodiments, a cropped image including each of the regions 508 may be generated by image processing module 110B and stored by storage module 110A for further analysis (e.g., SVP type classification) by machine learning module 110C. In other embodiments, an annotated / labeled image including each of the regions 508 may be generated by image processing module 110B and stored by storage module 110A for further analysis (e.g., SVP type classification) by machine learning module 110C.

[0035] 6, a system 600 for training a machine learning module 630 is shown. The machine learning module 630 may include the machine learning module 110C. The machine learning module 630 may be trained by a training module 620 of the system 600 to classify SVPs in FIM images. For example, the training module 620 may use machine learning ("ML") techniques to train the ML module 630 based on an analysis of one or more training datasets 610. In some embodiments, prior to training with the training dataset 610, at least one parameter of the machine learning module 630, such as a batch size, a dropout rate, a number of epochs, a kernel size, a stride, padding, etc., may be initialized with a value of that at least one parameter associated with another machine learning module (e.g., another ML model, another CNN, etc.) trained with another training dataset (e.g., "transfer learning").

[0036] The training data set 610 may include any number of data sets or subsets 610-610N. For example, the training data set 610 may include a first training data set 610A and a second training data set 610B. The first training data set 610A may include a first plurality of FIM images. As shown in FIG. 6, the first training data set 610A (e.g., the first plurality of FIM images) may include multiple SVP sizes (e.g., from very small ECD to large ECD). In some embodiments, each FIM image of the first plurality of FIM images may include only one SVP, and / or only one SVP type, and / or only one SVP size. In other embodiments, each of the first plurality of FIM images may include multiple SVPs and / or multiple SVP types and / or multiple SVP sizes. The second training data set 610B may include a second plurality of FIM images. The second plurality of FIM images may include multiple SVP sizes (e.g., from very small to large ECDs), which may or may not be different from the sizes of the first plurality of FIM images. In some embodiments, each FIM image of the second plurality of FIM images may include only one SVP, and / or only one SVP type, and / or only one SVP size. In other embodiments, the second plurality of FIM images may each include multiple SVPs and / or multiple SVP types and / or multiple SVP sizes.

[0037] A subset of either or both of the first training data set 610B or the second training data set 610B may be randomly assigned to the testing data set. In some implementations, the assignment to the testing data set may not be completely random. In this case, one or more criteria may be used during the assignment. In general, any suitable method may be used to assign data to the testing data set while ensuring that the distribution of FIM images containing SVPs of a particular size and / or type are appropriately assigned for training and testing purposes.

[0038] The training module 620 may train the ML module 630 by extracting feature sets from the FIM images of the training dataset 610 according to one or more feature selection techniques. For example, the training module 620 may train the ML module 630 by extracting a feature set from the training dataset 610 that includes statistically significant features. The training module 620 may extract the feature set from the training dataset 610 in various ways. The training module 620 may perform feature extraction multiple times, each time using a different feature extraction technique. In an embodiment, the feature sets generated using the different techniques may each be used to generate a different machine learning based classification model 640A-640N. For example, the feature set with the highest quality metric may be selected for use in training. The training module 620 may use the feature sets to build one or more machine learning based classification models 640A-640N that are configured to classify various SVPs.

[0039] The training data set 610 may be analyzed to determine any dependencies, associations, and / or correlations between the determined features in the unlabeled FIM images (e.g., those not indicating SVP type and / or presence) and the features of the labeled FIM images in the training data set 610. The identified correlations may have the form of a list of features. As used herein, the term "feature" may refer to any feature of an item of data that may be used to determine whether an item of data is within one or more particular categories. The feature selection technique may include one or more feature selection rules. The one or more feature selection rules may include feature generation rules. The feature generation rules may include determining which features occur a threshold number of times in the training data set 610 and identifying those features that meet the threshold as features.

[0040] A single feature selection rule may be applied to select features, or multiple feature selection rules may be applied to select features. Feature selection rules may be applied in a cascading manner, where feature selection rules are applied in a particular order and applied to the results of previous rules. For example, feature generation rules may be applied to the training dataset 610 to generate a first list of features. The final list of features may be analyzed by additional feature selection techniques to determine one or more feature sets (e.g., a set of features that can be used to classify SVPs). Any suitable computational technique may be used to identify the feature sets using any feature selection technique, such as filter methods, wrapper methods, and / or embedding methods. One or more feature sets may be selected according to a filter method. Filter methods include, for example, Pearson's correlation, linear discriminant analysis, analysis of variance (ANOVA), chi-square, combinations thereof, and the like. The selection of features according to a filter method is independent of any machine learning algorithm. Instead, features may be selected based on scores in various statistical tests for correlation with outcome variables.

[0041] As another example, the one or more feature sets may be selected according to a wrapper method. The wrapper method may be configured to use a subset of features and train the ML module 630 using the subset of features. Features may be added and / or reduced from the subset based on inferences drawn from previous models. Wrapper methods include, for example, forward feature selection, backward feature reduction, recursive feature reduction, combinations thereof, and the like. As an example, forward feature selection may be used to identify the one or more feature sets. Forward feature selection is an iterative method that starts with no features in the corresponding machine learning model. In each iteration, the feature that best improves the model is added until the addition of a new variable no longer improves the performance of the machine learning model. As an example, backward reduction may be used to identify the one or more feature sets. Backward reduction is an iterative method that starts with all features in the machine learning model. In each iteration, the lowest ranking features are removed until no improvement is observed upon feature removal. Recursive feature reduction may be used to identify the one or more feature sets. Recursive feature reduction is a greedy optimization algorithm that aims to find the best performing feature subset. Recursive feature reduction creates a model iteratively, setting aside the best or worst performing features at each iteration. Recursive feature reduction builds the next model with the remaining features until all features are exhausted. Recursive feature reduction then ranks the features based on their order of reduction.

[0042] As a further example, one or more feature sets may be selected according to an embedding method that combines the qualities of filter and wrapper methods. Embedding methods include, for example, least absolute shrinkage and selection operator (LASSO) and ridge regression, which implement a penalty function to reduce overfitting. For example, LASSO regression implements L1 regularization that adds a penalty equivalent to the absolute value of the coefficient magnitude, and ridge regression implements L2 regularization that adds a penalty equivalent to the square of the coefficient magnitude.

[0043] After the feature set is generated by the training module 620, the training module 620 may generate a machine learning based classification model 640 based on the feature set. A machine learning based classification model may refer to a complex mathematical model for data classification generated using machine learning techniques. In one embodiment, the machine learning based classification model 640 may include a map of support vectors representing boundary features. As an example, the boundary features may be selected from and / or represent the highest ranked features in a feature set. The training module 620 may use the feature sets determined or extracted from the training dataset 610 to build the machine learning based classification models 640A-640N. In some embodiments, the machine learning based classification models 640A-640N may be combined into a single machine learning based classification model 640. Similarly, the ML module 630 may represent a single classifier containing single or multiple machine learning based classification models 640 and / or multiple classifiers containing single or multiple machine learning based classification models 640.

[0044] The features may be combined in a classification model trained using machine learning approaches, such as discriminant analysis; decision trees; nearest neighbor (NN) algorithms (e.g., k-NN models, replicator NN models, etc.); statistical algorithms (e.g., Bayesian networks, etc.); clustering algorithms (e.g., k-means, mean shift, etc.); neural networks (e.g., reservoir networks, artificial neural networks, etc.); support vector machines (SVMs); logistic regression algorithms; linear regression algorithms; Markov models or chains; principal component analysis (PCA) (e.g., for linear models); multi-layer perceptron (MLP) ANN (e.g., for non-linear models); reservoir network replication (e.g., for non-linear models, typically for time series); random forest classification; combinations thereof and / or the like. The resulting ML module 630 may include a decision rule or mapping for each feature of each FIM image in the training dataset 610 that may be used to classify SVPs in other FIM images. In an embodiment, the training module 620 may train the machine learning based classification model 640 as a convolutional neural network (CNN), which is further described herein with respect to FIG. 8.

[0045] The feature and ML module 630 may be used to detect and / or classify SVPs in FIM images in the test dataset. In one embodiment, the prediction / result for each detected / classified SVP includes a confidence level corresponding to the likelihood or probability that each derived feature is associated with a particular SVP type and / or size. The confidence level may be a value between 0 and 1. In one embodiment, when there are two states (e.g., SVP or no SVP, SVP type A vs. type B, etc.), the confidence level may correspond to a value p, which refers to the likelihood that a particular detected / classified SVP is actually an SVP. In this case, the value 1-p may refer to the likelihood that a particular detected / classified SVP belongs to the second state (e.g., not actually an SVP). In general, multiple confidence levels may be provided for each detected / classified SVP in the test dataset, and for each feature when there are more than two states.

[0046] FIG. 7 is a flow chart illustrating an example training method 700 for generating the ML module 630 using the training module 620. The training module 620 may implement supervised, unsupervised, and / or semi-supervised (e.g., reinforcement-based) machine learning based classification models 640. The method 700 illustrated in FIG. 7 is an example of a supervised learning method; variations of this example of a training method are discussed below, however, other training methods may be similarly implemented to train unsupervised and / or semi-supervised machine learning models. The training method 700 may determine (e.g., access, receive, retrieve, etc.) data at step 710. The data may include FIM images and associated metadata. It should be understood that in some examples, the data may not include metadata and the training described herein may not use the corresponding metadata.

[0047] The training method 700 may generate a training data set and a test data set in step 720. The training data set and the test data set may be generated by randomly assigning FIM images (or a portion thereof) to either the training data set or the test data set. In some implementations, the assignment of FIM images (or a portion thereof) as training data or test data may not be completely random. As an example, a majority (or a portion thereof) of the FIM images may be used to generate the training data set. For example, 75% of the FIM images (or a portion thereof) may be used to generate the training data set and 25% may be used to generate the test data set. In another example, 80% of the FIM images (or a portion thereof) may be used to generate the training data set and 20% may be used to generate the test data set.

[0048] The training method 700 may determine (e.g., extract, select, etc.) one or more features in step 730 that can be used by a classifier to distinguish among different classes of detected SVPs, for example. As an example, the training method 700 may determine the set of features from a FIM image (or a portion thereof). In a further example, the set of features may be determined from data other than the FIM image (or a portion thereof), either in the training data set or the test data set. Such a FIM image (or a portion thereof) may be used to determine an initial set of features, which may be further reduced using the training data set.

[0049] The training method 700 may train one or more machine learning models at step 740 using one or more features. In one embodiment, the machine learning models may be trained using supervised learning. In another embodiment, other machine learning techniques may be employed, including unsupervised learning and semi-supervised. The machine learning models trained at 740 may be selected based on different criteria depending on the problem being solved and / or the data available in the training dataset. For example, machine learning classifiers may be subject to different degrees of bias. Thus, two or more machine learning models may be trained at 740 and optimized, improved, and cross-validated at 750.

[0050] The training method 700 may select one or more machine learning models to build a predictive model at 760. The predictive model may be evaluated using a test dataset. The predictive model may analyze the test dataset at step 770 to generate predicted SVPs present in the FIM image (or a portion thereof). The predicted SVPs present in the FIM image (or a portion thereof) may be evaluated at step 780 to determine whether such values ​​achieved a desired level of accuracy. The performance of the predictive model may be evaluated in a number of ways based on a number of true positive, false positive, true negative, and / or false negative classifications of the multiple data points represented by the predictive model.

[0051] For example, a false positive of a predictive model may refer to the number of times the predictive model misclassifies a particle in an FIM image as an SVP (or a particular SVP type) that is not actually an SVP (or is not a particular SVP type). Conversely, a false negative of a predictive model may refer to the number of times the machine learning model classifies a particle in an FIM image as not being an SVP when in fact the particle was a true SVP. True negatives and true positives may refer to the number of times the predictive model correctly classifies one or more particles in one or more FIM images as an SVP (or a particular SVP type). Related to these measurements are the concepts of recall and precision. In general, recall refers to the ratio of true positives to the sum of true positives and false negatives, which quantifies the sensitivity of the predictive model. Similarly, precision refers to the ratio of true positives to the sum of true positives and false positives. When such a desired level of accuracy is reached, the training phase is terminated and the predictive model (e.g., ML module 630) may be output in step 790. If the desired level of accuracy is not reached, subsequent iterations of the training method 700 may be performed beginning at step 710 with variations such as considering a larger collection of FIM images.

[0052] As described herein, the training module 620 may train the machine learning based classification model 640, which may include a convolutional neural network (CNN). FIG. 8 illustrates an example neural network architecture 800 of a CNN. Each of the machine learning based classification models 640 may include a deep learning model that includes one or more portions of the neural network architecture 800. The neural network architecture 800 may perform feature extraction on the multiple FIM images using a set of convolution operations, as described herein, which may include a series of filters used to filter each image. The neural network architecture 800 may perform several convolution operations (e.g., feature extraction operations). The components of the neural network architecture 800 illustrated in FIG. 8 are intended to be exemplary only. The neural network architecture 800 may include additional components and / or layers other than those illustrated in FIG. 8, as one of ordinary skill in the art may appreciate.

[0053] The neural network architecture 800 may include multiple blocks 804A-804D, each of which may include several operations performed on an input FIM image 802 (e.g., a FIM image as described above). The operations performed on the input FIM image 802 may include, for example, another convolutional layer, a dropout operation, a flattening operation, a dense layer, or zero or more operations (e.g., pooling, dropout, activation, normalization, BatchNormalization, other operations, or combinations thereof) followed by a Convolution2D (Conv2D) operation or a SeparableConvolution2D operation until an output of the neural network architecture 800 is reached. A dense layer may include a group of operations or layers starting with a dense operation (e.g., a fully connected layer) followed by zero or more operations (e.g., pooling, dropout, activation, normalization, BatchNormalization, other operations, or combinations thereof) until another convolutional layer, another dense layer, or the output of the network is reached. The boundary between feature extraction based on convolutional layers and feature classification using dense operations may be illustrated by the flattening operation, which may “flatten” the multidimensional matrices generated using feature extraction techniques into vectors.

[0054] The neural network architecture 800 may include multiple hidden layers, ranging from as few as one hidden layer to as many as four hidden layers. One or more of the multiple hidden layers may comprise at least one filter (e.g., sliding window 504) as described herein. The at least one filter may include a size of, for example, 3×3 pixels. The at least one filter may be applied to the input FIM image 802. In some implementations, the input FIM image 802 may be pre-processed before being provided to the neural network architecture 800. For example, the input FIM image 802 may be resized to a uniform size, such as 64×64 pixels. Other implementations are possible. The at least one filter may be applied to the resized input FIM image 802.

[0055] The neural network architecture 800 may include a number of hyperparameters and at least one activation function in each of the blocks 804A-804D. The hyperparameters may include, for example, batch size, dropout rate, number of epochs, kernel size, stride, padding, etc. The at least one activation function may include, for example, a rectified linear unit activation function or a hyperbolic tangent activation function. Exemplary values ​​for each of the hyperparameters are provided below and elsewhere herein, although it should be understood that these values, as well as the particular hyperparameters used, may vary during implementation. That is, the values ​​and selections of the hyperparameters discussed herein are meant to be illustrative only and not limiting.

[0056] The input FIM image 802 may be resized before being processed. As described herein, the input FIM image 802 may be resized to 64x64 pixels. In each of the multiple blocks 804A-804D, the input FIM image 802 may be processed according to a particular kernel size (e.g., number of pixels). For example, as shown in FIG. 8, the first block 804A may include 64 convolution filters, a kernel size of "3" with padding having the same value, and a rectified linear unit (RELU) activation function. The input FIM image 802 may then be passed to a second block 804B, which may include one or more pooling operations, such as a MaxPooling2D operation and a stride operation. The input FIM image 802 may then be passed to a third block 804C, which may include a BatchNormalization operation. The BatchNormalization operation may standardize the input FIM image 802 as it passes through each block, which may accelerate training and reduce generalization error of the neural network architecture 800. For example, in the third block 804C, the input FIM image 802 may be passed through a dropout layer, which may apply a dropout rate (e.g., 0.15) to prevent overfitting.

[0057] In some embodiments, the network architecture 800 may include a flattening layer and / or a dense layer that may receive output features determined as a result of passing the input FIM image 802 through multiple blocks 804A-804D of the network architecture 800. The output features may include multiple SVP features derived from the input FIM image 802 and / or from training the network architecture 800. The flattening layer may determine / generate an N-dimensional array based on the output features. The array may be passed to a final layer of the neural network architecture 800. For example, the array may then be passed through one or more dense layers and / or a second dropout layer.

[0058] The input FIM image 802 may be passed through a number of convolution filters in each of a number of blocks 804A-804D, and then an output 806 may be provided. The output 806 may include an indication of the type (e.g., silicon or protein) and / or size (e.g., ECD size) of each SVP detected and classified in the input FIM image 802. The output 806 may include a binary classification (e.g., "SVP / non-SVP"), a multi-class classification (e.g., "bubble, fiber, or glass piece"), a percentage (e.g., 70% confidence of SVP type A and / or size A), a numerical value (e.g., 0.7), a combination thereof, and / or the like.

[0059] FIG. 9 shows an illustrative diagram of an example process flow 900 for classifying one or more SVPs in a FIM image using a trained machine learning based classifier, such as ML module 630. As an example, as shown in FIG. 9, an FIM image including unclassified SVPs may be provided as an input to ML module 630. ML module 630 may process the FIM image using methods described herein to arrive at an output 920 including one or more classified SVPs. Output 920 may identify one or more characteristics of the SVPs. For example, output 920 may identify the type and / or size of each classified SVP.

[0060] FIG. 10 shows a table of statistics for an exemplary dataset that may be used to train and test one or more machine learning models described herein, such as the ML module 630. The dataset may include a total of 7,500 labeled SVPs (silicon and protein). As shown in FIG. 10, the dataset may include FIM images containing SVPs ranging from "very small" to "large" ECDs. As described above, a subset of one or both of the first training dataset 610B or the second training dataset 610B may be randomly assigned to a test dataset. This subset may correspond to the "test set" shown in FIG. 10, which may include 20% of the dataset. The remaining 80% of the dataset may be used for training as described herein.

[0061] FIG. 11 shows a table of parameters for training one or more machine learning models described herein. The table shown in FIG. 11 corresponds to a series of experiments performed using the neural network architecture 800 to determine optimal parameters for one or more machine learning models described herein. These experiments are labeled in the first column of the table shown in FIG. 11 with the corresponding batch size, dropout rate, activation function, and cross-validation score for each experiment. The cross-validation scores shown in FIG. 11 may correspond to the cross-validation step 750 of the training method 700 described herein.

[0062] For some of these experiments, the neural network architecture 800 included three hidden layers and focused on tuning the model's hyperparameters (e.g., batch size, epochs, dropout rate, stride / padding, and filter count) and the best activation function to use by each of the multiple blocks 804A-804D of the neural network architecture 800. As shown in FIG. 11, the batch sizes used ranged from 64 to 128. The impact of a given batch size was dependent on the corresponding number of training epochs used by the neural network architecture 800. On average, the best accuracy was derived from using smaller batch sizes over many epochs, which can be seen in the cross-validation scores shown in the table.

[0063] In a sample experiment illustrated in FIG. 11, the epoch effect was dependent on the batch size and the dropout rate. In general, with a large batch size and a large dropout rate, high accuracy was obtained with fewer training epochs. The impact of a given dropout rate was highly dependent on the number of training epochs. Given sufficient training time, the ideal dropout rate was determined to be approximately 15%. In general, adjusting the dropout rate resulted in the highest performance improvement following the second hidden layer of the neural network architecture 800. The number of strides and padding had little effect on the performance of the neural network architecture 800 in a statistically significant manner. The activation functions tested for use by each of the multiple blocks 804A-804D of the neural network architecture 800 included sigmoid (logistic function), Tanh (hyperbolic tangent), and ReLU (rectified linear unit). The sigmoid activation function produced the lowest test accuracy (FIG. 11). The ReLU activation function produced the highest test accuracy (Figure 11). The Tanh activation function was effective for short training cycles, but was inferior to the ReLU activation function for medium and large training cycles (e.g., epochs).

[0064] As discussed above, a number of output features may be determined as a result of passing the input FIM image 802 through the number of blocks 804A-804D of the neural network architecture 800. FIG. 12 illustrates example feature maps that may be generated by the neural network architecture 800. For example, a first input FIM image 802A may include at least one protein SVP, and feature maps 1202, 1206, and 1210 illustrated in FIG. 12 correspond to output features that may be determined as a result of passing the image 802A through the number of blocks 804A-804D of the neural network architecture 800. As another example, a second input FIM image 802B may include at least one silicon SVP, and feature maps 1204, 1208, and 1212 illustrated in FIG. 12 correspond to output features that may be determined as a result of passing the image 802B through the number of blocks 804A-804D of the neural network architecture 800.

[0065] Feature map 1202 may represent a feature map corresponding to passing image 802A through a first block 804A of neural network architecture 800, while feature map 1204 may represent a feature map corresponding to passing image 802B through the first block 804A of neural network architecture 800. Feature map 1206 may represent a feature map corresponding to passing image 802A through a second block 804B of neural network architecture 800, while feature map 1208 may represent a feature map corresponding to passing image 802B through the second block 804B of neural network architecture 800. Feature map 1210 may represent a feature map corresponding to passing image 802A through a third block 804C of neural network architecture 800, while feature map 1212 may represent a feature map corresponding to passing image 802B through the third block 804C of neural network architecture 800.

[0066] 13, there is shown a graph depicting the accuracy of neural network architecture 800. As shown in FIG 13, neural network architecture 800 becomes significantly more accurate once the number of iterations exceeds 75. As also shown by FIG 13, the accuracy of neural network architecture 800 does not improve significantly after the number of iterations exceeds 150.

[0067] Figures 14A-14D show a series of tables depicting the accuracy of neural network architecture 800. Figure 14A shows the percent accuracy based on SVP type, i.e., protein or silicon. Figure 14B shows the percent accuracy based on ECD size. Figure 14C shows the percent accuracy of neural network architecture 800 associated with the experiment and parameters used as shown in Figure 11, and Figure 14D shows the percent accuracy for the same experiment based on ECD size.

[0068] FIG. 15 shows the accuracy, precision, recall, and F1 score levels of the neural network architecture 800. As shown in FIG. 15, an overall accuracy of 82.3% was achieved for classifying SVPs as protein particles or silicon particles. FIG. 16 shows relevant statistics of the performance of the neural network architecture 800 based on ECD size versus particle number. Compared to current industry standards, such as using S-factor analysis, FIGS. 13-16 demonstrate that the neural network architecture 800 is more capable of distinguishing SVPs.

[0069] FIG. 17 shows a block diagram depicting an example environment 1700 for implementing the present methods and systems for particle classification. The example environment 1700 shown in FIG. 17 includes a computing device 1701 and a server 1702 connected through a network 1704. In an aspect, some or all steps of any method described may be performed by the computing device 1701 and / or the server 1702. The computing device 1701 may include one or more computers configured to store one or more of the image data 1720 and / or classification data 1722 (e.g., FIM images, associated data, and one or more machine learning models described herein). The server 1702 may include one or more computers configured to store the image data 1720 and / or classification data 1722. The multiple servers 1702 may communicate with the computing device 1701 through the network 1704.

[0070] The computing device 1701 and the server 1702 may each comprise a digital computer, generally including, in terms of hardware architecture, a processor 1708, a memory system 1710, an input / output (I / O) interface 1712, and a network interface 1714. These components may be communicatively coupled via a local interface 1716. The local interface 1716 may be, for example, but not limited to, one or more buses or other wired or wireless connections known in the art. The local interface 1716 may have additional elements, such as controllers, buffers (caches), drivers, repeaters, receivers, etc., omitted for simplicity, to enable communication. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0071] The processor 1708 can be a hardware device for executing software, particularly stored in the memory system 1710. The processor 1708 can be any custom-made or commercially available processor, a central processing unit (CPU), a coprocessor among several processors associated with the computing device 1701 and the server 1702, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions. When the computing device 1701 and / or the server 1702 are in operation, the processor 1708 can be configured to execute software stored in the memory system 1710, communicate data to and from the memory system 1710, and generally control the operation of the computing device 1701 and the server 1702 in accordance with the software.

[0072] The I / O interface 1712 can be used to receive user input from one or more devices or components and / or provide system output to one or more devices or components. User input can be provided, for example, via a keyboard and / or a mouse. System output can be provided via a display device and a printer (not shown). The I / O interface 1712 can include, for example, a serial port, a parallel port, a small computer system interface (SCSI), an infrared (IR) interface, a radio frequency (RF) interface, and / or a universal serial bus (USB) interface.

[0073] Network interface 1714 may be used to transmit and receive from computing device 1701 and / or server 1702 over network 1704. Network interface 1714 may include, for example, a 10BaseT Ethernet adapter, a 100BaseT Ethernet adapter, a LAN PHY Ethernet adapter, a Token Ring adapter, a wireless network adapter (e.g., WiFi, cellular, satellite), or any other suitable network interface device. Network interface 1714 may include address, control, and / or data connections to enable appropriate communication over network 1704.

[0074] The memory system 1710 can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, hard drives, tape, CD-ROM, DVD-ROM, etc.). Additionally, the memory system 1710 may incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that the memory system 1710 can have a distributed architecture where various components are located remotely from each other but can be accessed by the processor 1708.

[0075] The software in memory system 1710 may include one or more software programs, each of which includes an ordered list of executable instructions for implementing a logical function. In the example of FIG. 17, the software in memory system 1710 of computing device 1701 may include experimental data 1720, computationally derived data 1722, a prediction module 1726, and a suitable operating system (O / S) 1718. In the example of FIG. 17, the software in memory system 1710 of server 1702 may include experimental data 1720, computationally derived data 1722, and a suitable operating system (O / S) 1718. Operating system 1718 essentially controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control, and related services.

[0076] For purposes of illustration, application programs and other executable program components, such as the operating system 1718, are illustrated herein as separate blocks, with it being recognized that such programs and components may reside at various times in different storage components of the computing device 1701 and / or the server 1702. Any of the methods described herein may be stored in computer readable instructions embodied on a computer readable medium. A computer readable medium may be any available medium that can be accessed by a computer. By way of example, and not intended to be limiting, computer readable media may include "computer storage media" and "communications media." "Computer storage media" may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Exemplary computer storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer.

[0077] FIG. 18 shows a flowchart of an exemplary method 1800 for particle classification. Steps of the method 1800 may be performed in whole or in part by a single computing device, multiple computing devices, and the like. For example, the method 1800 may be performed by one or more of the devices of the system 100, such as the client device 112 via the application module 113. As another example, the method 1800 may be performed by one or more of the devices of the system 600, such as by the ML module 630. As a further example, the method 1800 may be performed by the neural network architecture 800. In still further examples, some steps of the method 1800 may be performed by one of the devices and / or systems described above, and other steps of the method 1800 may be performed by another device and / or system.

[0078] In step 1810, the computing device may receive training data. The training data may include a plurality of input images, such as a plurality of flow imaging microscope images associated with a pharmaceutical product (e.g., FIM images and / or first training data set 610A described herein). Each input image of the plurality of input images may include at least one sub-visible particle (SVP). The at least one SVP may include a protein particle, a silicon particle, a fiber particle, an air bubble, a glass particle, etc. The at least one SVP may include an equivalent circular diameter (ECD) in the range of 1.125 to 130 micrometers.

[0079] In some embodiments, the computing device may receive second training data. The second training data may include image metadata associated with the plurality of input images, such as metadata associated with the FIM images described herein (e.g., second training data set 610B described herein). The image metadata may include metadata associated with the plurality of flow imaging microscope images. The image metadata may indicate a location of at least one SVP in each input image of the plurality of input images.

[0080] At step 1820, the computing device may generate a plurality of training images. For example, the computing device may generate the plurality of training images based on a plurality of input images (e.g., images selected for training according to the training method 700). Each training image of the plurality of training images may include at least one SVP. Each training image of the plurality of training images may include a uniform size, such as a size of at least 64×64 pixels. The uniform size may be a result of preprocessing the plurality of training images as described herein.

[0081] In step 1830, the computing device may train a convolutional neural network (CNN) (e.g., neural network architecture 800). For example, the computing device may train the CNN based on a plurality of training images. The trained CNN may be configured to classify one or more SVPs in one or more test images as including a first SVP type or a second SVP type. The first SVP type may include, for example, a protein, and the second SVP type may include silicon. As another example, the trained CNN may be configured to classify one or more SVPs in one or more test images as including a particular SVP type (e.g., silicon, protein, or air bubble). The trained CNN may include at least three hidden layers. Each hidden layer of the at least three hidden layers may include at least one filter including a size of at least 3×3 pixels (e.g., sliding window 504).

[0082] Training the CNN may further include determining a plurality of hyperparameters. The plurality of hyperparameters may include one or more of a batch size ranging from 64 to 256, or a dropout rate ranging from 5% to 50%. Training the CNN may further include determining an activation function. The activation function may include a rectified linear unit activation (RELU) function or a hyperbolic tangent activation (Tanh) function.

[0083] FIG. 19 shows a flowchart of an exemplary method 1900 for particle classification. Steps of the method 1900 may be performed in whole or in part by a single computing device, multiple computing devices, and the like. For example, the method 1900 may be performed by one or more of the devices of the system 100, such as the client device 112 via the application module 113. As another example, the method 1900 may be performed by one or more of the devices of the system 600, such as by the ML module 630. As a further example, the method 1900 may be performed by the neural network architecture 800. In still further examples, some steps of the method 1900 may be performed by one of the devices and / or systems described above, and other steps of the method 1900 may be performed by another device and / or system.

[0084] At step 1910, the computing device may receive at least one input image (e.g., a FIM image). The at least one input image may include at least one sub-visible particle (SVP). The at least one input image may include at least one flow imaging microscope image associated with a pharmaceutical product. In some examples, the computing device may receive image metadata associated with the at least one input image. The image metadata may include metadata associated with the at least one flow imaging microscope image. The image metadata may indicate a location of the at least one SVP.

[0085] In step 1920, the computing device may generate at least one pre-processed image. The computing device may generate the at least one pre-processed image based on the at least one input image and, in some cases, image metadata. For example, generating the at least one pre-processed image may include resizing the at least one input image to at least 64×64 pixels. As another example, generating the at least one pre-processed image may include determining a location of the at least one SVP based on at least one segmentation algorithm (e.g., agglomerative clustering, water flow transformation, etc.). The computing device may use at least one segmentation algorithm to determine a location of the at least one SVP when the image metadata does not indicate a location (or when the image metadata is not used by method 1900). In other examples, the computing device may use at least one segmentation algorithm to determine a location of the at least one SVP to confirm / verify a location indicated by the image metadata.

[0086] In step 1930, the computing device may determine a classification of the at least one SVP. For example, the computing device may determine a type of the SVP. The computing device may determine the classification of the at least one SVP using a trained convolutional neural network (CNN), such as neural network architecture 800. Additionally or alternatively, the computing device may determine the classification of the at least one SVP by passing the at least one preprocessed image through the trained CNN.

[0087] The computing device may train the CNN. For example, the computing device may train the CNN based on a plurality of training images. The trained CNN may be configured to classify one or more SVPs in one or more test images as including a first SVP type or a second SVP type. The first SVP type may include a protein, and the second SVP type may include silicon. The trained CNN may include at least three hidden layers. Each hidden layer of the at least three hidden layers may include at least one filter including a size of at least 3×3 pixels (e.g., the sliding window 504). Training the CNN may include determining a plurality of hyperparameters. The plurality of hyperparameters may include one or more of a batch size ranging from 64 to 256, or a dropout rate ranging from 5% to 50%. Training the CNN may further include determining an activation function. The activation function may include a rectified linear unit activation (RELU) function or a hyperbolic tangent activation (Tanh) function.

[0088] In step 1940, the computing device may output an indication of the classification of the at least one SVP. For example, the computing device may indicate via a user interface or other mechanism that the at least one SVP comprises a protein particle. Other implementations are possible, such as providing an indication via a message, report, etc. In yet other embodiments, the computing device may store a data record indicating the classification of the at least one SVP.

[0089] FIG. 20 shows a flowchart of an exemplary method 2000 for particle classification. Steps of the method 2000 may be performed in whole or in part by a single computing device, multiple computing devices, and the like. For example, the method 2000 may be performed by one or more of the devices of the system 100, such as the client device 112 via the application module 113. As another example, the method 2000 may be performed by one or more of the devices of the system 600, such as by the ML module 630. As a further example, the method 2000 may be performed by the neural network architecture 800. In still further examples, some steps of the method 2000 may be performed by one of the devices and / or systems described above, and other steps of the method 2000 may be performed by another device and / or system.

[0090] In step 2010, the computing device may receive at least one input image (e.g., a FIM image). The at least one input image may include a plurality of sub-visible particles (SVPs). The at least one input image may include at least one flow imaging microscope image associated with a pharmaceutical product. In some embodiments, the computing device may receive image metadata associated with the at least one input image. The image metadata may include metadata associated with the at least one flow imaging microscope image. The image metadata may indicate a location of each SVP of the plurality of SVPs.

[0091] In step 2020, the computing device may generate a plurality of pre-processed images. Each pre-processed image of the plurality of pre-processed images may include one of the plurality of SVPs. The computing device may generate the plurality of pre-processed images based on at least one input image. Additionally or alternatively, the computing device may generate the plurality of pre-processed images based on at least one input image and image metadata. For example, generating the plurality of pre-processed images may include resizing the at least one input image to at least 64×64 pixels. As another example, generating the plurality of pre-processed images may include determining a location of each SVP of the plurality of SVPs based on at least one segmentation algorithm (e.g., agglomerative clustering, water flow transformation, etc.). The computing device may determine a location of each SVP of the plurality of SVPs using at least one segmentation algorithm if the image metadata does not indicate a location (or if the image metadata is not used by method 2000). In other embodiments, the computing device may use at least one segmentation algorithm to determine the location of each SVP of the plurality of SVPs to confirm / verify the location indicated by the image metadata.

[0092] In step 2030, the computing device may determine a classification of each SVP of the plurality of SVPs. For example, the computing device may determine a type of each SVP of the plurality of SVPs. The computing device may determine a classification of each SVP of the plurality of SVPs using a trained convolutional neural network (CNN), such as the neural network architecture 800. Additionally or alternatively, the computing device may determine a classification of each SVP of the plurality of SVPs by passing each preprocessed image of the plurality of preprocessed images through a trained CNN.

[0093] In step 2040, the computing device may output an indication of the classification of each SVP of the plurality of SVPs. For example, the computing device may indicate the classification of each SVP of the plurality of SVPs via a user interface or other mechanism. Other implementations are possible, such as providing the indication via a message, report, etc. In yet other embodiments, the computing device may store a data record indicating the classification of each SVP of the plurality of SVPs.

[0094] Although the methods and systems have been described in connection with preferred embodiments and specific examples, it is not intended that the scope be limited to the specific embodiments described, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0095] Unless otherwise expressly stated, it is in no way intended that any method described herein should be construed as requiring that its steps be performed in a particular order. Thus, unless a claim for a method does not actually recite the order in which its steps should be followed, or unless otherwise expressly stated in the claim or specification to be limited to a particular order, no order is intended to be inferred in any respect. This holds true against all possible implicit bases for interpretation, including questions of logic regarding the placement of steps or sequence of operational flow, obvious meaning derived from grammatical organization or punctuation, and the number or type of embodiments described herein.

[0096] Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the method and compositions described herein which equivalents are intended to be encompassed by the following claims.

Claims

1. It is a method, The receiving of training data, which includes multiple input images, wherein each of the multiple input images includes at least one subvisible particle (SVP), by a computing device. The process involves generating multiple training images based on the aforementioned multiple input images, wherein each of the training images includes at least one SVP. The method includes training a convolutional neural network (CNN) based on the aforementioned plurality of training images, wherein the trained CNN is configured to classify one or more SVPs in one or more test images as including a first SVP type or a second SVP type. A method wherein the training data includes image metadata associated with the plurality of input images, and the image metadata indicates the location of at least one SVP in each of the plurality of input images.

2. The method according to claim 1, wherein the plurality of input images include a plurality of flow imaging microscope images associated with a pharmaceutical product.

3. The method according to claim 1, wherein the image metadata associated with the plurality of input images indicates the initial classification of at least one SVP.

4. The at least one SVP includes protein particles, silicon particles, fiber particles, bubbles, glass particles, or other known particle types. The at least one SVP includes an equivalent circular diameter in the range of 1.125 to 130 micrometers, Each of the aforementioned training images includes at least 64 x 64 pixels in size, The first SVP type contains a protein, or The method according to claim 1, wherein the second SVP type is at least one of those containing silicon.

5. The method according to claim 1, wherein the trained CNN includes at least three hidden layers.

6. The method according to claim 5, wherein each of the at least three hidden layers includes at least one filter having a size of at least 3 × 3 pixels.

7. The method according to claim 1, wherein training the CNN further comprises determining a plurality of hyperparameters, the plurality of hyperparameters including one or more of a batch size in the range of 64 to 256 or a dropout rate in the range of 5% to 50%.

8. The method according to claim 1, wherein training the CNN further includes determining an activation function, the activation function including a rectified linear unit activation function or a hyperbolic tangent activation function.

9. It is a method, The computing device receives training data including at least one input image containing at least one subvisible particle (SVP), Based on the aforementioned at least one input image, at least one preprocessed image is generated. A trained convolutional neural network (CNN) is used to determine the classification of the at least one SVP based on the at least one preprocessed image, The trained CNN outputs a representation of the classification of the at least one SVP, A method wherein the training data includes image metadata associated with the at least one input image, and the image metadata indicates the location of the at least one SVP in each of the at least one input images.

10. The method according to claim 9, wherein the at least one input image includes at least one flow imaging microscope image associated with a pharmaceutical product.

11. The method according to claim 9, further comprising verifying the location of the at least one SVP indicated in the image metadata.

12. To generate the aforementioned at least one preprocessed image, Resizing the aforementioned at least one input image to at least 64 x 64 pixels, or The method according to claim 9, comprising at least one of determining the location of the at least one SVP based on at least one segmentation algorithm.

13. The method according to claim 9, wherein the trained CNN comprises at least three hidden layers, and each of the at least three hidden layers comprises at least one filter having a size of at least 3 × 3 pixels.

14. The method according to claim 9, wherein the trained CNN includes a plurality of hyperparameters, the plurality of hyperparameters including one or more of a batch size in the range of 64 to 256 or a dropout rate in the range of 5% to 50%.

15. The method according to claim 9, wherein the trained CNN includes an activation function, the activation function includes a rectified linear unit activation function or a hyperbolic tangent activation function.

16. It is a method, The computing device receives training data including at least one input image containing multiple subvisible particles (SVPs), The process involves generating a plurality of preprocessed images based on at least one input image, wherein each of the plurality of preprocessed images includes one of the plurality of SVPs. A trained convolutional neural network (CNN) is used to determine the classification of each of the multiple SVPs based on the multiple preprocessed images, The trained CNN outputs a display of the classification for each of the multiple SVPs, A method wherein the training data includes image metadata associated with the at least one input image, and the image metadata indicates the location of at least one SVP in each of the at least one input images.

17. The method according to claim 16, further comprising determining the location of each of the plurality of SVPs based on at least one segmentation algorithm.

18. The method according to claim 16, wherein the at least one input image includes at least one flow imaging microscope image associated with a pharmaceutical product.

19. Further comprising verifying the location of the at least one SVP shown in the image metadata, The method according to claim 18, wherein the image metadata includes metadata associated with the at least one flow imaging microscope image and indicates the initial classification of the at least one SVP.

20. The plurality of SVPs include protein particles, silicon particles, fiber particles, bubbles, or glass particles, Each of the plurality of SVPs includes an equivalent circular diameter in the range of 1.125 to 130 micrometers, or The method according to claim 16, wherein the CNN is trained using training images, each having a size of at least 64 x 64 pixels, one of the above.

Images