Systems and methods for characterizing fiber compositions

Abstract

Presented herein are systems and methods for characterizing fiber compositions. In certain embodiments, methods and systems described herein utilized trained machine learning algorithms to characterize (e.g., automatically) fibers in a fiber sample.

Description

Systems and Methods for Characterizing Fiber Compositions CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Application No.63 / 603,555 filed November 28, 2023, the contents of which are incorporated by reference in its entirety. FIELD OF THE INVENTION

[0002] This disclosure relates generally to methods, systems, and architectures for analysis of fiber samples. More particularly, in certain embodiments, the disclosure relates to characterization of fiber samples using machine learning. BACKGROUND

[0003] Pulp mills and papermakers require careful control of input raw materials. The wood fiber source significantly affects the pulping process of breaking wood chips into pulp. Likewise, fiber properties are affected by the same operation. In addition to the final product performance, the fiber properties also affect processability. More long fibers result in increased sheet drainage time during sheet formation and moisture retention when the sheet is pressed and dried. Numerous subtleties differentiating both distinct species, and individual species grown in different geographies contribute to unique property contributions. Components like pitch, lignin, cellulose, and hemicellulose composition can vary significantly between species easily mistaken for one another by forestry operations and lumber yards. Different chemical compositions affect the process differently and require different operating conditions for optimal pulping as well as coproduct extraction. Fibers can also be modified by refining, which imparts mechanical energy to the fibers to reduce stiffness and causes cellulose fibrils to begin separation from the fiber (fibrillation).

[0004] The paper pulp composition, consisting of blends of different wood fiber types and treatments, affects multiple final product properties in interacting ways and determines process operating conditions. Longer, wider, flatter, softwood fibers, from trees like pine andspruce increase fiber entanglement, which improves tensile strength, adds absorbance, as well contributes to a smooth surface ideal for printing. Shorter, thinner, hardwood fibers, like birch and aspen, lead to bulkier, softer papers and contribute to crush strength for products like boxes.

[0005] Pulp and papermakers attempt to control the input composition of wood species to the pulping process and concentrations of component fibers to the papermaking process and adjust corresponding operating conditions to achieve desired properties. Better process and quality control systems and methods are needed. SUMMARY

[0006] Among other things, the present disclosure provides methods and systems for the characterization of fibers (e.g., cellulosic fibers), including paper pulp fibers, suitable for numerous applications. Embodiments of the present disclosure use machine learning-based methods, systems, and techniques to characterize samples containing fibers.

[0007] Prior to the present disclosure, characterization of a pulp sample’s composition required manual processes (e.g., microscopy and manual counting and description of fibers) as is further described below. In accordance with various embodiments, rather than relying on experiential based methods including an element of manual review for fiber characterization, the present disclosure provides a variety of methods and systems for assessing and describing fiber characteristics through learning (e.g., machine learning) rather than prior knowledge or real-time judgment from an observing human.

[0008] In one aspect, the disclosure encompasses methods of fiber (e.g., plant fiber, wood fiber) characterization, the methods comprising: accessing (e.g., receiving), by a processor of a computing device, an image (e.g., a microscopy image) of a fiber sample comprising one or more fibers; (e.g., automatically) segmenting, by the processor, using a segmentation module, the image to isolate at least one fiber segment of the one or more fibers of the fiber sample image; and (e.g., automatically) characterizing, by the processor, using a classification module, the at least one fiber segment as belonging to at least one fiber class.

[0009] In some embodiments, the one or more fibers comprise one or more stained fibers (e.g., an iodine-stained fiber).

[0010] In some embodiments, the one or more fibers are not stained (e.g., using iodine).

[0011] In some embodiments, the image is a fiber analyzer (e.g., a MorFi) image.

[0012] In some embodiments, the image is a microscopy image of the fiber sample.

[0013] In some embodiments, the microscopy image is a high magnification microscopy image (e.g., a microscopy image obtained at 4X, 20X, 40X, 60X, 100X, or higher magnification).

[0014] In some embodiments, the one or more fibers are not manually annotated (e.g., manually characterized by a user).

[0015] In some embodiments, the at least one fiber class corresponds to a classification of plant fibers (e.g., tree fibers).

[0016] In some embodiments, the at least one fiber class is a hardwood fiber or a softwood fiber.

[0017] In some embodiments, the methods comprise obtaining the image of the fiber sample using a microscope or a fiber analyzer.

[0018] In some embodiments, the methods further comprise characterizing the fiber sample.

[0019] In some embodiments, the methods comprise characterizing an amount, a proportion, and / or a percentage of mass of the at least one fiber class in the sample.

[0020] In some embodiments, the methods comprise characterizing an amount (e.g., proportion, percentage of mass) of old corrugated cardboard (OCC) and / or unbleached kraft pulp (UKP) in the fiber sample.

[0021] In some embodiments, the methods comprise classifying the at least one fiber segment as being an anomalous fiber.

[0022] In some embodiments, the image segmentation module comprises one or more machine learning algorithms.

[0023] In some embodiments, the methods comprise training the image segmentation module on a plurality of images of a training dataset (e.g., microscopy images, fiber analyzer images, grayscale images, color images, e.g., RGB color images).

[0024] In some embodiments, the training dataset comprises a plurality of images of known, substantially pure fibers.

[0025] In some embodiments, the methods comprise color transforming the plurality of images of the training data set (e.g., color transforming the plurality of images of the training data set from a first color space to a second color space).

[0026] In some embodiments, color transforming the plurality of images of the training data set comprises transforming the training data set images to LAB color space (CIELAB color space, L*a*b* color space) (e.g., from RGB (red-green-blue) color space to LAB color space).

[0027] In some embodiments, the methods comprise mode centering the training dataset images (e.g., subtracting the image mode from each color channel)(e.g., mode centering a color transformed images).

[0028] In some embodiments, the methods comprise performing a principal component analysis (PCA) transformation on the color transformed images.

[0029] In some embodiments, the methods comprise training at least one of the one or more machine learning algorithms using the PCA transformed images.

[0030] In some embodiments, the methods comprise using the at least one machine learning algorithm trained using the PCA transformed images to eliminate correlations between color channels in the plurality of images of the training data set.

[0031] In some embodiments, the methods comprise using the at least one machine learning algorithm (e.g., different from the PCA trained machine learning algorithm of embodiment 23) (e.g., a GMM, a BGMM) to segment the at least one fiber segment of the one or more fibers of the fiber sample image.

[0032] In some embodiments, the at least one machine learning algorithm is a Bayesian Gaussian mixture model (BGMM).

[0033] In some embodiments, the methods comprise removing fiber-fiber intersections and / or background in each of the plurality of images of the training dataset (e.g., through use of a segmentation mask).

[0034] In some embodiments, segmenting the fiber sample images comprises generating a mask (e.g., a binary mask) to isolate the at least one fiber segment of the one or more fibers of the fiber sample image.

[0035] In some embodiments, the methods comprise performing one or more of a sequence of color space transformations, principal component analysis, mixture clustering (e.g., Bayesian Gaussian mixture clustering), and a morphological operation on the fiber sample image (e.g., to generate a mask).

[0036] In some embodiments, segmenting the image comprises transforming (e.g., color transforming) (e.g., a sequence of color transformations) the fiber sample image (e.g., transforming the image from a first color space to a second color space).

[0037] In some embodiments, the methods comprise transforming the fiber sample image to a LAB color space (CIELAB color space, L*a*b* color space) (e.g., from an RGB (red-green- blue) fiber sample image to a LAB fiber sample image) (e.g., from a grayscale fiber sample image to a LAB image).

[0038] In some embodiments, the methods comprise mode centering the fiber sample image (e.g., mode centering a color transformed fiber sample image).

[0039] In some embodiments, transforming the fiber sample image comprises a grayscale transformation.

[0040] In some embodiments, transforming the fiber sample comprises binarization of the fiber sample image (e.g., after transforming the fiber sample image to a grayscale image).

[0041] In some embodiments, transforming the fiber sample comprises skeletonization of the fiber sample image (e.g., after transforming the fiber sample image to a binary image).

[0042] In some embodiments, the methods comprise removing (e.g., by the processor, e.g., using a background removal module, e.g., a machine learning module) one or more small objects from the fiber sample image.

[0043] In some embodiments, at least one of the one or more small objects are fines from the fiber sample image.

[0044] In some embodiments, the method comprises dividing the fiber sample images (e.g., by the processor) into a plurality of tiles, wherein at least a subset of the tiles each comprise at least a portion of one of the one or more fibers of the fiber sample image.

[0045] In some embodiments, the classification module comprises one or more machine learning modules.

[0046] In some embodiments, at least one of the one or more machine learning modules comprises a neural network with one or more dense layers (e.g., two dense layers, three dense layers, or more).

[0047] In some embodiments, at least one (e.g., at least two, at least three, or all) of the one or more dense layers comprises at least 5, at least 10, at least 15, at least 20 or more neurons.

[0048] In some embodiments, at least one of the one or more machine learning modules comprises a convolutional neural network (CNN).

[0049] In some embodiments, the methods further comprise characterizing the at least one fiber segment based on fiber width (e.g., mean width), fiber color, fiber length, fiber texture and / or fiber shape.

[0050] In some embodiments, the at least one fiber segment comprises a plurality of fiber segments.

[0051] In some embodiments, the one or more fibers comprises a plurality of fibers.

[0052] In some embodiments, the at least one isolated fiber segment comprises a plurality of isolated fiber segments.

[0053] In some embodiments, the at least one characterized fiber segment comprises a plurality of isolated fiber segments.

[0054] In another aspect, the disclosure encompasses methods of characterizing a fiber sample (e.g., plant fiber, wood fiber), the methods comprising: accessing (e.g., receiving), by a processor of a computing device, an image (e.g., a microscopy image) of a fiber sample comprising a plurality of fibers; segmenting, by the processor, (e.g., using a segmentationmodule) the image to isolate a plurality of fiber segments of the plurality of fibers; classifying, by the processor, (e.g., using a classification module) at least a portion of the fiber segments as belonging to at least one fiber class (e.g., a fiber type); and determining, by the processor, a composition of the fiber sample based on, at least, the classifications of the portion of the segmented fibers.

[0055] In another aspect, the disclosure encompasses systems for characterizing a fiber sample, the systems comprising: an imaging device (e.g., a microscope, a fiber analyzer, a sensor); a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method described herein.

[0056] In another aspect, the disclosure encompasses systems for characterizing a fiber sample, the systems comprising: a device configured to access (e.g., receive) one or more fiber sample images; a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method described herein.

[0057] In another aspect, the disclosure encompasses methods (e.g., of characterizing fiber samples in a method of manufacturing fiber-based products), the methods comprising: performing one or more processing steps to generate (e.g., manufacture) a fiber-based product from a fiber-based input; obtaining a sample of the fiber-based input after undergoing at least one of the one or more processing steps; characterizing (e.g., using a method of embodiments 1 to 47) the processed fiber-based input using at least one machine learning module (e.g., a characterization module, a segmentation module)(e.g., one or more machine learning modules) to obtain one or more fiber properties corresponding to the characterized, processed fiber-based input; and altering the at least one of the one or more processing steps based on, at least, the one or more fiber properties.

[0058] In some embodiments, the method is a method of papermaking.

[0059] In some embodiments, the fiber-based product is a paper product (e.g., toilet paper, paper, cardboard, paper towels, newsprint, etc.).

[0060] In some embodiments, the fiber-based input comprises recycled paper products.

[0061] In some embodiments, the at least one processing step comprises a chemical or a mechanical process.

[0062] In some embodiments, the at least one processing step comprises deinking, refining, pulping (repulping), blending, hornification, bleaching, enzymatic treatments, mercerization, regeneration, or handsheeting. BRIEF DESCRIPTION OF THE DRAWING

[0063] The foregoing and other objects, aspects, features, and advantages of the present disclosure will become more apparent and better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0064] Figure 1 shows exemplary microscopic Images of Papermaking Fibers. Figure 1, panels A-H show microscopic images of common biological fiber types Found in Papermaking Processes. Figure 1, panel A shows an exemplary image of Alder fibers. Figure 1, panel B shows an exemplary image of Aspen fibers. Figure 1, panel C shows an exemplary image of birch fibers. Figure 1, panel D shows an exemplary image of contorta tree fibers. Figure 1, panel E shows an exemplary image of cotton fibers. Figure 1, panel F shows an exemplary image of contorta eucalyptus fibers. Figure 1, panel G shows an exemplary image of pine fibers. Figure 1, panel H shows an exemplary image of spruce fibers. Images are provided courtesy of Swedish Cellulose Corporation (SCA).

[0065] Figure 2 shows an exemplary high level block flow diagram of a papermaking process.

[0066] Figure 3 shows an exemplary schematic drawing of a wood fiber’s structure, and the effect of processing.

[0067] Figure 4 shows components of an exemplary Artificial Intelligence System.

[0068] Figure 5 shows an exemplary depiction of discrete convolution operation.

[0069] Figure 6 shows example images from the ImageNet dataset.

[0070] Figure 7 shows example images from the MS CoCo dataset.

[0071] Figure 8 shows an example visualization of features learned by an exemplary convolutional neural network.

[0072] Figure 9 shows a flowchart of an exemplary transfer learning process.

[0073] Figure 10, panels A-L shows segmentation Steps for an individual image, according to an illustrative embodiment. FIG.10, panel A shows an original Image. FIG.10, panel B shows a L*a*b* Image. FIG.10, panel C shows a mode centered L*a*b* image. FIG. 10, panel D shows a PCA transform of FIG.10, panel C. FIG.10, panel E shows Gaussian mixture components of image FIG.10, panel D. FIG.10, panel F shows binarization of image E. FIG.10, panel G shows a post processed binary image. FIG.10, panel H shows a medial distance transformation of FIG.10, panel G. FIG.10, panel I shows a medial axis skeleton of FIG.10, panel F. FIG.10, panel J shows identified intersections. FIG.10, panel K shows FIG.10, panel G with intersections removed. FIG.10, panel L shows remaining fiber segments.

[0074] Figure 11 shows a flowsheet of the background removal operation (BGR), according to an illustrative embodiment.

[0075] Figure 12 shows a flowsheet of the segmentation operation, according to an illustrative embodiment.

[0076] Figure 13, panels A-L show exemplary color space transformation histograms. Figures 13, panels A-L show histograms of pixel colors in four different color spaces. Figure 13, panels A, B, and C show the original red, green, and blue color space, respectively. Figure 13, panels D, E, and F show a transformation to L*a*b* color space, where Figure 13, panel D shows the L* channel, Figure 13 panel E shows the a* channel, and Figure 13, panel F shows the b* channel. The transformation is attempting to isolate the effect of background illumination and staining intensity to a single channel. Figure 13 panels G, H, and I show transformation of the L*a*b* space by subtracting the image mode from each image, attempting to center the fiber background information on zero. Figure 13 panels J, K, and L show a final principal components transformation of the centered L*a*b* color space in order to remove linear correlation between the channels.

[0077] Figure 14 shows exemplary color space transformation scatterplots. Figure 14, panels A-D show 3d scatterplots of sampled pixels corresponding to the above histograms foreach of the color space transformations. Figure 14, panel A shows sample pixels in the original red, green, and blue color space. Figure 14, panel B shows sample pixels transformed to L*a*b* color space, attempting to isolate the effect of background illumination and staining intensity to a single channel. Figure 14, panel C shows transformation sample pixels of the L*a*b* space by subtracting image mode from each image, attempting to center the fiber background information on zero. Figure 14, panel D shows a final principal components transformation of the centered L*a*b* color space in order to remove linear correlation between the channels.

[0078] Figure 15 shows an exemplary architecture diagram (e.g., for image classification).

[0079] Figure 16 shows an exemplary First Layer Diagram. Figure 16, panel A shows network inputs of 2 color channels of the input image. Figure 16, panel B shows learned convolution kernels. Figure 16, panel C shows convolution outputs, which are the result of convolving the network inputs with the corresponding kernels. Figure 16, panel D shows linear combinations of convolution outputs. Figure 16, panel E shows layer activations where non- linearity is applied to threshold weak responses and highlight features useful for classification.

[0080] Figure 17 shows an exemplary architecture diagram.

[0081] Figure 18 shows an exemplary flowsheet of the classification and post-processing operations.

[0082] Figure 19 shows exemplary scatterplot of samples from the simulated fiber distribution.

[0083] Figure 20 shows exemplary images of Brownian surfaces of varying roughness.

[0084] Figure 21 shows exemplary microscopy images.

[0085] Figure 22 shows exemplary graphs of fiber widths and color.

[0086] Figure 23 shows example fiber segments.

[0087] Figure 24 shows an exemplary confusion matrix of test set classification results.

[0088] Figure 25 shows convolutional kernels learned by three network layers.

[0089] Figure 26 shows an exemplary confusion matrix of test set classification results.

[0090] Figure 27 shows exemplary GradCAMs of best classified test segments.

[0091] Figure 28 shows exemplary GradCAMs of worst classified test segments.

[0092] Figure 29 shows best and worst classified images. From top to bottom, each row represents a single species – Alder, Pine, Spruce, Aspen, Birch, Contorta, and Eucalyptus.

[0093] Figure 30 shows: Left Panel). Classification accuracy 95 percent confidence intervals for each slide. Right panel). Total number from each predicted class by slide.

[0094] Figure 31 shows an exemplary distribution of misclassification rates for both hardwood (blue) and softwood (orange).

[0095] Figure 32 shows an exemplary histogram of outcomes of bootstrap simulations representing realistic process conditions.

[0096] Figure 33 shows an exemplary length of the 90 percent confidence interval for estimated process composition versus number of slides in test sample.

[0097] Figure 34 shows an exemplary histogram of segment observation likelihoods under the training distribution.

[0098] Figure 35 shows exemplary results for fiber mixtures.

[0099] Figure 36 shows an exemplary graph of accuracy versus the model’s capacity (in terms of trainable parameters).

[0100] Figure 37 shows an exemplary series of images related to the image segmentation processes. Figure 37 panel A shows a grayscale image of an exemplary fiber sample. Figure 37 panel B shows a binarized image of the fiber sample. Figure 37 panel C is a skeletonized image of the fiber sample. Figure 37 panel D shows distances from edge (e.g., how far a given pixel is from a detected edge). Figure 37 panel E shows fiber segments with intersections of overlapping fibers removed.

[0101] Figure 38 shows a series of images comparing known and labeled recycled old corrugated cardboard (OCC) fibers and unbleached Kraft pulp (UKP) fibers.

[0102] Figure 39 shows a series of tiles for samples UKP fibers and OCC fibers illustrating different transformations of samples including a skeletonized image, an image showing pixel intensity corresponding to the distance of the pixel to the medial axis (eg thedistance from points on the skeletonized image), the distance of a pixel to a detected edge, and intensity of the pixels.

[0103] Figure 40 shows a series of streams from which fiber samples are obtained. Each stream from which samples could be taken are identified as 1-8 in the figure.

[0104] Figure 41 is a scatter matrix of the top ten latent features, OCC (Blue), UKP (Orange).

[0105] Figure 42 shows a series of images of different fiber subtypes.

[0106] Figure 43 shows a series of images of different fiber subtypes.

[0107] Figure 44 shows exemplary images of different fiber subtypes found in industrial samples.

[0108] Figure 45 shows exemplary images of different fiber subtypes found in industrial samples.

[0109] Figure 46 shows exemplary images of different fiber subtypes found in industrial samples.

[0110] Figure 47 shows an image comparing component fractions of a training set of images (left panel) as compared to actual images of fiber mixtures (right panel).

[0111] Figure 48 shows a graph estimating the quantification of UKP and OCC and a simulated 80% UKP-OCC mixture (80% UKP, 20% OCC).

[0112] Figure 49 shows an exemplary image without fines removed (top panel) as compared to an image with fines removed (bottom panel).

[0113] Figure 50 is a parity plot, which compares the model prediction of the mass percentage of OCC in a recipe to the actual recipe mass percentage of OCC.

[0114] Figure 51 shows effects of a fines correction process on UKP (top) and OCC (bottom) data.

[0115] Figure 52 shows exemplary results from industrially prepared samples.

[0116] Figure 53 shows three panels of feature cumulative distributions, each corresponding to a cumulative distribution of a feature in the latent space.

[0117] Figure 54 shows that fiber observation likelihoods differ between UKP, OCC, and mixed samples.

[0118] Figure 55 is a block diagram of an exemplary cloud computing environment, used in certain embodiments.

[0119] Figure 56 is a block diagram of an example computing device and an example mobile computing device used in certain embodiments.

[0120] The features and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. CERTAIN DEFINITIONS

[0121] About: The term “about”, when used herein in reference to a value, refers to a value that is similar, in context to the referenced value. In general, those skilled in the art, familiar with the context, will appreciate the relevant degree of variance encompassed by “about” in that context. For example, in some embodiments, the term “about” may encompass a range of values that within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%, 12%, 11%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, or less of the referred value.

[0122] Characterize: Many methodologies described herein include a step of “characterizing.” Those of ordinary skill in the art, reading the present specification, will appreciate that such “characterizing” can utilize or be accomplished through use of any of a variety of techniques available to those skilled in the art, including for example specific techniques explicitly referred to herein. In some embodiments, characterizing involves manipulation of a physical sample. In some embodiments, characterizing involves consideration and / or manipulation of data or information, for example utilizing a computer or other processing unit adapted to perform a relevant analysis. In some embodiments, characterizing involves receiving relevant information and / or materials from a source. In some embodiments, characterizing involves comparing one or more features of a sample or entity to a comparable reference.

[0123] Substantially: As used herein, the term “substantially” refers to the qualitative condition of exhibiting total or near-total extent or degree of a characteristic or property of interest. The term “substantially” is therefore used herein to capture the potential lack of completeness inherent in many biological phenomena. For example, as used herein, a substantially pure sample refers to a sample which is derived from a known source of fiber. Recognizing that processes (e.g., a manufacturing process) or surrounding environments may introduce certain contaminants into a sample, a substantially pure sample is one where at least 90%, at least 95%, at least 98%, at least 99%, or 100% of material in a fiber sample is derived from a known source of fiber. DETAILED DESCRIPTION

[0124] It is contemplated that systems, architectures, devices, methods, and processes of the claimed invention encompass variations and adaptations developed using information from the embodiments described herein. Adaptation and / or modification of the systems, architectures, devices, methods, and processes described herein may be performed, as contemplated by this description.

[0125] Throughout the description, where articles, devices, systems, and architectures are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are articles, devices, systems, and architectures of the present invention that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present invention that consist essentially of, or consist of, the recited processing steps.

[0126] It should be understood that the order of steps or order for performing certain action is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.

[0127] The mention herein of any publication is not an admission that the publication serves as prior art with respect to any of the claims presented herein. The Background section ispresented for purposes of clarity and is not meant as a description of prior art with respect to any claim.

[0128] Documents are incorporated herein by reference as noted.

[0129] Presented herein are technologies related to characterization of fibers in a sample (e.g., a fiber sample). In certain embodiments, provided technologies relate to methods and systems for artificial intelligence-based (e.g., machine learning, neural network) based analysis or characterization of images of fiber samples. In certain embodiments, provided technologies relate to methods and systems which incorporate artificial intelligence-based analysis or characterization of fiber samples to alter one or more processes for manufacturing fiber-based products (e.g., paper, cardboard, and the like). Artificial intelligence-based characterization of fiber samples provides for faster, more objective, and customizable characterizations and analyses, over more subjective, manual fiber characterizations currently used in the industry.

[0130] Present technologies determine a pulp sample’s composition according to a Technical Association of the Pulp and Paper Industry (TAPPI) standard by classifying and counting individual fiber species in microscopic images. Examples of microscope images of a variety of fiber species are shown Figure 1. Determining a pulp composition is currently done by a time consuming, subjectively defined, and human error prone manual process, requiring specialized experience. A microscopist must visually trace the long fibers across the field of view, searching for identifiable (based on subjective experience with reference samples) pore structures without losing track of which fiber is which.

[0131] Prior to the present disclosure, a microscopist observed and utilized numerous fundamental features such as pore structure, length, and width to identify fibers. However, a deeper question emerges: if a distribution is only defined based on known, observed features, how do we know we have observed the underlying distribution of variations useful for informing our applications of interest? Adding to the difficulty, behavior of pulp is emergent from a collection of individual fibers, each with their own unique “personality” yet contributing to an amorphous collection. The concept of “average” behavior is not particularly informative to the study of paper production disruptions as issues often arise as the result of extreme performances. Such as fluid flows leaking through a path of least resistance in a package, or a crack beginning at a weak point before propagating. Accordingly, studying the distribution of fibers in apopulation, their components, and their unique contributions to the collection would assist in understating.

[0132] Conservatively, just 1500 rolls of toilet paper have more fibers than there are people on the planet. (14.2mg / m ∗ 2.92mm / fiber = 0.0415mg / fiber; (227gm / roll) / (0.0415mg / fiber) = 5.5million fibers in a roll; 8 Billion people / 5.5 million fibers per roll = 1500 rolls)

[0133] While image level class labels are easily obtained by preparing known pure samples, annotating images at an individual fiber level is not feasible. Therefore, a method allowing individual fiber enumeration in mixtures of different species pulps, trained using single species data is desirable.

[0134] Among other things, the present disclosure provides an answer to a long-standing question in the field: given that different species of trees are categorized based on obvious and consequential differences of appearance, life cycle, geography, climate, environment, and physical properties, what differences are observable using the fundamental building blocks they grow from?

[0135] To this end, the present disclosure introduces and describes systems and methods for the characterization of paper pulp fibers, suitable for numerous applications, that addresses the “unknown” feature issue, as well as extensively evaluate the systems theoretical limits. Provided systems include five separately constructed yet codependent operations: 1. Color Correction: To mitigate illumination and staining differences. 2. Segmentation: To isolate individual fiber pieces (segments) from the whole fiber network in the images. 3. Classification and Anomaly Detection: To identify each individual segment of the fibers and highlight segments that are unusual. 4. Quantification: To estimate the true composition of fibers in the process from the individual segment identifications. 5. Simulation: To evaluate the system with respect to a known ground truth and identify and prioritize improvements to the system.

[0136] Using technologies described herein, 91% overall accuracy differentiating between hardwood and softwood fibers was achieved. Additionally, it is predicted that using 10 pulp samples from the papermaking process would yield a 90% confidence interval of the hardwood / softwood composition of 3:5%, which demonstrates the potential utility of artificial intelligence to wood fiber science.

[0137] Further, confounding factors in the data influence both which fiber species are present in an image, as well as our observation of the images themselves, make it difficult for a model to learn more complex features. Confounding factors change with each slide, this means that observing complete fibers rather than segments, may not be as important as dealing with confounding factors. Elimination or mitigation of such confounding factors may ease learning of more complex, and unknown, textural features. 1.1 Analysis and Characterization Images and Systems

[0138] In certain embodiments of the technologies described herein, any application appropriate images and systems to obtain appropriate images may be used.

[0139] Images of fiber samples can be obtained using a sensor of an imaging device. For example, without limitation, microscopy images or fiber analyzer images (e.g., images obtained from a MorFi fiber analyzer) can be used in embodiments described herein.

[0140] In certain embodiments, microscopy images can be high magnification microscopy images. In certain embodiments, high magnification microscopy images are taken at a magnification of at least 4X (e.g., at least 10X, at least 20X, at least 40X, at least 60X, at least 100X or greater). In certain embodiments, a high magnification microscopy image is taken using a high magnification objective lens. In certain embodiments, a high magnification objective lens has a magnification of about 100X (e.g., about 4X, about 10X, about 20X, about 40X, about 60X, about 100X). In certain embodiments, an objective lens has a numerical aperture ranging from about 0.1 and about 1.3 (e.g., 0.13, 0.3, 0.5.0.75, 0.85, 1.25, 1.3). In certain embodiments, a camera is used to produce image signals representative of an image based on data corresponding to light detected. Image signals can be sent from a camera to a device, such as a computer system (e.g., as described herein), either by a wired or wireless connection.

[0141] In certain embodiments, images obtained are color images (e.g., RGB images, CMYK) or monochromatic images obtained using an imaging device. In certain embodiments, an imaging device is a camera (e.g., a color camera, e.g., a color CCD camera). In certain embodiments, a camera can perform spectral sampling (e.g., sampling based on light band wavelengths). Pixels of a camera dedicated for detection of separate colors can be used to extract the intensity of light included in a given spectral band (e.g., a red color band, a green color band, a blue color band), thus allowing a spectral detection scheme of various wavelengths. In certain embodiments, monochromatic images are obtained using an imaging device, such as, example, a confocal microscope or a microscope with a monochromatic CCD camera.

[0142] A computer system can include one or more central processing units (CPUs) and associated memory (including volatile and non-volatile memory, such as, RAM, ROM, flash, optical and magnetic memory) and a display for presenting information to a user. Memory can store one or more computer programs that can be executed by CPUs to store and process image data and produce images of fiber samples. Computer programs can be executed by a computer to implement a method according to one or more embodiments whereby characterizations of fiber samples can be performed. 1.2 Fiber Samples and Staining

[0143] Technologies described herein involve imaging fiber samples. In certain embodiments, a fiber sample is a known fiber sample. In some embodiments, fiber samples are obtained from a material of interest, which includes fiber-based products or fiber-based inputs (e.g., for a manufacturing process of fiber-based products). In some embodiments, a fiber-based product is produced after one or more processing steps, for example as in a manufacturing method of a fiber-based product.

[0144] In certain embodiments, a fiber sample is a fiber-based input or a fiber-based product. In certain embodiments, a fiber sample must be prepared prior to imaging (e.g., prior to microscopy imaging). For example, in certain embodiments, preparing a fiber sample for imaging (and subsequent characterization) involves disintegration of the material of interest into a dilute fiber solution. In some embodiments, a dilute fiber solution comprises about 0.1 weight % of dispersed fibers 005 weight % 01 weight % of dispersed fibers 1 weight % of dispersedfibers, 5 weight % of dispersed fibers, 1 weight % of dispersed fibers. In some embodiments, a dilute solution comprises from about 0.01 weight % of dispersed fibers to 10 weight % of dispersed fibers, from about 0.01 weight % of dispersed fibers to 5 weight % of dispersed fibers, from about 0.01 weight % of dispersed fibers to 1 weight % of dispersed fibers, from about 0.01 weight % of dispersed fibers to 0.1 weight % of dispersed fibers, from about 0.01 weight % of dispersed fibers to 0.8 weight % of dispersed fibers.

[0145] In certain embodiments, a dilute fiber solution is then placed onto a substrate and allowed to dry. In certain embodiments, a substrate is a substantially optically transparent substrate. For example, a substantially optically transparent substrate can include, but is not limited to, a substantially transparent glass slide, a substantially transparent plastic slide, and the like. In certain embodiments, a dried fiber sample can be imaged with or without performing subsequent staining operations.

[0146] In certain embodiments, images can be used of fibers which have been stained (e.g., using a microscopy dye). Stains can be used to detect, reveal, or visually enhance physical and chemical characteristics of fibers including, but not limited to, classes of wood pulps including as hardwood and softwood fibers, compositions of fibers, porosity of fibers, and orientation of fibers, cell types, and features of fiber walls (e.g., thickness of walls, etc.). In some embodiments, composition of fibers can include material within or on fibers including, but not limited to, cellulose, hemicelluloses, lignin, pectin, extractives, callose, suberin, protein, fungal matter, phloem (bark), flavanols, and starch. In some embodiments, chemical and mechanical processing can affect visual appearances of fibers. In some embodiments, chemical and mechanical processing can affect how stains interact with fibers. Examples of chemical processes include pulping (e.g., kraft pulping, mechanical pulping, semichemical pulping, sulfite pulping) (or repulping), bleaching (e.g., by oxidation using chlorine dioxide, oxygen, ozone, or hydrogen peroxide), enzymatic treatments, mercerization (e.g., treatment of material with a strong alkali solution), and regeneration (e.g., a process in which cellulose is solubilized and converted back into cellulose). Examples of mechanical processes include application of mechanical force(s) (e.g., stress, strain) (e.g., deformations or cracking due to mechanical forces), mechanical refining, and hornification (e.g., drying). Contaminants may also affect interactions with stains (e.g., contaminants from recycled paper products).

[0147] In certain embodiments, stains used in technologies described herein include, but are not limited to, iodine-based stains, acid fuchsin, acridine orange, alcian blue, analine blue, anilium sulfate, astra blue FM, auramine O, benzopurpurin, berberine, calcofluor white, chlorazol black E, congo red, direct blue 1, direct orange 15, direct yellow 11, fluorenol yellow, hematoxylin, Herzberg reagent, indigo, lacmoid, lignin pink, malachite green, osmium tetroxide, phorogucinol, rhodamine B, safranin, Schiff’s reagent, Simon’s stain Direct Orange 15, Sudan black B, Sudan red 7B, toluidine blue O, and trypan blue. In certain embodiments, stains and staining procedures used in technologies described herein include, but are not limited to, Alexander stain, Alexander modification stain, anilinium sulfate stain, astra blue-safranin stain, Behrens bast fiber stain, Behrens stain, Boast stain, bright stain, Bucher stain, cold soda stain, Cross & Bevan ferric ferri-cyanide stain, Du Pont general stain, Ferric chloride-ferricyanide calcium nitrate stain, Graff “A” stain, Graff “C” stain, Graff F stain, Green-Yortson stain, Herzberg stain, iodine-potassium iodide Lugal stain, iodine-potassium iodide stain, Isernberg non-wood stain, Jenke stain, Kantrowitz-Simmons stain, Klemm stain, Klemm Sudan stain, Kratochvil stain, Laughlin stain, lignin pink stain, Lofton-Merrit stain, Lofton-Merrit Wisbar modification, Mäule stain, Noll’s stain, Noll-Hahn stain, p-Nitroaniline stain, Schiff reaction, Scultz’s stain, Schultze stain, Schwalbe stain, Selleger stain, Scaffer’s stain, Simons’ stain, sulfanilic acid stain, Sutermeister stain, Toluidine blue B stain, Vetillart stain, Wiesner stain, Wilson stain, Wisbar stain, Wisbar II stain, and Wurster di-stain.

[0148] In certain embodiments, the TAPPI standard T-401 can be used to prepare a fiber sample for imaging.

[0149] Upon reading the present disclosure, a person of skill in the art would be able to select an application appropriate stain or staining method for use with technologies described herein.

[0150] In certain embodiments, technologies used herein are used to identify classes of wood pulps to which fibers belong to. In certain embodiments, fibers are classified as being or belonging to softwood pulps or hardwood pulps. Exemplary stains used to detect softwood and hardwood pulps include, but are not limited to Alexander modification stain, Du Pont general stain, Klemm stain, Mäule stain, and iodine-based stains. In certain embodiments, fibers are classified as being or originating from kraft pulp (e.g., unbleached kraft pulp) and / or recycled materials (e.g., old corrugated cardboard). In certain embodiments, technologies herein are usedto identify anomalous fibers (e.g., fibers not previously characterized or part of training data) or anomalous materials (e.g., contaminants) in a fiber sample. In certain embodiments, no stains are used in technologies for characterizing fibers.

[0151] In certain embodiments, images can be used of fibers which have not been stained. For example, images taken with a fiber analyzers may not use or require stains to identify characteristics of fibers of a fiber sample. In certain embodiments, a microscope can be used to obtain images of fibers which have either been stained or not stained. 1.3 Training Data

[0152] A person of skill in the art would be able to choose application appropriate training data sets for training machine learning modules (e.g., image segmentation modules, image characterization modules) described herein.

[0153] In certain embodiments, a training dataset comprises known, substantially pure fibers. In certain embodiments, a training dataset comprises images of fibers from only a single class or type of fiber sample (e.g., only from OCC, only from UKP). In certain embodiments, a training dataset comprises images of fibers from multiple classes or types of fiber samples (e.g., mixed samples). In certain embodiments, a training dataset does not comprise images of fiber samples obtained from mixed fiber samples.

[0154] In certain embodiments, a training dataset comprises simulated images of fiber samples. For example, fiber samples can be simulated (generated) using known or predicted features. In certain embodiments, training datasets include simulated mixed fiber samples.

[0155] In certain embodiments, a training dataset comprises images obtained after one or more processing steps have been carried out on a fiber-based input to a processing step. For example, in a method of paper manufacturing (e.g., as shown in Figure 2), samples can be taken (e.g., before, during, or after) from a pulping process, a refining process, or a deinking process.

[0156] In some embodiments, a training dataset is an application-specific training dataset whereby a user processes a fiber-input (e.g., a known fiber-input) using one or more processing steps.

[0157] In certain embodiments, a training dataset includes color images. In some embodiments, color images are CMYK or RGB color images (e.g., images having three channels of intensity corresponding to red, green, and blue wavelength bands). In certain embodiments, atraining dataset includes microscopy images, fiber analyzer images, or other images from an application appropriate device. In certain embodiments, a training dataset includes grayscale images. 1.4 Segmentation of Fibers

[0158] In certain embodiments described herein, an image segmentation module is used to identify one or more fiber segments in an image provided. A person of skill in the art would be able to modify, add, or remove application appropriate steps depending on images provided for segmentation and any processing steps conducted prior to segmentation.

[0159] In certain embodiments, images provided to an image segmentation module have been transformed into an application appropriate data format prior to image segmentation. In certain embodiments, an image may undergo one or more color space transformations prior to segmentation. As used herein, color space transformation describes conversion of an image in one set of colors (e.g., a first color space) to another (a second color space). For example, an image can be color space transformed from an RGB image to a LAB image (an image in the LAB color space). A LAB or CEILAB image is an image that uses a color space, which uses three values to measure and compare colors: Luminance, L* and 4 corners of the a*, b* plane corresponding to the colors red, green, blue, and yellow. In some embodiments, images are transformed into a grayscale image, a binary image, or a skeletonized image.

[0160] In certain embodiments, images (e.g., images from a training dataset) can be standardized (e.g., after one or more color transformations). In certain embodiments, mode centering is used to standardize images. In certain embodiments, mode centering sets an image background color to zero. In certain embodiments, mode centering involves subtracting the image mode from each color channel. In certain embodiments, mode centering is performed on a color transformed image (e.g., color transformed training dataset images). In certain embodiments, principal component analysis (PCA) is used as a trainable color space transformation. In certain embodiments, PCA is used to learn a transformation of the input L*a*b* components into three new orthogonal components from a sample of pixels from many images. In certain embodiments, a trained PCA model is used to eliminate correlation betweeninput color channels in images before segmentation (e.g., by Bayesian Gaussian mixture models).

[0161] In certain embodiments, an image segmentation module includes one or more machine learning modules. In certain embodiments, a training dataset is used to train an image segmentation module. In certain embodiments, an image segmentation module is pre-trained (e.g., not trained by a user).

[0162] In certain embodiments, one or more transformations are performed on a fiber image to be segmented or a training dataset image to generate a segmentation mask. In certain embodiments, a segmentation mask is applied to an image of a fiber (e.g., a color transformed image). A segmentation mask can be used to cover or remove features of an image that a user would like to exclude (e.g., for training or analysis). For example, a segmentation mask can be used to mask fiber-fiber intersections and / or background image features. In certain embodiments, a segmentation mask can be used to mask confounding features.

[0163] In certain embodiments, a transformation used on training dataset images or fiber sample images is a morphological operation. A morphological operations can transform an image based on shapes found in an image. In certain embodiments, a morphological operation can be used on an image that has undergone a transformation (e.g., grayscale transformation, binarization, etc.). In some embodiments, multiple morphological operations can be used on an image to produce a desired output. In certain embodiments, a morphological operation includes erosion (e.g., to reduce sizes of objects in an image, e.g., a binary image), dilation (e.g., to increase sizes of objects in an image), opening (e.g., an erosion followed by a dilation), closing, (e.g., dilation followed by erosion), morphological gradient (e.g., subtracting an eroded image from a dilated image), hit-or-miss transform, top-hat transform (e.g., white top-hat, black top- hat), skeletonization (e.g., by eroding an image until objects are reduced to a skeletal form), and pruning (e.g., for removing artifacts from a skeletonized image). In certain embodiments, morphological operations can be used to process images as part of generating a segmentation mask.

[0164] In certain embodiments, one or more transformations may be applied to an image before training a machine learning module. For example, a sequence of transformations can be applied to an image or set of training dataset images prior to training a machine learning module(e.g., an unsupervised model) used in segmentation (e.g., Gaussian mixture models, e.g., Bayesian Gaussian mixture models).

[0165] In certain embodiments, one or more transformations can be used to generate a mask. As described herein, a mask can cover portions of an image such that certain portions of the image will not be analyzed. In certain embodiments, a mask can be used to remove overlapping clumps of fibers. In certain embodiments, a mask can be used to limit transformations (e.g., color transformations) to certain data within an image (e.g., to only fiber segments within an image).

[0166] In certain embodiments, a fiber sample image is divided (e.g., by a processor) into a plurality of tiles. One or more of the plurality of tiles can then further be segmented (e.g., using technologies described herein, e.g., using a segmentation module) to identify one or more fiber segments in a tile. For example, a long fiber can be divided into tiles and, in each of the tiles, the fibers in the tiles can further be segmented and classified. In certain embodiments, at least a subset of tiles each comprise at least a portion of one of the one or more fibers of a fiber sample image. In certain embodiments, tiles are sized depending on, for example, types of fibers being imaged, magnification, computational processing power, and machine learning model(s) being used. In certain embodiments, tiles are square tiles. In certain embodiments, tiles are regularized into a particular size (e.g., such that a machine learning module does not learn tile sizes). In certain embodiments, tiles are about 64 pixels x 64 pixels, about 128 pixels x 128 pixels, about 256 pixels x 256 pixels, about 512 pixels x 512 pixels. In certain embodiments, it is not important that tiles are of a particular size, but that they are regularized to the same sizes. 1.5 Characterization of Fibers

[0167] In certain embodiments a fiber in a fiber sample can be characterized using technologies disclosed herein. In certain embodiments, a segment of a fiber is classified. In certain embodiments, a plurality of segments are classified. In certain embodiments, a fiber segment or a fiber is classified as belonging to a particular class of fibers. In certain embodiments, a class of fibers includes softwood or hardwood fibers. In certain embodiments, a class of fibers is a class of plant fiber (e.g., wood fiber). In certain embodiments, a class of fibers corresponds to a species of wood (e.g., alder wood, aspen, birch, cotton, pine, eucalyptus, spruce, contorta, or another species of application appropriate wood). In certain embodiments, a fiber isclassified as an anomalous fiber (e.g., a fiber that does not belong to a dataset on which a characterization model is trained).

[0168] In certain embodiments, a fiber or segment of a fiber can be characterized based on one or more properties of the fiber or fiber segment including, but not limited to shape (e.g., width, length, curl, kink), fiber color (e.g., color after a stain is applied), and texture features can be used to classify a fiber. In certain embodiments, fibers can be characterized based on properties including spacing or frequency of pits and ridges, edge thickness, or shape of a tip of a fiber. In certain embodiments, segmented images (e.g., generated using technologies described herein) are used to characterize fiber segments. In certain embodiments, segmented images are used to generate spatial features, which are used by a characterization module to characterize fibers.

[0169] In certain embodiments, a classification module is used to characterize a segment of a fiber or a fiber using technologies disclosed herein. In certain embodiments, a classification module includes one or more machine learning modules. In certain embodiments, a machine learning module includes a neural network with one or more layers. In certain embodiments, the neural network has at least one, at least two, at least three, at least four, at least five or more layers (e.g., densely connected layers). In certain embodiments, at least one, two, three, four, or more layers are dense layers. A dense layer is a layer that is deeply connected with its preceding layer which means neurons of the dense layer are connected to every neuron of its preceding layer. In certain embodiments, a layer of the neural network has at least 5, at least 10, at least 15, at least 20 or more neurons. In certain embodiments, a classification module uses one or more convolutional neural networks (CNNs).

[0170] In certain embodiments, there are one or more biases that may affect characterization. Accordingly, upon reading the present disclosure, a person of skill in the art would be able to take appropriate precautions or implement one or more steps to reduce biases. For example, biases may be introduced by dilutions (e.g., highly concentrated fiber samples), fiber / fiber intersections, disintegration of a fiber sample (e.g., as a fiber sample is prepared for imaging, e.g., using the TAPPI standard), fines content, and fines settling. For example, at lower dilutions, more pixels are removed from an image to handle fiber / fiber intersections. In certain embodiments, fiber intersections may occur in different distributions depending on the types offibers present as well as dilution, so intersections may be removed differently in mixtures. In certain embodiments, with less disintegration, certain kinds of fibers (e.g., UKP fibers) fibers have higher propensity for clumping that of other fibers (e.g., OCC fibers) resulting in more fibers corresponding to a particular fiber class being removed from an image. In certain embodiments, some samples have higher fine content and particles below a certain threshold are removed from an image prior to intersection removal. Removing fines along with particles results in more pixels from particular fiber classes (e.g., OCC fibers) being removed from an image. In certain embodiments, samples may have a higher fine content and is more sensitive to settling. If allowed to settle for a longer during dilution, more fines will be removed from the mixture.

[0171] In certain embodiments, while one or more fiber segments can be classified, a distribution of features can be used to classify a fiber sample based on, at least, features obtained from a plurality of classified fiber segments. For example, rather than classifying a fiber sample based on an individual fiber’s classification (e.g., as hardwood or softwood), aggregates features obtained from fiber segments can be used to determine a distribution of features. A distribution of features can then be correlated with substantially pure samples of fibers. In certain embodiments, distributions can be further correlated with simulated mixtures of fibers. In certain embodiments described herein, a sample can be determined as having a proportion of features corresponding to a predicted mixture. 2.1 Papermaking Process

[0172] Figure 2 shows an exemplary process of paper manufacturing. Ignoring forestry management and logging for simplicity, a generic papermaking process consists of two high level operations. The first, pulping, where wood chips are reduced to a solid liquid slurry of microscopic fibers and water, using either mechanical or chemical energy to dissolve the lignin holding the plant cells together, and breaking the chips down to their smallest components. The second, forming, where the liquid is separated from the solid fibers using a combination of filtration, mechanical pressing, and heat.2.1 Fiber Structure

[0173] Wood and resulting fibers, a type of tree cell, have been extensively studied throughout history and their structure at the microscopic and molecular scale is well characterized. Fibers, illustrated in Figure 3, of different species share common structural elements with size, shape, and composition differences between species as well as significant random variations with an individual species. At a general level, a wood fiber is a long, walled tube, without branches which tapers to a close at each end. Identifiable features at a microscopic scale include the fiber wall and pores. Multiple cell types are evident in wood pulp. Wood fibers are significantly the most common type, and, rarer, are the short, wide, and porous vessel cells. 2.2 Mathematical Morphology

[0174] Mathematical morphology consists of operations that act directly on the shape of objects in an image. Operations can act on either binary images, where each pixel value is a 0 or a 1, or on grayscale images. Mathematical morphology consists of two primary operations which are useful as building blocks for more complex operations. First, erosion, as the name implies, is the operation of shrinking the edges of an object, uniformly, around the object. The converse, dilation is the operation of expanding the object edges. More complicated operations can be built using both operations in sequence. For example, morphological opening and closing, defined respectively as erosion, then dilation, and conversely dilation then erosion. Opening and closing are useful for the removal or exaggeration of small objects in an image. Skeletonization consists of a series of erosions until the object of interest is a single pixel wide. The distance of every pixel in the object from the skeleton can then be computed by counting the number of erosion before its removal. 2.3 Fiber Characterization Instruments

[0175] Previously, a MorFi fiber analyzer instrument may be used to assess fiber characteristics. The MorFi is the closest tool to a fiber composition measurement described in literature prior to the present disclosure. The flow through machine uses a high-resolution camera to image a very dilute pulp solution. Staining the pulp prior to measurement is not required. The images collected at the flow cell are analyzed using mathematical morphology.The images are binarized and fibers then skeletonized. Sample statistics, such as width, curl, kink, and length, are collected for fibers that can be isolated without branches on the skeleton. Images of the pulp solution are collected until a sufficient number of fiber statistics are obtained. There are several primary limitations to this device. First, the camera uses a low magnification and does not measure any microscopic information. Second, overlapping clumps of fibers are discarded, potentially creating a sampling bias against types of fibers likely to clump. Third, the instrument is only capable of computing features defined by the software and does not have the ability to learn new features. While useful to measure fiber sizes, the device is incapable of determining fiber type or composition of the pulp.

[0176] Among other things, embodiments of the present disclosure uses microscopy images, rather than those collected by a fiber analyzer, and seeks to understand fiber characteristics through learning (e.g., machine learning) rather than prior knowledge. 2.4 Wood Fiber Microscopy Using TAPPI Standard T-401

[0177] A method for estimating the composition of wood fiber species in paper pulps is by the manual identification and counting of individual fibers according to TAPPI Standard T- 401. The method involves the disintegration of the material of interest, either paper pulp, or possibly paper products, into a dilute solution (0.05 weight %) of dispersed fibers. A droplet of the solution is then placed on a slide glass and the water is allowed to evaporate. Fibers are then stained with an iodine stain which highlights different features, depending on the species and specific stain, by interacting with lignin in a fiber. In this way, the chemical composition and the morphology of fibers, are observed.

[0178] The technique does not explicitly require specialized equipment or materials, simply a microscope with at least 100x magnification, microscopy stains, and other supplies commonly found in many labs (hotplate, dropper, glassware, etc.). However, specialized training is required to learn the distinguishing features as well as trace fibers across the field of view, counting individual fibers. This process is also labor intensive, requiring the close attention of the microscopist throughout the count.2.5 Artificial Intelligence, Machine Learning, and Deep Learning

[0179] A simplified depiction of the components of a generic artificial intelligence system provided herein is shown in Figure 4. In general, an AI system takes input information, processes the input using a model and produces a decision as a result. Correctness of a decision is scored with a performance metric, called a cost function, which provides feedback that an optimization algorithm uses to determine how to adjust the model parameters to improve its performance. Four common tasks in AI are classification, clustering, object detection, and segmentation. Classification involves the separation of samples into known groups, where clustering refers to unknown groups. Segmentation uses either classification or clustering on a pixel level to divide an image into distinct regions. Like segmentation, object detection involves localizing interesting areas of an image, however instead of focusing on pixel level labeling, object detection models are used to identify a bounding box around as well as classify objects of interest.

[0180] When training machine learning models to split data into different subsets used exclusively for training a model, validating model performance as training progresses (without influencing the training itself) and for test evaluation of a final model is important. Subsets should be based on logical partitions in a dataset based on the nature of a prediction. For example, a mammography dataset should be partitioned by patient, so that identifying features, unique to each patient are not simply memorized by the model and then used to make "predictions" during testing. A model that performs well on training data but poorly on test data is said to overfit to the training data. There is a tradeoff between the ability of a model to learn complicated, descriptive, features and its preference to memorize unique features of each sample. The extent of a model’s ability to learn a multitude of complex features is its capacity. A capacity of a given model can be quantified by the model’s number of learnable parameters. Models are frequently evaluated in terms of accuracy as well as capacity, and related to capacity, throughput. For complex tasks, higher capacity might be required to explain all possible underlying variations. For others, a model might be preferred that has only enough flexibility to choose the best among many explanatory factors. Also, models that enable high throughput to accuracy, may be preferred.

[0181] Training data may be labeled, where each sample property of interest is known, or unlabeled where it is not known which properties are useful or do not have access to theproperty of interest. Models trained using labeled data are called supervised learning models. When using supervised learning models, a measurement between a prediction and a labeled result can directly be computed. This allows for direct optimization of the model’s parameters to minimize the difference between prediction and label. Unsupervised learning is used to identify patterns in unlabeled data, using a performance metric that makes some assumptions regarding structure of data and evaluating how close the model of the data fits the assumed structure. Principal components analysis and Gaussian mixture models, described next, are two examples of unsupervised models. 2.6 Principal Components Analysis

[0182] Principal Components Analysis (PCA) is an unsupervised statistical learning model used to find a linearly shifted, scaled, and rotated representation of original data where each new axis is uncorrelated and is scaled according to an amount of the original data variation is explained. When used for dimensionality reduction, the top N components, sorted by data variation explained, are used for subsequent learning algorithms. The remained of the components are discarded. In embodiments of the present application, all components are retained, and PCA is used to remove correlation in the color space, allowing for finer grained comparison between subtler colors in the fibers. 2.7 Gaussian Mixture Models

[0183] Gaussian Mixture Models (GMM) are unsupervised learning models that attempt to learn a mixture of component Gaussian distributions that have the same marginal distribution as the observed data. Each component is assigned a probability of occurrence, with all component probabilities summing to one, along with a mean and standard deviation defining a Gaussian probability density. Gaussian mixture models are useful for unsupervised clustering of data, with each sample assigned to the component which gives the highest observation probability. An alternative, but equivalent, Bayesian formulation of GMM, Bayesian GMM (BGMM), allows for incorporation of a prior assumed component probabilities. Mixture models can be used for unsupervised image segmentation, by assigning each pixel to a component, and image compression, by assigning each pixel the mean color values of its assigned component.This does not account for any geometric structure to the pixels or variations in the local color distributions across an image.

[0184] GMM is an effective tool to identify which pixels belong to the background color component and which ones correspond to fiber component. This allows removal of irrelevant background information from images prior to classification.

[0185] Both PCA and GMM, are unsupervised and powerful in their elegance, however ultimately limited by their simplicity when applied to structured data like images. Convolutional neural networks described in the following section, exploit spatial information in images rather than treating pixels independently. 2.8 Neural Networks

[0186] Several high-level neural network taxonomies are highlighted herein and an effort can be made to explain the specific reasons each might be used in embodiments of the present technologies. A person of skill in the art would be able to select an application appropriate neural network. 2.9 Convolutional Neural Networks

[0187] Neural networks are a popular, and increasingly common, choice of model for AI systems due to their generic structure and broad efficacy across domains. Neural networks have a desirable property of being universal approximators. That is, when given enough neurons, they have a capacity to fit any function or data to within a specified tolerance. Smaller tolerances can be achieved by adding neurons to the network. Neural networks function by mapping a set of input features into output targets through a composition of nonlinear functions. Each layer of the network takes as input the outputs of the previous layer and produces as output a new set of hidden or latent features. Each hidden feature is the result of a nonlinear function, or activation, of the inputs. So each layer is composed of multiple activation functions and the network is composed of multiple layers. Features computed by the final layer of the model are referred to as the latent representation of the data. Modern networks use millions of parameters and consist of compositions of many layers, hence the phrase "deep learning."

[0188] Shown in Figure 5 is a depiction of a convolutional kernel. The key concept is that the kernel represents a sliding window over the input with each output element representing a dot product between the kernel and the corresponding inputs. A convolutional neural network (CNN) consists of a composition layers with one or more convolutional kernels whose parameters are optimized to create the most accurate mapping between input features and output target pairs. Since the kernel is shared across the entire input, convolutional neural networks are able to learn to identify spatially invariant features appearing anywhere in an image. It is common to reduce the spatial dimension of intermediate outputs prior to the next layer’s operation. Called pooling, this step is typically done by dividing the input into non overlapping sub-regions pixels which cover the input and then pooling the pixel values in each sub-region into a single value, usually by taking the maximum or average values. 2.10 Regularization, Augmentation, and Synthetic Data

[0189] Regularization takes many different forms, but the effect is to encourage the magnitude of the learnable parameters towards zero so that only the most informative of the observed features become relevant to the prediction. Traditional regularization takes the form of a direct penalty scaled on the magnitude of the parameters that is added directly to the cost function. Regulation techniques are further described below.

[0190] Related to regularization, augmentation is the strategy of introducing random variations to the original data, mitigating the risk of overfitting and memorizing a static dataset. Numerous strategies for basic augmentations exist, such as random affine transformations, color space transformation, image corruption, etc.. More sophisticated techniques have been studied as well. Of particular interest are the copy paste (copying and pasting segmented objects from one image into another), cutmix (cutting and pasting rectangular regions from one image to another), and mixup strategies (blending multiple images). Augmentations have been shown to be a form of regularization and can increase the training value of small datasets. Ronneberger et al. describes how multiple simple augmentations, and a biologically inspired elastic deformation are essential to the success of their U-Net model, discussed further below.

[0191] Related to image augmentation is the generation of synthetic data. Tremblay et al. demonstrate that under suitable conditions synthetic datasets can approach the training accuracyof real-world datasets. In most cases, collecting a real-world dataset is time consuming and expensive, therefore performance gains by supplementing such a dataset with large quantities of cheap data can be an effective way to improve model performance. However, as with augmentation, additional data means additional training time and therefore computational cost, so this approach is best used when data is expensive and computation is cheap. 2.11 Wavelets and Gabor Filters

[0192] Introduced here in the context of the convolution operation, Wavelets are a class of function with the properties of being approximately localized and integrating to zero and are commonly used as convolution kernels. Making use of the convolution operation with a wavelet kernel, and related to the Fourier transformation, the wavelet transformation has the benefit of extracting both frequency and localization information. Numerous strategies based on wavelets have been introduced in image and signal processing. Of significant utility for surface analysis is the wavelet transform modulus maxima (WTMM) approach for multifractal analysis.

[0193] One particularly flexible construction of two-dimensional wavelet is the Gabor filter. Gabor filters are constructed by composing a two-dimensional Gaussian with a sinusoid and are parameterized by the Gaussian’s covariance matrix and the sinusoids frequency, yielding 4 adjustable parameters. Gabor filters are proposed as a plausible model for human vision and are a staple in computer vision, often incorporated into feature extraction strategies, discussed in greater detail below.

[0194] Unfortunately, the WTMM method, limited by its speed, is impractical for the fiber analysis application, which requires much higher throughput than medical image analysis. Gabor filters, although intuitively promising due to their ability to flexibly represent natural shapes and textures with few parameters, suffer from a computational cost that scales with the size of the filter. With larger images this operation is faster in Fourier space but doing so eliminates the representative power convolutional neural networks gain from composing many layers of small filters, with many paths through the network, in sequence. 2.12 Image Classification Architectures

[0195] An architecture of a neural network refers to design of the network, including how different layers are connected and which operations are used in which layer. While some models have found widespread success, the theory of architecture selection is lagging far behind our ability to compute them. Many choices are based on empirical evidence rather than theory. For this reason, several popular "off the shelf" architectures are reviewed, despite their conspicuous adjacency, rather than close relation, to the fiber image analysis domain; focusing on establishing a baseline perimeter and interpretation of a problem with little to no prior literature.

[0196] Notable in their popularity (as measured in citation volume) are VGG (Visual Geometry Group), ResNet, Inception, Xception, MobileNet, and their variations.

[0197] VGG consists simply of a deep series of convolutional and pooling layers. Where the pooling layers reduce the spatial dimension of the input by dividing the input into tiles and merging pixels within a tile into a single pixel using a summary statistic, like maximum or mean pooling. In VGG, maximum pooling is used. With an expanding number of filters in each convolutional layer, and a reduction in spatial dimensions with each pooling, the network projects spatial information depthwise, into the feature dimension.

[0198] Resnet uses skip connections, allowing the original information to be propagated throughout the model, allowing the network to learn residual information on top of the original image, and improving the propagation of gradient information from the cost function back to the early layers of the network.

[0199] Inception was intended by its authors to advance the study of deep convolutional networks in a new direction from the traditional stacking of layers of convolution and pooling. A number of clever choices are responsible for Inception’s improved performance over VGG but the main novelty is the use of multiple scales of convolution kernel used in parallel, with the resulting feature maps then concatenated between layers. Additionally, auxiliary classifiers, using features from intermediate layers, are added to the cost function during training, encouraging more useful information to be back propagated to early layers of the model. While ResNet type models are more accurate, Inception type models are more efficient computationally. Several variations of Inception have been released including a combination of Inception and ResNet, predictably dubbed “Inception-ResNet”.

[0200] Building on the success of the hypotheses that lead to Inception, the key concept of Xception is the implementation of a depthwise separable convolution kernel instead of standard convolution. This is achieved by factoring the standard convolution operation into two steps, a convolution operation within a channel, followed by a convolution between channels. While this is significantly more efficient computationally, there is a slight reduction in the accuracy due to the practical learning differences. MobileNet is a lightweight version of Xception specifically optimized to be efficient enough for use on personal devices.

[0201] All five of these architectures are built into the popular deep learning package, Keras, making them convenient choices when benchmarking classification performance on new datasets.

[0202] As discussed herein, most architectures are described in the context of a small handful of benchmark datasets. The best performing architectures on one dataset may not directly translate to the best performance on an adjacent, rather than directly related, task. In particular, the large capacity of these models as they are described in the literature (10s of millions of parameters) may be inappropriate, or even problematic, for tasks described herein. 2.13 Object Detection

[0203] Object Detection is not typically an unsupervised task, so is generally unsuitable for the task, where it is limited to image level labels for pure species, and no labels for mixtures. Further, in many cases it is not possible to isolate a single fiber in a bounding box of any size as their aspect ratio frequently causes them to cross image boundaries. However, the task bears mentioning both as a component of an exhaustive review around the perimeter of our problem, as well as inspiration for our architecture design. Additionally, future work might involve the collection of labels for small scale objects like fiber pores, which would be useful for subsequent fiber classification.

[0204] The Regions with CNN features or R-CNN architecture is a building block of detection systems and consists of a region proposal, a convolutional network, and supervised classification. Feature Pyramid Networks uses a series of image downscaling operations,forming a pyramid of images used with a series of corresponding convolutional kernels intended to represent scale invariant features. 2.14 Segmentation

[0205] Typically, image segmentation research strategies are exclusively supervised, requiring time consuming creation of manual labels. Fortunately, for many segmentation tasks this can be mitigated using transfer learning.

[0206] The U-Net model is one of the more fundamental designs of segmentation architectures, demonstrating state of the art performance on the cell tracking challenge on its introduction. The model attempts to project spatial information into a context dimension, before recombining with the original spatial information to learn a pixel wise segmentation map.

[0207] Mask R-CNN combines both classification and segmentation tasks, with the insight that key features underlying classification in natural images are useful for segmentation as well.

[0208] Unfortunately, the above segmentation models are supervised, requiring labelled data, which is infeasible to collect for the current problem at this time. Unsupervised segmentation models do exist, however they are based on general assumptions which may not be valid in this context. Furthermore, they may provide limited improvement for the added computational cost over techniques such as GMM. Specifically, for the task described herein, isolation of numerous overlapping objects is required. Each object is unique but has very similar characteristics to each other. In such a situation, a nuanced assumption specific to the own data is required to resolve overlapping fibers, but very mild assumptions about the background. 2.15 ImageNet

[0209] Multiple benchmark datasets are popular within the computer vision community and provide researchers with a way to compare cutting edge models. Two of the most popular and comprehensive are the ImageNet for image classification, and Microsoft CoCo (Common Objects in Context) dataset for image segmentation.

[0210] While earlier datasets were introduced, the advent of modern deep learning can be traced to the introduction of the ImageNet dataset and launch of the yearly ImageNet Large Scale Visual Recognition Challenge. The ImageNet dataset consists of 1.2 million images covering 1000 different classes. The goal of which was to provide a foundation for computer researchers to develop image recognition models without need to collect their own data. It has become a key benchmark for classification performance. Since a large amount of literature reports ImageNet results, it is useful for estimating training time or inference throughput for similar datasets and architectures.

[0211] The MS CoCo dataset contains over 200 thousand annotated images with 1.5 million objects across 80 categories. In addition to classification the dataset provides labeled examples for object detection, human key point identification, panoptic segmentation, and human pose identification. Pretrained architectures, trained on MS CoCo tasks are available as well, but are not as common as ImageNet trained architectures.

[0212] Examples from ImageNet and MS CoCo are shown in Figures 6 and 7, respectively. 2.16 Transfer Learning

[0213] Transfer Learning allows neural network practitioners to use large networks trained with a large dataset on one task and fine tune them for use in different tasks with less available data; significantly reducing the computational barrier to deploy a model. This technique is based on a concept that many tasks share low level features. At the lowest levels, a network learns to identify lines, then contours, then basic shapes, seen in Figure 8. Only in the final layers does a network learn to identify specific arrangements of shapes, such as eyes and a nose on face to identify a person or animal. An exemplary transfer learning strategy is outlined in the flowchart in Figure 9. Parameters learned to solve one task might be a useful starting point when learning to solve another related task. For example, a network trained to identify a variety of everyday objects like books, cups, couches, and televisions, might be adapted to identify more specific objects, like rocking chairs or recliners.

[0214] Conveniently many models provided in the Keras package may be optionally initialized for transfer learning and fine tuning with provided parameters learned by training onthe ImageNet dataset. A wide variety of other pretrained architectures exist however they tend to specialize in one way or another towards improving computational efficiency or improving accuracy. Naturally, this is an area of ongoing progress and regular improvements, so between the time of writing and publication new architectures and strategies will have been inevitably introduced. Additionally, more rigorous benchmarking of the popular VGG, Inception, Xception, ResNet, and MobileNet models included in Keras may have been undertaken.

[0215] However, from the features visualized in Figure 8, it can be seen that a typical model has learned many features which would be completely irrelevant to fiber classification. This suggests that models pretrained on imagenet or similar may not be a useful starting points for fiber classification. Unfortunately, the vast majority of literature uses the same benchmark datasets. Only brief review is required to realize there is significant space in the literature for open source, large scale, manufacturing and materials benchmark datasets. With so many pretrained architectures dependent on a handful of datasets, many applications are not well suited for transfer learning. In addition, when lots of researchers are focusing on the same problems in the same datasets, they might adapt some tunnel vision to the problems faced in other applications. Many corporations fund the generation of their own, proprietary datasets. 2.17 Wood Classification

[0216] To date, the fiber classification problem has not been studied in the literature, however Ristiawanto et al. and Taqyudin et al. study classification of cedar wood planks into three quality categories using both bottleneck features computed by a pretrained Xception model and a Histogram of Oriented Gradients (HOG) feature. Notably, the HOG method significantly outperformed use of the pretrained features with a performance of 90% versus 50%. Although limited data was used this suggests the specific features necessary for classifying the wood grain are not well represented by the pretrained networks. It is possible this is due to the lack of more salient global features in the wood grains compared with the ImageNet dataset. Wood grains are however oriented in the planks like objects of interest have a natural orientation in imagenet.2.18 Biomedical Applications

[0217] Although literature specific to classification problems in wood science is sparse, there is an abundance of literature relating to biomedical science. Classifications in biomedical contexts still hold some key differences. First, classifications must be distinguished in images at the cellular level versus images at the organ level, like a CT scan or mammogram, both fine grained classification problems. While the latter is a fine-grained problem, like fiber classification, certain characteristics make it a poor analogue. First, the human body is by nature oriented, and has a distinct global structure. There is a natural z axis defined by drawing a line through the head to the heels, and is similarly oriented with respect to rotations or translations in the 2d plane. Cellular images, however, display no such orientation, and only some global structure. In this context, opportunities for augmentation are significant.

[0218] Foundational is the formulation of the so-called U-Net architecture for cell segmentation. Like the ImageNet dataset, the cell segmentation challenge provides a benchmark for biological image researchers to demonstrate the efficacy of their algorithms in a context relevant to their focus. While paper fibers are technically a type of biological cell making up the structure of a tree, they display significant differences relative to the type of cells U-Net is demonstrated on.

[0219] It is also worth noting that many biomedical images are collected with similar instrumentation, and suffer the same fallbacks as fiber images. 2.19 Manufacturing Defect Detection

[0220] Another well studied analogue is the classification of defects in manufacturing. Gabor filters can be used in detection of defects in images of textiles. Zhang et al. used banks of Gabor filters and Gaussian mixture models, Bissi et al. used banks of Gabor filters and principal components analysis, and recently Chen et al. combine a genetic algorithm to tune the parameters for a bank of Gabor filters with an RCNN layer. Additionally, Zhao et al. describes a multi scale convolutional neural network for defect classification and Li et al. provides an overall review of textile defect detection strategies. Significant in the context of fiber classification is the seeming necessity of feature engineering rather than pure deep learning. It is possible mostpopular model architectures are overly specialized for ImageNet type problems. The comprehensive description of currently utilized features can inform new specialized architectures for different domains.

[0221] While there are similarities, it is not immediately obvious that knowledge from this domain is transferable to fiber classification. While the fibers themselves might be analogous to paper fibers, the defect classification takes place at a much larger scale, focusing on the high- level structure. For many types of textiles, high-level structure is expected to be highly ordered, and defects disordered, a condition exploitable by Gabor filters. Finally, as implied by the term "Manufacturing Defect Detection," the task is to identify and localize defects. As previously discussed, this is problematic for our task of separating overlapping fibers which frequently cross the boundaries of an individual image. 2.20 Open-Source Software Packages

[0222] In the present example, open-source software was used to perform processes. Built from a Python foundation, NumPy is utilized to manage array operations and linear algebra, SciPy for optimization, statistical, and curve fitting functionality, Scikit-Learn for basic machine learning, notably PCA and BGMM, Tensorflow and Keras for neural networks, and OpenCV and Scikit-Image for computer vision and image analysis respectively. 2.21 Hypothesis

[0223] Existing devices and standards for the measurement of fiber characteristics are unsatisfactory for emerging needs. Commercial fiber analyzers are incapable of the type of detailed quantification demanded by the task, and manual estimation following T-401 is labor intensive and subject to human judgement. Deep learning has the potential to replace the human element and uncover a deeper understanding of fiber characteristics through the learning of features most optimal for solving the prescribed task. 2.21.0 The fiber classification problem is a good fit for deep learning.

[0224] While it is currently popular amongst the deep learning research community to study models that are broadly generalizable focus on ImageNet and other benchmark datasetshas created tunnel-vision, leading to models that are generally good at a wide variety of tasks but can fail miserably in more specific circumstances. Unfortunately for many manufacturing applications, the “generally good at a lot of things” philosophy might be insufficient. Important features are often incredibly subtle and difficult to distinguish from unimportant variations. The relative accuracies of benchmarked models might not be representative of performance on significantly different datasets. Expanding benchmarks to a very different domain can yield important understanding of why some classes of model are successful in some domains but ineffective in others, leading to better strategies to generalize across domains. The simply shaped, but subtly variable fibers used in paper manufacturing are an excellent example of this difficulty. The normal feature variations used to distinguish between classes in the ImageNet dataset, when present in microscopic images of single fibers, are often typical of all classes. That is to say, the fibers all display a wide variety of typical features that are shared by all classes, and only subtle distinguishing features between classes. Fibers have a primitive structure, incorporating this prior knowledge can improve the ability of the model to learn the subtler features that might otherwise be masked by the observed variations of the overall structure. 2.21.1. This task may require a significantly different model from those popular with the deep learning community. Additionally, the explicit incorporation of knowledge of prior structure into the model may be a benefit of approaches described herein.

[0225] Generalizing the ability to make predictions beyond the scope of the original training data reduces the need to collect data. In some cases, generalization to situations for which no data is available is required. In this project, it is easy to generate large quantities of labelled data for images of pure fibers, however it is very time consuming to label individual fibers of different species in mixed pulps. For this reason, it is desirable to generalize from labeled images of pure pulps to unlabeled images of mixed pulps. While there may be differences in the preparation of mixed pulps versus pure pulps, the identity of the individual fibers doesn’t change. 2.21.2. Information learned from individual fibers segmented from pure pulps of known species should be generalizable to unknown mixtures.COLOR CORRECTION AND SEGMENTATION

[0226] A prerequisite to classification is extracting image segments isolated from individual fibers. It is necessary to take precaution to ensure the sample of segments extracted is both representative and high quality. Representative means there should be no filtering process inherent to the extraction algorithm that preferentially biases certain segments dependent on the fiber type. High quality meaning well isolated from other fibers, the background, or contaminants and with no spuriously removed intersections, yielding the maximum size of non- intersecting segments possible for a given image. A sequence of color space transformations, principal components analysis, Bayesian Gaussian mixture clustering, and basic morphological operations was used to isolate fiber segments with no need for manually annotated data. The image in each step of the segmentation process is depicted step by step in Figure 10. The background removal and the segmentation steps are summarized in the diagrams depicted in Figures 11 and 12 respectively. 3.1 Color Space Transformations

[0227] Color space transformations, visualized for a sample of pixels from images in the training dataset as histograms in Figure 13 and 3d scatter plots in Figure 14, empirically proved essential for preparing raw microscopy images for machine learning. These consisted of 1. Simple RGB (red, green, blue) to L*a*b* (Luminance, L*, and the 4 corners of the a*, b* plane corresponding to the colors red, green, blue, and yellow), 2. Subtracting the image mode from each color channel. The first transformation attempts to isolate the effect of perceptual lightness and darkness from subtle color variations. Through their experimental preparation, fibers will undergo various degrees of staining, where resulting lighter or darker fibers may not indicative of fiber species. Subtly, a* and b* variations might still be distinguishable despite variations in staining conditions. Mode centering is then used to standardize images so that the most commonly observed color, the background, is set to zero, intuitively representing that no information should be present. This is essential and standard normalization or minmax normalization will not work because the imagewise statistics are highly dependent on the consistency of fibers in the slide, and, therefore, number of fiber pixels versus background pixels in an image. These steps are visualized for a single image in Figure 10, panels A-C.3.2 Principal Components Analysis and Gaussian Mixture Models

[0228] Principal component analysis was used as a trainable color space transformation to learn a transformation of the input L*a*b* components into three new orthogonal components from a sample of pixels from many images. The trained PCA model is used to eliminate correlation between input color channels in images before segmentation by Bayesian Gaussian mixtures. The same sample of pixels, now PCA transformed, are then used to train a Bayesian Gaussian mixture model, which learns component clusters in the color space, and is employed to segment images where pixels are placed into the learned groups. Essential to the training of both PCA and BGMM using a sample over many images instead of training by images was the use of mode centering. Otherwise, multiple clusters are evident in the data, and the spurious partition is easily learned. Using mode centering reveals the most natural component distributions in the data, the background, and the fiber. Random uniform noise between -0.5 and 0.5 is added to the training sample prior to training to transform the data from a discrete to a continuous space; essential prior to application of BGMM for good convergence and to prevent the model from clustering on the discrete values. Seen in Figure 10, panels D and E. Learning the color space transformation and removal of the background over many images simultaneously is intended to make the system more robust to variations between the slides. 3.3 Morphological Operations

[0229] Finally, a series of morphological operations are applied to clean the resulting mask followed by removal of fiber-fiber intersections (Figure 10, panels F & G). Small objects and holes are removed followed by binary closing to remove cracks, median filtering to smooth edges, and binary erosion to eliminate remaining background on object edges (Figure 10, panel H). The medial axis skeleton is computed and pruned of small false branches and used to identify intersections (Figure 10, panel H). During the medial axis skeletonization, the distance of each skeleton pixel from the object edge is computed. Centered at each intersection pixel, within a circle with radius equal to the maximum observed distance over the whole skeleton, the local maximum distance is taken then pixels falling within the circle with radius equal to the local maximum distance (Figure 10, panel J) are removed from the binary image (Figure 10, panelK). Isolated objects in the binary image are then used as segmentation masks to extract fiber segments from the colored images (Figure 10, panel L). CLASSIFICATION AND ANOMALY DETECTION

[0230] Where training resources are limited, very little hyper parameter tuning can be afforded. Instead, the focus was on choices that are robust across the hyper parameter space, rather than choices that require tedious parameter searches to achieve better results. Acknowledging that such a methodological tuning and architecture search is likely to improve results, such an activity is probably best justified after demonstrating proof of concept and should be weighed against simply increasing the size of the data, which is not possible with benchmark datasets. By establishing a baseline, classification results could be collected and subsequent interpretations could be made about important aspects of the underlying data and which strategies and architecture choices are most promising. 4.1 Preprocessing

[0231] Segments with areas less than 5000 pixels are discarded to avoid debris and segments without enough information, where the minimum area was determined by observing samples above and below various thresholds. Images with an L* mode value less than 100 are also excluded as these images were observed to have a very high concentration of fibers, making it difficult to obtain high quality segments, and the mode value is not representative of the background color.

[0232] The size of the segments extracted depends somewhat on the fiber type, since the size of the fiber is influenced by its type, but mostly depends on the pulp consistency, and therefore the number of intersections. Some species, like eucalyptus, have more curvature than others, so cause more intersections. As with color in the segmentation, it’s desirable to eliminate as many spurious partitions in the data as possible prior to classification. Furthermore, a wide variety of sizes poses issues when allocating memory to specify the neural network graph on the GPU. Typically, images are either resized to standard size or are zero padded. In this case long fibers were split into smaller tiles, mitigating both concerns. By splitting segments into tiles samples can be provided in known batch sizes and control memory usage. Additionally, there issignificantly less wasted computation of zero elements surrounding the fiber. This allows for larger models within the constraints of hardware and faster processing. First the segments are rotated by 90 degrees, if necessary, so that the vertical dimension is the largest. The length of the vertical axis is then divided into the smallest number of integer valued increments such that each has length at least as large as the desired tile size, and defines where the segment is sliced into horizontal strips. Each horizontal strip is then sliced in the same way as vertically, where necessary. Each tile is then padded to the specified tile size with equal spacing on opposite sides. A tile size of 128 by 128 pixels is used to provide a balance between receptive field and size of model on the GPU. Also, 128 pixels is large enough to contain the full width of most fibers observed, but not large enough to accommodate vessel cells.

[0233] Once the segment dataset is generated, a full pass over the training data to compute the channelwise mean and variance of images for later standard normalization is performed. A second full pass over the data, this time applying the previously computed normalization is applied and another PCA model is fit; different from during segmentation as it excludes background information.

[0234] After both initial transformation fitting passes, they are applied inline during training after the augmentations described in the following section. As with color augmentations, transformations are applied exclusively to the fiber pixels. Concurrently to fitting the image transformations, a separate normalization and PCA transformation is learned for the supplemental features and applied in the same way.

[0235] Following normalization and PCA, the tile background is set to -2, a constant value outside the range of values observed in the sample. 4.2 Augmentation

[0236] Random channelwise color augmentations are applied based on statistics obtained from the background portions of the image during the initial segmentation. For each image, the mean of each color channel is taken in mode centered L*a*b* space. The variance of the means are then taken and channelwise color shifts to the training images are applied by sampling a normal distribution of mean zero, and variance as estimated by taking the variance of the imagebackground region channelwise means. Particularly, augmentations are applied only to the fiber region covered by the mask computed during segmentation so that all tiles use the same background value. Any time modeling choices can be made objectively, with no hyper parameter adjustment or choice left to the scientist, it satisfies the natural desire for an elegant solution and often improves generalization, rather than overfitting hyper parameters to training and validation data. Microscopy systems and the images they produce over time undergo long term shifts in background illumination or inconsistent preparations. These variations should not affect the outcome of the proposed method so adding such random color shifts to the training data mitigates the possibility the model will learn spurious partitions of the data by memorizing specific color schemes. Because fibers have no natural orientation on a microscopy slide, rotating them does not change their identity, and any system should be robust to rotational variations. Unbounded random rotation augmentations are applied, with constant padding of -2 for pixels that are undefined after rotation, as well as random left / right and up / down flips. 4.3 Preliminary Network Architecture

[0237] Preliminary classification experiments are intended to yield an understanding of the role of shape, color, and texture features toward fiber classification. We use a simple architecture, depicted in Figure 15, enhanced by precomputed aggregate features, such as mean width or color.3 layers of 6x6 kernels were used with 12 filters each with intermediate 2x2 max pooling and batch normalization. Precomputed features are passed through a branch consisting of 3 dense layers with 20 neurons each. Following the final convolution, a global max pooling operation is applied to flatten the spatial dimensions of the image inputs, resulting in a vector of spatial features. The spatial features are concatenated with the dense branch output and this collection of aggregate and spatial features is used by a final linear classification layer. 4.4 Network Architecture

[0238] While fibers are visually rich, the variety of features observed does not necessarily translate to features that can easily be separated between classes by the model. High capacity models, such as those commonly used on the imagenet benchmark (resnet, inception, xception etc) commonly used for other more mainstream computer vision applications tendto overfit quickly and consume significant GPU resources, with or without transfer learning from imagenet. Rather than learning to recognize a wide variety of shapes and textures to identify visually very different classes in other applications, the sheer diversity of apparently non- informative features observed across images and classes means the problem is much more likely to revolve around identifying a limited collection of features and compositions thereof, such as the spacing or frequency of pits and ridges, edges thickness, or shape of the tips. For this reason, a small network consisting of a convolutional branch and a densely connected branch, concatenated prior to a linear softmax classification layer, is used.

[0239] The convolutional branch of the architecture consists of 4 separable convolution layers, each separating the input into component channels and each consisting of 4 kernels per channel. Separable CNNs are used as linear correlations between input color channels are already removed with PCA and different features seem to appear better in one channel or another. This makes learning separate kernels for each channel, and then learning linear combinations between the resulting feature maps, tasks separable CNN’s are intended to do, an obvious choice. The first layer Figure 16 uses a kernel of size 7x7, so as to learn an initial set of larger scale texture filters more directly and, with two input channels, is still computationally affordable. Subsequent layers use a kernel size of 3x3 to allow the model to learn the spatial compositions of the first layer’s textures. The first two layers both learn 4 pointwise linear combinations of their respective feature maps and the last two use 8. The architecture is arranged with pooling layers between the convolutional layers, with increasing number of output pointwise combinations, spatial information is projected into an increasingly higher dimension feature domain but spatial domain with smaller dimensions. Maximum pooling is of size 2x2 is used following each intermediate separable convolution layer, with the last layer using global maximum pooling over each channel. Additionally, the input image is resized by ½, ¼, 1 / 8, and 1 / 16, padded on the edges with zeros, and the channels are concatenated directly with the max pooling output of at the layer of corresponding size, skipping the prior layers. This allows for direct operation on the input image at each scale, in addition to textures learned by previous layers. Finally, the features from each layer are pooled and passed directly to the final classification, skipping the later layers. Batch normalization is applied once, after the random rotation, and before the first convolution operation. Excluding the skip connections, this model issimilar to VGG or Alexnet, and the skip connections are intended to incorporate a feature pyramid in the architecture.

[0240] Two dense layers with 10 neurons each are used to bottleneck the 35 supplemental features remaining after PCA into a stronger, lower dimensional, latent representation. Batch normalization and Gaussian noise of standard deviation of 1 was applied to the input to the first dense layer. Dropout is applied at a uniform rate of 0.3 prior to the first dense layer and decreased to 0.2 prior to the second. A batch normalization step follows each dense layer. Note that typically, batch normalization should be applied immediately before the dense layer, and after the noise and / or dropout layers. This should be corrected if future models require a dense branch, which is discussed further in future chapters. L1 and L2 regularizations of 1E-4 are applied to the dense weights and an L2 regularization of 1E-4 was applied to the bias parameters in order to weaken the effect of the precomputed feature information on training so that it supplements, rather than overpowers, the convolutional branch. Gaussian error linear units are used for a nonlinearity, rather than rectified linear units, due to their demonstrated performance improvement over the latter.

[0241] An exemplary overview of a network structure is depicted in Figure 17. 4.5 Training

[0242] An early stopping condition is used to terminate training when the validation loss has not decreased in 5 consecutive epochs, 1 more epoch than the cycle length of the learning rate schedule, giving the model an opportunity to escape a local minima or saddle point that might cause premature termination.

[0243] A cyclical learning schedule, a strategy for scheduling learning rates during training in a cyclical pattern with average rate decaying over time, was used. It is suitable for large scale learning as it helps escape large plateau like saddle points or local optima, and requires significantly less tuning than other learning rate schedules. For cyclical learning, there is a risk of incomplete learning with too fast a decay. This risk is acceptable in order to keep training time manageable. Another advantage of cyclical learning is the ability to test multiple learning rates during the same experiment. Anecdotally, training with this data tends to get stuckin poorly generalizable saddle points and local optima with default optimizer strategies, regardless of choice of learning rate or momentum, so strategies like reducing learning rate with each training plateau are not as effective. Max and initial learning rates of 0.02 and 0.0001 were selected based on early experimentation with 1 / 30th the data without cyclical learning, and provide a reasonable range. Much higher learning rates are unstable with cost increasing to infinity or repeatedly overshooting local minima. Too low, and learning is too slow to be practical and can settle in shallow saddle points or local optima it would otherwise escape. 4.6 Anomaly Detection

[0244] Accuracy varies by slide in the training dataset. Slide preparation consistency matters, but long run variability in human systems is inevitable. Fragile, or brittle, the system is likely to break when stretched too far from its training data. Long run variations in fibers have never been studied in this way before. In conjunction with model predictions should be a system that alerts the user when strange fibers appear.

[0245] Anomaly detection proceeds similarly to removal of the background pixels, using a combination of PCA and GMM. Here, instead of operating on individual pixels in a 3- dimensional color space, anomaly detection is performed on whole segments, using the features learned by the model's final layer, and is used to empirically estimate the probability density of the training data; allowing for the detection of anomalies in production by isolating segments that have very low probability of occurrence under the training distribution. Additionally, anomaly detection mitigates generalization risk as it has the capability of identifying long range drift in the distributions of the data observed during production. The feature vector representation of each segment image in the training data is reduced from 35 dimensions to 25 using PCA and dropping the least significant components accounting for a cumulative 5 percent of variation in the data explained by all components.

[0246] Figure 18 is a flowsheet of classification and post-processing operations.

[0247] The probability density of the feature space of the training data is then estimated using a Gaussian mixture model with 15 components.QUANTIFICATION

[0248] A fiber drawn from the process might be, randomly, either hardwood, or, softwood, like flipping a coin, also known as a Bernoulli trial. In these circumstances, the number of one fiber type out of all fibers observed follows the binomial distribution, and using its properties it is possible to estimate confidence intervals within which the process composition is likely to be, if each fiber is identified accurately. Of course, no test is perfect, adding uncertainty. Knowledge of the test’s distribution of misclassification rates can be incorporated into the quantification of composition and its uncertainty using equation 5.1.

[0249] Where µobs is the observed composition, µobs_corrected is the corrected observed composition, pfsis the probability of falsely classifying a hardwood as a softwood and pfhis the probability of classifying a softwood as a hardwood.

[0250] Like process composition, validation accuracy, or false positives or negatives, can also be modeled using the binomial distribution. Whether a fiber is correctly classified as its ground truth label or not can be considered a Bernoulli trial as well. However, uncertainty again arises when considering how sample preparation and imaging conditions vary between slides. Inconsistent staining, background illumination, or debris, among other sources, all vary between slides and affect the model’s accuracy. Random accuracies can be modeled using the beta distribution. Further, by compounding the beta and binomial distributions, where each trial is a coin flip with a random accuracy, the beta binomial distribution is constructed; an essential model for estimating the parameters controlling the randomly distributed rates of correct prediction. Parameter estimation is performed by their direct optimization to maximize the likelihood equation of the data given in equation 5.2.Whereand

[0251] After estimation of the beta binomial parameters, performance of the system is evaluated by determining the length of the 90% confidence interval expected of the system under identically distributed preparation conditions. Confidence intervals are obtained using the bootstrap method as described in algorithm 1.SIMULATION 6.1 Motivation

[0252] It is impossible to evaluate the theoretical limit of the system’s performance when the underlying feature distributions (and even features) are unknown. This makes it difficult to discern if a particular system has reached its performance limit with respect to the model configuration, dataset size, hyper parameter tuning, or statistical limit in distinguishing one perfectly characterized population from another. Fibers are ideal objects for simulation due totheir simultaneous simplicity, diversity, and ease of experimental observation. In order to evaluate the performance limit, the system’s performance is tested using simulated microscopy images. In this way, it is enabled to exactly control which features are present in the simulated images, control the statistical distributions of different fiber species and background noise, and gain exact knowledge of the individual fiber ground truth in mixture images, which is not possible without manual labeling in real fiber mixtures.

[0253] Providing prior knowledge to the system in terms of pretrained models, or handcrafted “expert” features can improve the model performance. While such features are explicitly computed and incorporated into the model in the previous sections, this section explores a different method for their incorporation. By simulating data from a simplified, but representative distribution, it is possible to provide a reasonable starting point for closely related data as well as provide an estimate for performance, providing clues for future architecture choices and informing hyper parameter optimization. Extending the concepts, it’s easy to see how a model backbone incorporating exact knowledge from a simulated ground truth might have additive utility as a standalone feature extractor, or as a starting point for fine tuning. While the length and the width can exactly be known, the model is desired to learn the convolutional features it needs to calculate those values on their own. Yielding parameters and features that can be transferable to adjacent tasks. This methodology also enables hyperparameter optimization to get as close as possible to the theoretical limit (with the least data). 6.2 Methods

[0254] For this work, first is to simulate microscopy images following the algorithm 2 below. A collection of fibers are initialized by sampling their widths and lengths from specified Gaussian distributions. Three different fiber types are simulated, 2 with the same width, and 2 with the same length, as seen in the scatter plot of Figure 19. An overlap in the distributions of our simulated fiber types was intentionally incorporated for emphasis that, in a real-world problem fibers whose observable feature distributions overlap cannot be separated.

[0255] Fibers are placed randomly on the 2d X-Y plane by choosing an origin point and random rotation between 0 and 2 pi. Wraparound boundary conditions are used to account for fibers that cross the image boundary. Fibers are placed in a random order and fiber opacities aresimulated using “Alpha blending” using equation 6.1 and 6.2. In equation 6.1, the alpha or transparency of the new image a01 is computed by linearly interpolating the transparencies of the background image, a1 and the image to be placed over the background a0. The color of the new image rgb01is then computed by linearly interpolating between that of the background, rgb1, and the "over" image, rgb0, using the respective a values to control the extent of interpolation.

[0256] Fiber color is held constant, but luminance and opacity are randomized. In order to add realistic spatial variations to the images, as well as incorporate unique patterns into each image (theoretically enabling the memorization of such patterns), a background, simulated as a fractional Brownian motion (fBm) surface using Stein’s method, is incorporated. The surface generation is parameterized by a single variable, the Hurst exponent, or roughness, which varies between 0 and 1 and is representative of the tendency of variations in the image to occur over a long range, smooth, or short, rough. A high value of the Hurst exponent indicates smoothness, and low value indicates roughness. Although more variability may be observed in the background of real microscopy images than is representative with fBm surfaces (let alone in fibers represented with rectangles), they represent the kind of variations an effective model should learn when to use and when to ignore. Each fBm surface is generated with a random roughness and random color as described in algorithm 2. Surfaces of various roughness can be seen in Figure 20. Each image is simulated independently from each other image. Note that in the previous sections discussed above, due to the details of the image collection campaign, large numbers of images (each slide) have effectively the same background color distribution. As every image is independent from each other, additional background variation would be ideal, but not practical to obtain. Additional information or features can be included in the images at additional computation cost, such as simulated pits of various size and circularity, or fiber roughness independent of the background. Simulations are primarily limited by the size of theBrownian surfaces. For this work 256 x 256 is used, in contrast to the 1920 x 2560 images used in prior sections. The resulting images as shown in Figure 20.

[0257] The accuracy ceiling for the simulated fibers is determined by calculating the likelihood of observing each fiber under each of three distributions, using exact knowledge of the fibers length and width and exact knowledge of the generating distribution parameters. The predicted class is then the class whose distribution is the most likely. The ceiling is also determined when only the width is known. In real scenarios, the generating distribution parameters can only be estimated, and only if the class and the features that are being looking for are already known. When the parameters of the source distributions and fiber width are known, a theoretical accuracy ceiling of 64% can be achieved. When both are known, the theoretical ceiling is 96%. Note that because the fiber classes are balanced, a baseline, random chance, model would yield 33% accuracy. RESULTS AND DISCUSSION

[0258] All processing is performed on a 4gb Nvidia Jetson Nano. Images are stored in their original zipped folder on a ( ) USB HDD. Intermediate segmentation results are stored on a Samsung T7 USB SSD

[0259] From segment extraction through training the classification model, detecting anomalies, and simulating realistic process performance, the system took 6 days to extract segments from 30 images each from 423 slides. The neural network was trained in roughly 4 days. Testing was performed on 30 images each. Albeit cumbersome, that such a task is possible on this type of device is notable and opens up many opportunities for deployment in a manufacturing environment. 7.1 Preliminary Results

[0260] A preliminary sample of labelled tiles containing portions of individual fibers was constructed using approximately 0.3 % of available data. Specific sample sizes and outlined in Table 1. Training, validation, and test sets are partitioned by slide. A tile size of 256 x 256 and restricted the dataset to objects with an area of 25000 pixels or greater, nearly 40 percent of a single tile (see Figure 23 for perspective). Both choices represent an attempt at subjectively balancing the number of samples that meet the threshold, with large enough minimum segment size to be representative of an entire fiber (consequently eliminating non-fiber objects).

[0261] During the segmentation process, summary statistics are collected at the image and fiber level. Image level observations are useful to ensure the train, validation, and test splits are not substantially different. Fiber level statistics on their own are significant predictors of class. Both average width (obtained during medial axis skeletonization) and color possess high level partitions between fiber types. Softwood, coniferous fibers are wider, and typically lighter colored, hardwood fibers are significantly narrower and darker due to lower resistance to staining. The distribution of fiber widths, and mode centered, PCA transformed color, for the various types, is depicted in Figure 21. Table 1. Summary of Preliminary Dataset7.2 Preliminary Classification Results

[0262] Three models were trained, using each branch individually, and using both branches. While the individual branches both achieve an out of sample test accuracy of approximately 30 %, compared to a baseline of approximately 13 % for totally random predictions. Combining both branches, with no other changes, yields approximately 43 % accuracy. The feature branch allows the model to learn color or width features that would otherwise consume network capacity. However, increasing capacity increases risk of overfitting. By providing prior knowledge to the model, albeit very specific, capacity can be added to the model, but it maintain a similar risk of overfitting. This allows incorporation of very complex prior information, but can add noticeable overhead, particularly during inference when examples are not reused. 7.3 Discussion of Preliminary Classification Results

[0263] Studying the confusion matrix in Figure 24, cotton, eucalyptus, and birch fibers are all classified with greater than 30 percent accuracy. Further, softwoods, alder, aspen, birch, and eucalyptus and hardwoods, contorta, pine, and spruce are easily distinguishable from one another, so the majority of misclassified fibers are at least from the same group as the erroneously predicted class. Inspecting the kernels learned by the model also provides clues towards more suitable network architectures. Figure 25 depicts the convolutional kernels learned by the network during training and provides clues towards more suitable network architectures. While it looks like some textural features may have been learned in the first layer of the model, it’s possible they simply depict noise. It appears that most of the kernels learned shape features. It’s possible that there was not sufficient capacity for all informative features, or some subtler features were not observed frequently enough. Complicating the problem, there is no orientation to the fibers, so multiple rotated variants of the same kernel are necessary to detect the same feature in any rotational configuration.7.4 Segment Extraction

[0264] Results of the segmentation process for a single image are shown in Figure 10. By removing intersections, most resulting segments are isolated from a single fiber. This enables the generation of a set of object level annotations from image level class labels. This process is applied to 30 images consisting of seven hardwood and softwood species. Additionally, the system, trained using only softwood and hardwood images is used to generate segments from 30 images from each of 2 cotton slides. The dataset resulting from the segmentation process is summarized in Table 2. Table 2. Summary of Dataset7.5 Classification

[0265] The system achieves 91 percent accuracy classifying hardwoods and softwood fibers, as measured on the out of sample test set. Additionally, misclassification rates are balanced between both, with 9 percent of each group falsely classified as the other. Figure 26 shows a confusion matrix of test set classification results.

[0266] Gradient Class Activation mappings, GradCAMs, use the gradient from the final convolutional layer, of the cost for a given class to visualize which pixels in the input image most strongly relate to that class. Figures 27 and 28 show GradCAMs computed for the best and worst classified images (respectively), as measured by the magnitude of the difference of the class probabilities and the ground truth. It can be seen that the model has learned interesting textural variations. Visually, the contorta fiber confidently misclassified as a hardwood appears to be an artifact of the segmentation process, with large portions apparently removed with the intersection. The resulting segment appears curved, like eucalyptus, and thin, like other hardwoods. In contrast, the most confident, correctly classified fiber, a eucalyptus, appears to focus on points tangential to where the intersection is removed. Possibly focusing on both the texture, as evidenced by activation in early layers, as well as estimating the width by looking at the slope of the fiber edge at the center-point of the fiber, touching the intersection. Although notconcrete, GradCAMs are a critical component to evaluating and debugging a CNN. In this case it appears the model has learned both spatial and textural features and their compositions.

[0267] While the confusion matrix (e.g., shown in Figure 24) provides an overall performance metric and the GradCAMs give intuitive, subjective, evidence that the CNN has learned the desired structure, they provide a poor intuitive visual for understanding more generally how context affects classification performance. For this analysis, Figure 29, the proportion of correctly classified fibers is computed for each image in the test set, and 95 percent confidence intervals are estimated for accuracy are estimated using the Wald approximation. Images with the highest lower confidence interval are chosen to represent the best classified images, and likewise, images with the lowest upper confidence interval are selected to represent the lowest performing images. For each image, all segments classified by the system are outlined in green for correct classifications, and red for incorrect classification. The confidence for each is displayed in the miniature bar chart plotted overlain on the image, adjacent to its respective fiber. The left bar corresponds to hardwood confidence, and the right bar corresponding to softwood. Incorrectly classified fibers have a red color bar, while correctly classified fibers have a green bar. This shows overall trends in classification performance, as well as visualization of segments in their original context and where the system makes confident or uncertain predictions. Building on the image level evaluations, confidence intervals for the classification accuracies for each slide are shown in Figure 30 (left panel), with totals for each predicted class in Figure 30 (right panel). Importantly, rate of correct classification varies by slide, consistent with the intuitive hypothesis that varying staining conditions, imaging conditions, and sampling conditions, yield observed segments that are not identically distributed. As shown next, this variability controls how many samples are required to make actual estimates of the true process composition. If the slide itself is interpreted as a confounding factor, when there is a change from one slide to another, the fiber type is potentially changed. Additionally, the method can be affected by different staining and imaging conditions. This and the observable differences in the different images in the previous figure shows, by definition, clear evidence of confounding.7.6 Quantification

[0268] The estimated probability density function for misclassification rates for an individual slide is shown in Figure 31. While the overall accuracy of both species is balanced, the distribution of misclassification rates is imbalanced, due to more variability in accuracy across slides.

[0269] A histogram of simulated outcomes using representative conditions is shown in Figure 32. A scenario representing a process composition of 50 / 50 hardwoods and softwoods, of 1000 fibers per slide, with misclassification rates estimated from 25 simulated validation slides, and performance evaluated on 5 test slides is simulated 10000 times and a 90 percent confidence interval of ±3.5%, or 7% long, is estimated by taking the middle 90% quantile of simulated outcomes. The corresponding outcome for various numbers of slides and test fibers is shown in Figure 33. 7.7 Anomaly Detection

[0270] Histograms of segment likelihoods for the training, validation, and test sets, as well as cotton are shown in Figure 34. Notably, segment likelihoods occur with visually equivalent probability densities, indicating model stability across the splits. However, cotton fibers have significantly lower observation likelihoods, 77 percent of cotton fibers have lower likelihood of occurrence under the training distribution than 95 percent of the training data. This is sufficient for indicating when anomalous fibers are present in the process, provided they have an equivalent magnitude of difference from the training data. 7.8 Classification of Fiber Mixtures

[0271] Finally, 8 slides with mixtures of fibers are processed, and for each an example annotated image is shown in Figure 35. While the ground truth is unlabeled and unknown, visual evaluation of annotated results confirms the model is consistent when extended to mixtures. Surprisingly, these images provide not only additional evidence of confounding but some more clarity about the source. In Figure 35, the first 3 panels in the bottom row show the sameartifacts. A large, faint, orange smudge, emerging from the bottom middle of each image. Now check the shape and location of the light grey dots and notice the correspondence across the images. While most of them would be removed with the background, those that overlap the translucent fiber, either originating from the background or foreground with different implications, can affect the images of any fibers observed in that position. If such unique markers are present in the training data, the model can learn to identify and erroneously relate their presence to a specific fiber type. 7.9 Simulation Results

[0272] The effect of the system’s capacity on its accuracy is studied by varying the number of convolution filters. In order to explore only the impact on the convolution component of the model we eliminate both the precomputed features branch as well as the skip connections. Using 4 layers of separable convolution, we keep the number of filters the same across layers and observe the classification accuracy, with respect to simulated fibers, versus an increasing number of parameters (e.g., as shown in Figure 36).

[0273] Comparing the ceiling with the capacity experiment results, we can see that we are approaching the theoretical ceiling when width is known. Although seemingly trivial when given the exact features, to learn them from the images themselves requires learning a complex subset of features which separate background noise from fiber edges as well as how to compose combinations of these subset to identify what is obvious at large scale, but difficult to see when only looking at 7x7 patches of pixels at a time as in layer 1. While the model can only learn to use one feature (width, because that is the only generating feature observable after removing intersections that differentiate fiber types), learning it from the small scale forces the model to learn a set of building block features that can be useful for other tasks. 7.10 Exemplary Use Cases

[0274] The present disclosure provides an array of systems and methods for characterizing fibers (e.g., cellulosic fibers) which may, in turn, be useful in several downstream processes. From the creation of new products to the enhancement of otherwise existing products,pure populations of and / or specific compositional mixes of fibers can be useful, and those of skill in the relevant art will recognize such uses.

[0275] Any of a variety of fiber sources may be used in accordance with the present disclosure. In some embodiments, fibers may be or comprise natural cellulosic fiber and / or regenerated cellulosic fibers. By way of non-limiting specific example, fibers may be sourced from any wood (e.g., soft or hardwoods), cotton, linen, rayon, lyocell, modal, Tencel, Viscose, bamboo, hemp, jute, soy, corn, sorghum, wheat, flax, and / or nettle.

[0276] Various populations of fibers (e.g., those form particular plants or other sources) can have one or more property that makes it particularly suitable for downstream applications. In some embodiments, fibers characterized in accordance with the methods and systems of the present disclosure may be separated into compositions selected for one or more particular property. Exemplary properties include softness (e.g., for clothing and home textiles), absorbency (e.g., clothing and home textiles), comfort (e.g., clothing), durability (e.g., textile applications), versatility (e.g., textiles, industrial fabrics, clothing), environmental sustainability, and affordability. In some embodiments, provided systems and methods are useful in creating various forms of paper. In some embodiments useful in paper making, some of the processes and characteristics are described in Malachowska et al., Sustainability, 12, 7219 (2020), the disclosure of which is hereby incorporated by reference in its entirety. 7.11 Discussion

[0277] The overwhelming difficulty of the fiber classification problem is that the objects themselves are very simple with overlapping distributions of both simple and subtle features as well as confounding background features. Providing the model with a backbone of fundamental knowledge of the expected features makes identification of subtle features more effective, despite context variation.

[0278] Fortunately, the baseline system achieves an accuracy of 91% when classifying hardwood and softwood fibers. This accuracy cannot be achieved using width and color features alone, which must imply the presence of smaller scale features, like texture or edge patterns, soobservation of a complete fiber may be unnecessary for acceptable results. Leftover information from segmentation, particularly the distance to nearest edge transformation of the fiber network is computed but not used after intersection elimination. This map could be included as an additional channel to the input images, assisting the model with the fundamental task of learning the species-specific compositions of texture with shapes.

[0279] There are no small scale, or subtle, features present in the simulated fibers and the model is therefore using all of its capacity to learn the shapes. With the baseline model, the fiber color and shape features are precomputed, so it is not necessary for the model to learn them. Additional capacity causes overfitting, even with relatively few parameters, strongly suggesting the presence of obvious (to the network), but spurious, partitions in the training data unrelated to the species. significant diversity of small scale / textural patterns. This is intuitive given the unique life cycle of each individual tree, let alone fiber. Process conditions and equipment cause even more variability. While some subset of these diverse features should be useful, we are further confounded by variation in the staining and imaging conditions of the fibers, to which a good model should be invariant. More capacity may be necessary to extract the basic shape features, however this capacity will be used for memorizing context and overfitting to training data unless care is taken.

[0280] For this contrived task, the model necessarily learns a concept of fiber width, as it’s the only feature used to differentiate fiber species during simulation (besides length, which, as discussed, is not interpretable by the model). To have learned width means that the model has some concept of a distance measure between edge pixels. While softwood species are not separable based on width, the arrangement of pit features relative to the general shape of the fiber might be. Incorporating the fundamental features into the model should better guide the learning through the possible feature space. Rather than learning both features and spatial arrangement simultaneously the model is allowed to first learn features which describe the arrangement, from simulated data; preserving experimental data to inform subtle features. Establishing whether or not the convolutional branch of the baseline system learned to coax both spatial, and textural features through the network kernels requires additional evidence.

[0281] Prioritizing possible improvements to the baseline is challenging because the theory of network architecture selection and tuning has not yet caught up to our ability to executeincreasingly complicated networks. However, some intuitive reasoning based on the simulated dataset results yields a multitude of viable pathways. Allowing the model more capacity to simply learn shapes at multiple scales should better inform its knowledge of composition. Since texture is a high variance feature, prone to memorization and confounding with the background, a black and white (binary) branch should regularize the network by dedicating capacity towards the interpretation of fiber shapes. This branch may or may not be pretrained on simulated fibers and may or may not pass its features to the textural branch. Obviating the pretrained option, simulated examples need not be simulated microscopy slides, but could be simulated tiles. Since the computational cost of simulated Brownian surfaces increases with the size of the surface, tiles can be generated efficiently enough on the fly to enable an effectively infinite training dataset, even after incorporating additional complexities like separate fiber and background roughnesses.

[0282] Additional experimentation using simulated fibers at multiple scales can inform how the network composes low and high resolution. Full resolution should mean that the model learns edges and how to distinguish edges from background noise in early layers and how to compose them in deeper layers. Low resolution implies interpretation of shapes as a whole. Both of which might be valuable depending on if you want to give the model a shortcut to understand those features (so it can use preliminary layers to understand texture), or if you want the model to share features informing the construction of large shapes from small features identified in early layers. Both choices represent a balance between overfitting and capacity given the availability of data.

[0283] Finally, the work can be extended by using the pixel level ground truth from the simulation and enumerating individual simulated fibers to train a segmentation model. This model can then be a starting point for transfer learning using synthetic mixtures of real fibers and background illumination, providing an efficient starting point for further fine tuning with a small hand labeled dataset. CONCLUSIONS

[0284] The system achieves an overall accuracy of 91 percent with variance in accuracy between slides defining how many physical slide samples must be prepared Slide preparationand imaging is the slower step in actual application as the Nvidia Jetson can easily compute a large numbers of fibers per slide; enough to effectively eliminate uncertainty from the binomial distribution and leaving only uncertainty in the beta distributed slidewise misclassification rates. Additionally, anomalies that are significantly different enough from segments observed during training, such as cotton fiber, can be detected in the pulp. Subjectively, the system generalized well to slides with mixtures of fibers, proving the initial hypothesis that AI is useful to fiber characterization.

[0285] With respect to simulated fibers, the system, and limitations it introduces, approaches but does not achieve the theoretical accuracy ceiling. This implies that the baseline system, in its current configuration, specified herein, has potential for improvement. However, in realistic data, features are often correlated, such that in the absence of one we might make inferences from the others. Despite not achieving the theoretical limit with respect to simulated fibers, the system achieves approximately 91% accuracy, classifying real softwood and hardwood fibers, with respect to the experimentally collected images using a comparable capacity. Since the width and color features alone are insufficient for such accuracy this implies both the presence of easily distinguishable low-level features and suggests the potential for additional improvements with additional model capacity. However, such improvements depend on breaking the effect of the confounding variables and the memorization of unique textural patterns present in the training data.

[0286] With a baseline method for process composition estimation from microscopy images, as well as a suite of visualization, simulation, and testing tools, established, the choice of which improvements to prioritize becomes more obvious. Balancing cost to implement against the magnitude of likely improvement, or risk versus reward, informs a strategy for future work.

[0287] As we concluded, strategies for dealing with confounding should be prioritized. In the long term, during continuous application to an industrial process, the data collection should be the focus. Potential strategies used in automated cell counting to deal with instrument drift; such as separate collection of contemporaneous background images, analyzing, instead of the raw image, the result of subtracting the background from the image of interest. We also see value in learning high level shapes using a high-capacity network applied to binarized fibers. Since we presume that low level, textural features are problematic as confounders, eliminating themaltogether is one option. Further, by pretraining a binary branch, we may be able to transfer some notion of large-scale composition and use fine tuning to learn texture. However, the risk remains that there is preference to learn confounding factors.

[0288] We propose that this task is equivalent to learning a composition of the background distribution with the fiber distribution. From our perspective, we see that large portions of each image are exclusively background. Most simply, by collecting a dataset of tiles from the background, in addition to fibers, we can augment our dataset by blending tiles from the background and subtle color shift augmentations with fiber tiles; thus, attempting to create a more continuous, rather than discretely partitioned by slide, dataset.

[0289] With the advent of generative models in text and image synthesis, we would be remiss in their exclusion. With the easy access to purely background tiles, there are numerous techniques with which we can learn, the distribution of fibers, and separately, the underlying noise distribution. Depending on the goal, generation of new background samples, or inference of the real-world sample distribution, different models are recommended. Recent literature tends to prioritize the generation of new samples, rather than inference tasks like anomaly detection.

[0290] Looking beyond the issue of confounding, the present task only dealt with classification of virgin softwood fiber. Of great importance to papermakers is understanding the impact of recycled fibers of unknown source. Moving towards learning features from the data, rather than simply processing the data, self-supervised techniques could be adopted. The jigsaw reconstruction task, where a model is trained to reassemble a randomly permuted grid of the original, is an intuitive choice as the features we hypothesize to be useful for characterization appear visually to be useful for image reassembly. Additionally, we note that such a task would force the model to learn fine-grained distinctions between visually very similar objects. Latent representations is learned in this way should be a useful starting point for subsequent tasks like density estimation.

[0291] Finally, we consider the incorporation of wavelets, such as the Gabor filter, with powerful flexibility to represent structures we know to be present in the fiber, such as edges, pores, and roughness, into existing networks. Inspired by the observation that the early layers of convolutional networks learn Gabor-like filters, the implementation of an 11x11 kernel, restricted to a Gabor function with learnable parameters, as the first layer of a convolutionalneural network may be advantageous. As a plausible building block, the development of a layer built by composing two banks of separable one-dimensional kernels, restricted to families of wavelet functions and acting separately on the x and y axes. Such a construction would be more computationally efficient for larger scale kernels as well as maintain the property of allowing multiple pathways through the model. 8 Exemplary Testing of Unknown Samples

[0292] In addition to embodiments described above, the present example describes additional improvements to methods and techniques described for fiber characterization.

[0293] A goal of this example is to demonstrate an automated computer vision and machine learning method to estimate compositions of mixtures of recycled old corrugated cardboard (OCC) mixed with unbleached Kraft pulp (UKP) from process fiber analyzer images or other microscope based images.

[0294] Additionally, among other things, this example demonstrates improved understanding of the fundamental fiber features that are visible in images and confounding factors introduced by the imaging instrumentation, data collection plan, and manufacturing process.

[0295] Figure 37 shows a series of images related to the image segmentation processes. Figure 37 panel A shows a grayscale image of an exemplary fiber sample. Figure 37 panel B shows a binarized image of the fiber sample. Figure 37 panel C is a skeletonized image of the fiber sample. Figure 37 panel D shows distances from edge (e.g., how far a given pixel is from a detected edge). Figure 37 panel E shows fiber segments with intersections of overlapping fibers removed.

[0296] Figure 38 shows a series of images comparing known and labeled recycled old corrugated cardboard (OCC) fibers and unbleached Kraft pulp (UKP) fibers.

[0297] Figure 39 shows a series of tiles for samples UKP fibers and OCC fibers illustrating different transformations of samples including a skeletonized image, an image showing pixel intensity corresponding to the distance of the pixel to the medial axis (e.g., thedistance from points on the skeletonized image), the distance of a pixel to a detected edge, and intensity of the pixels.

[0298] Figure 40 shows a series of streams from which fiber samples are obtained. Each stream from which samples could be taken are identified as 1-8 in the figure.

[0299] For classification, a series of steps are used to train and transfer learning models based on samples obtained from various streams as identified:

[0300] Stage 1 (unrefined fibers): the model initializes parameters randomly and trains using unrefined samples of UKP and OCC fibers.

[0301] Stage 2 (lab refined fibers): the model incorporates learned parameters from stage 1 and trains on refined fiber samples.

[0302] Stage 3 (industrial refined fibers): parameters from stage 2 are fine-tuned on industrial refined fibers.

[0303] Parameters learned in stage 2 were found to generalize well to industrial data. The fiber level accuracy for pure samples is consistent between lab prepared, University of Maine (UMO) and industrial instruments (~70%) using a stage 2 model.

[0304] Limited improvement in fiber level accuracy was found when comparing samples obtained at stage 2 versus stage 3.

[0305] During training, the network learns to project spatial features in the images into a spaceless feature vector. The probability density of the resulting latent space is estimated for both UKP and OCC separately. Figure 41 is a scatter matrix of the top ten latent features, OCC (Blue), UKP (Orange).

[0306] Following training of a classification model in which the latent space is learned, separately, two probability density estimation models are trained on both unbleached kraft pulp (UKP) and old corrugated cardboard (OCC, a source of recycled pulp). With these models, the likelihood of observing a mixed sample, containing both fiber types, is estimated under both density models. The likelihood of observing the mixed sample is maximized by optimizing / estimating the ratio of OCC to UKP.

[0307] Figures 42 and 43 show images of different fiber subtypes. Figures 42 and 43 are a byproduct / visual of the probability density estimate of the feature latent space. Two different density models were fit to the latent features learned during classification. One is fit using only the UKP fibers and the other using only the OCC fibers. The two different density estimates allow computation the likelihood a given fiber would be observed under each distribution, with can be estimated as a percentage of each fiber species in the mixture by maximizing the overall likelihood of observing a mixed sample. With a lot of samples, the percentage is theoretically equivalent to picking one fiber or another at random from the mixture. A Gaussian mixture model (for both UKP and OCC) where there are a user defined number of component Gaussian distributions, and during training the parameters of each distribution and the fraction of samples belonging to each component are estimated.

[0308] Each component represents a fiber subtype so Figures 42 and 43 show how a sample of fibers of each type are categorized into subtypes under each model. Figure 42 shows how UKP fibers are categorized into subtypes under the UKP density estimate on the right and under the OCC estimate on the left. Figure 43 shows OCC under the UKP estimate on the right and OCC estimate on the left. Several prototypical fiber subtypes are apparent. For example, the third row of the UKP subtypes (left side) in Figures 42 and 43 and 7th row of the OCC subtypes in Figures 42 and 43 are small, more circularly shaped fine particles. Some subtypes show separate fiber features like end points or curved versus straight edges. This is not presently directly used in classification but there are some UKP components that appear very infrequently in OCC fibers and vice versa. However, this visual is valuable for interpreting new data. Figures 42 shows pure UKP fibers under both models and Figure 43 shows pure OCC fibers, both prepared with University of Maine’s lab scale equipment. Figures 44-46 showed the components under each model for mixed samples processed by industrial equipment. In Figures 44-46, shapes that do not fit well with either of the distributions observed in the University of Maine (i.e., lab scale) training data. In particular we see a lot more globular particles with nonlinear edges that don’t appear to be an artifact of the system but rather a different fiber subtype indicative of different equipment.

[0309] Figure 47 shows an image comparing component fractions of a training set of images (left panel) as compared to actual images of mixtures (right panel). Mixture componentfractions were found to change proportionally with composition of the mixtures. In the left panel, blue is representative of UKP, while orange is representative of OCC.

[0310] Figure 49 shows a graph estimating the quantification of UKP and OCC and a simulated 80% UKP-OCC mixture (80% UKP, 20% OCC). A bias is observed when the maximum likelihood estimation of the ratio is applied to pure samples. For example, a pure UKP sample yields an estimate of ~10% OCC, while a pure OCC sample yields an estimate of ~85% OCC. The x-axis of the figure shows the proportion of predicted OCC in the sample, where 0 corresponds to no predicted OCC and a value of 0.9 would correspond to 90% OCC. The estimates shown were corrected during testing by transforming the observed interval from ~10%-85% to correspond to 0%-100%.

[0311] The above estimate provides an estimated ratio of OCC to UKP, which is converted to a mass fraction of OCC in the mixture. The conversion is described below:

[0312] n = number of fibers in sample

[0313] i = fiber index

[0314] X_i = feature vector for fiber i=1…n

[0315] y = species, y = 0 →UKP, y = 1 →OCC

[0316] P(X_i | y) = probability of observing X_i, given its species is y.

[0317] P(Y=1) = 1-P(Y=0) = Percentage of OCC in the mixture

[0318] P(X_i) = P(Y=1) *P(X_i | y=1) + P(Y=0)*P(X_i | y=0)

[0319] →Optimize Θ to maximize Σ ln(Θ *P(X_i | y=1) + (1-Θ)*P(X_i | y=0))

[0320] →The value of Θ where the above sum is maximized is the estimated composition of the sample.

[0321] Bias is corrected by transforming the interval so that composition estimates for pure UKP (100% UKP) and OCC (100% OCC) correspond to 0 and 1. Tile Fraction is converted to Area Fraction using the following formula: occ_avg_area*p_occ / (occ_avg_area*p_occ+ukp_avg_area*(1-p_occ))

[0322] Where p_occ is the estimated fraction of OCC tiles in the sample. The average areas for OCC (occ_avg_area) and UKP (ukp_avg_area) are computed from pure samples.

[0323] In order to address biases in the process due to fines, a fines correction process is used. The following steps are used to correct for fines:

[0324] 1. Process each image twice - once without removing any small objects, and once using a typical threshold. Record the number of segment pixels remaining in each image.

[0325] 2. Take the difference between the number of segment pixels in each image.

[0326] 3. Repeat the process (e.g., at least an additional time) recited in steps 1 and 2 for all images.

[0327] 4. Compute the mean rate of fines removal for both UKP and OCC.

[0328] Figure 49 shows an exemplary image without fines removal as compared to an image with fines removal. The method shows a slight bias by underpredicting the mass percentage of OCC in the composition from the mixture recipes as shown in Figure 50. Figure 50 is a parity plot, which compares the model prediction of the mass percentage of OCC in a recipe to the actual recipe mass percentage of OCC. The solid line is representative of the actual percentage of OCC, while the data points and dotted lines correspond to the model’s output. The model was trained and calibrated using only single fiber data. Mixture recipes are used only for evaluating accuracy and not during calibration.

[0329] Without wishing to be bound to any particular theory, the bias resulting in an underestimation of OCC content may be associated with an underestimation of fines content in the images. Particles smaller than 1 pixel cannot be observed in the images, but these particles represent mass that is accounted for in the weight measured by, for example, lab scales, or industrially, in mass flow meters. Though there is an underprediction, the strength of the system is in understanding the distribution of fiber characteristics, which can be supplemented by other methods to understand fine content.

[0330] Figure 51 shows the effect of the fines correction process described above. Fine correction resulted in a slight improvement to the accuracy of the model. Without wishing to be bound to any particular theory, fines removal may be a source of underestimation of OCC content (e.g., as shown in Figure 50). In Figure 51, the mean UKP fines removal rate (for apure UKP sample) was 0.029, the mean OCC fines removal rate (for a pure OCC sample) was 0.046. The mixture fines removal rates were as shown below:

[0331] 25% OCC: 0.032, 0.033,

[0332] 20% OCC: 0.030, 0.034,

[0333] 15% OCC: 0.032, 0.031

[0334] The values shown are the mean removal rate of OCC fines in the mixture.

[0335] The method is sensitive to the modification of the fibers as well as the overall solids content of the pulp sample. As a final challenge, blind samples were analyzed to estimate their contents, but the results showed a significant deviation from fibers observed in Figure 50. The method described estimates OCC fractions of 0.55, 0.54, and 0.78 for the three blind mixtures. Results are shown in Figure 52. The overall solids composition was significantly higher in the blind samples as compared to the test samples analyzed previously in Figure 50. Without wishing to be bound to any particular theory, the fibers were likely modified differently in the industrial samples as compared to those previously analyzed. A lab scale refiner was used in the preparation of the samples used for samples analyzed in Figure 50. An industrial scale refiner may have been used with the blind samples. In addition to refining the fibers, industrial refiners also chop the fibers, reducing their length and generating particles not observed when producing samples on a laboratory scale. Figures 44, 45, and 46 show different fiber subtypes found in industrial samples.

[0336] Figure 53 shows the cumulative distribution function (CDF, integral of a probability distribution function) for 3 different latent features for the two training samples (labeled UKP and OCC) and the 3 blind mixture samples from an industrial test source (Mix 1, Mix 2, Mix 3). The distributions are shown to differ between substantially pure samples from training data sets (OCC, UKP) and cumulative distributions from mixes generated industrial test data sets (Mix 1, Mix 2, and Mix 3).

[0337] Figure 54 shows a histogram of a measure of the likelihoods of observing a given fiber in a given sample, for a sample of UKP, OCC, and each of 3 mixtures. Lower numbers are less likely and higher numbers represent fibers that more common. The mean likelihood of observing a fiber from each sample is shown on the right. If the fibers of each species in themixture are identically distributed to fibers in the training data, the likelihood of observing a sample of fibers from a mixture should not be lower than the likelihood of observing a random sample of fibers from each species in training. In particular there appears to be a grouping of fibers in the mixtures which are unlikely to be observed under either the UKP or OCC training distributions. A small “bump” in the tail of the mixtures that is not present in the pure training samples, indicating that some portion of the fibers in the mixtures were not observed during training.

[0338] The results presented herein can be framed as a strength of the method as the method is able to characterize many features of the pulp that affect the final product and detect deviations that are not obvious with existing tools. It is also possible to use the system to identify fiber subtypes, such as certain types of fines shown in Figures 42 and 43 as well as in Figures 44, 45, and 46.

[0339] There are also several nuances differentiating the original dataset analyzed to produce the results in Figure 50 and the data generated in the work with industrial samples.

[0340] 1. Image modality: The initial work described above made use of microscopy images, in which a pulp sample is stained using a microscopy dye and imaged under a conventional microscope. The work on industrial samples made use of images taken using a fiber analyzer (a MorFi fiber analyzer). The fiber analyzer does not require the staining step, which eliminates some variation in the fiber images, but has a significantly lower resolution than the microscopy images.

[0341] 2. Pulp Type: The initial work described above explored pure, unmodified fiber species, the additional work explores refined unbleached kraft pulp and recycled old corrugated cardboard. There is some overlap in the distribution of fibers in the OCC, so some fibers appear similar to those present in UKP, with slight modification after one or more loops through the recycling process. 9. Software, Computer System, and Network Environment

[0342] Certain embodiments described herein make use of computer algorithms in the form of software instructions executed by a computer processor. In certain embodiments, thesoftware instructions include a machine learning module, also referred to herein as artificial intelligence. As used herein, a machine learning module refers to a computer implemented process (e.g., a software function) that implements one or more specific machine learning algorithms (e.g., neural networks), such as an artificial neural network (ANN), a convolutional neural network, random forest, decision trees, support vector machines, and the like, in order to determine, for a given input, one or more output values. In certain embodiments, the input comprises alphanumeric data which can include numbers, words, phrases, or lengthier strings, for example. In certain embodiments, the one or more output values comprise values representing numeric values, words, phrases, or other alphanumeric strings. In certain embodiments, the one or more output values comprise an identification of one or more response strings (e.g., selected from a database).

[0343] In certain embodiments, machine learning modules implementing machine learning techniques are trained, for example using datasets that include categories of data described herein, for example, microscopy images (e.g., stained microscopy images, unstained microscopy images), fiber analyzer images, and the like. Such training may be used to determine various parameters of machine learning algorithms implemented by a machine learning module, such as weights associated with layers in neural networks. In certain embodiments, once a machine learning module is trained, e.g., to accomplish a specific task such as identifying certain response strings, values of determined parameters are fixed and the (e.g., unchanging, static) machine learning module is used to process new data (e.g., different from the training data; e.g., infer a result) and accomplish its trained task without further updates to its parameters (e.g., the machine learning module does not receive feedback and / or updates). In certain embodiments, machine learning modules may receive feedback, e.g., based on automated review of accuracy or human user review of accuracy, and such feedback may be used as additional training data, to dynamically update the machine learning module. In certain embodiments, two or more machine learning modules may be combined and implemented as a single module and / or a single software application. In certain embodiments, two or more machine learning modules may also be implemented separately, e.g., as separate software applications. A machine learning module may be software and / or hardware. For example, a machine learning module may be implemented entirely as software, or certain functions of a module may be carried out via specialized hardware(e.g., via an application specific integrated circuit (ASIC), field programmable gate arrays (FPGAs)).

[0344] In certain embodiments, machine learning modules implementing machine learning techniques may be composed of individual nodes (e.g. units, neurons). A node may receive a set of inputs that may include at least a portion of a given input data for the machine learning module and / or at least one output of another node. A node may have at least one parameter to apply and / or a set of instructions to perform (e.g., mathematical functions to execute) over the set of inputs. In certain embodiments, node instructions may include a step to provide various relative importance to the set of inputs using various parameters, such as weights. The weights may be applied by performing scalar multiplication (e.g., or other mathematical function) between a set of inputs values and the parameters, resulting in a set of weighted inputs. In certain embodiments, a node may have a transfer function to combine the set of weighted inputs into one output value. A transfer function may be implemented by a summation of all the weighted inputs and the addition of an offset (e.g., bias) value. In certain embodiments, a node may have an activation function to introduce non-linearity into the output value. Non-limiting examples of the activation function include Rectified Linear Activation (ReLu), logistic (e.g., sigmoid), hyperbolic tangent (tanh), and softmax. In certain embodiments, a node may have a capability of remembering previous states (e.g., recurrent nodes). Previous states may be applied to the input and output values using a set of learning parameters.

[0345] A layer is a building block in a deep learning architecture composed of nodes. In particular, a layer is a set of nodes that receives data input (e.g., weighted or non-weighted input), transforms it (e.g., by carrying out instructions, e.g., applying a set of functions e.g., linear and / or non-linear functions), and passes transformed values as output (e.g., to the next layer). In certain embodiments, the set of nodes in a particular layer may share the same parameters and instructions without interacting with each other. A machine learning module may be composed of at least one layer (e.g., ordered). Examples of types of layers include convolutional layers (e.g., layers with a kernel, a matrix of parameters that is slid across an input to be multiplied with multiple input values to reduce them to a single output value); fully connected (FC) layers (e.g. all nodes are connected to all outputs of the previous layer); recurrent layers, long / short term memory (LSTM) layers, gated recurrent unit (GRU) layers (e.g., nodeswith the various abilities to memorize and apply their previous inputs and / or outputs); batch normalization (BN) layers (e.g., layers that normalize a set of outputs from another layer, allowing for more independent learning of individual layers); activation layer (e.g., layers with nodes that only contain an activation function); (un)pooling layers [e.g., layers that reduce (increase) dimensions of an input by summarizing (splitting) input values in defined patches).

[0346] In certain embodiments, the performance of a machine learning module may be characterized by its ability to produce an output data that reproduces an input data with specific accuracy. To achieve specific accuracy, a training process is performed to find optimal parameters, such as weights, for every node in every layer of the machine learning module. In certain embodiments, the training process of a machine learning module may involve using output data to calculate an objective function (e.g., cost function, loss function, error function) that needs to be optimized (e.g., minimized, maximized). For example, a machine learning objective function may be a combination of a loss function and regularization parameter. The loss function is related to how well the output is able to predict the input. The loss function may take various forms, like mean squared error, mean absolute error, binary cross-entropy, categorical cross-entropy, for example. The regularization term may be needed to prevent overfitting and improve generalization of the training process. Typical regularization techniques include L1 Regularization or Lasso Regression, L2 Regularization or Ridge Regression, and Dropout (e.g., dropping layer outputs at random during training process).

[0347] In certain embodiments, objective function optimization of a machine learning module may involve finding at least one (e.g., all) of the present global optima (e.g., as opposed to local optima). A typical algorithm for objective function optimization follows principles of mathematical optimization for a multi-variable function and relies on achieving specific accuracy of the process. Examples of objective function optimization algorithms include gradient descent, nonlinear conjugate gradient, random search, Levenberg-Marquardt algorithm, limited-memory Broyden-Fietcher-Goldfarb-Shanno algorithm, pattern search, basin hopping method, Krylov method, Adam method, genetic algorithm, particle swarm optimization, surrogate optimization, and simulated annealing.

[0348] In certain embodiments, available input data includes training data and validation data, e.g., where the validation data is separate and non-overlapping with the training data.Training data is used during the training process to optimize a model, whereas validation data is used to check the accuracy of the model while operating on previously unseen data. In certain embodiments, training data is divided into batches (e.g., portions) that is sequentially used (e.g., in random order) as sets of inputs to train a model. In certain embodiments, a model is trained multiple times (e.g., epochs) (e.g., at least two epochs, at least three epochs, at least 4 epochs, at least 5 epochs or more) on the entire set of training data.

[0349] As shown in Figure 55, an implementation of a network environment 5500 for use in providing systems, methods, and architectures as described herein is shown and described. In brief overview, referring now to Figure 55, a block diagram of an exemplary cloud computing environment 5500 is shown and described. The cloud computing environment 5500 may include one or more resource providers 5502a, 5502b, 5502c (collectively, 5502). Each resource provider 5502 may include computing resources. In some implementations, computing resources may include any hardware and / or software used to process data. For example, computing resources may include hardware and / or software capable of executing algorithms, computer programs, and / or computer applications. In some implementations, exemplary computing resources may include application servers and / or databases with storage and retrieval capabilities. Each resource provider 5502 may be connected to any other resource provider 5502 in the cloud computing environment 5500. In some implementations, the resource providers 5502 may be connected over a computer network 5508. Each resource provider 5502 may be connected to one or more computing device 5504a, 5504b, 5504c (collectively, 5504), over the computer network 5508.

[0350] The cloud computing environment 5500 may include a resource manager 5506. The resource manager 5506 may be connected to the resource providers 5502 and the computing devices 5504 over the computer network 5508. In some implementations, the resource manager 5506 may facilitate the provision of computing resources by one or more resource providers 5502 to one or more computing devices 5504. The resource manager 5506 may receive a request for a computing resource from a particular computing device 5504. The resource manager 5506 may identify one or more resource providers 5502 capable of providing the computing resource requested by the computing device 5504. The resource manager 5506 may select a resource provider 5502 to provide the computing resource. The resource manager 5506 may facilitate aconnection between the resource provider 5502 and a particular computing device 5504. In some implementations, the resource manager 5506 may establish a connection between a particular resource provider 5502 and a particular computing device 5504. In some implementations, the resource manager 5506 may redirect a particular computing device 5504 to a particular resource provider 5502 with the requested computing resource.

[0351] FIG.56 shows an example of a computing device 5600 and a mobile computing device 5650 that can be used to implement the techniques described in this disclosure. The computing device 5600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 5650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart- phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0352] The computing device 5600 includes a processor 5602, a memory 5604, a storage device 5606, a high-speed interface 5608 connecting to the memory 5604 and multiple high- speed expansion ports 5610, and a low-speed interface 5612 connecting to a low-speed expansion port 5614 and the storage device 5606. Each of the processor 5602, the memory 5604, the storage device 5606, the high-speed interface 5608, the high-speed expansion ports 5610, and the low-speed interface 5612, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 5602 can process instructions for execution within the computing device 5600, including instructions stored in the memory 5604 or on the storage device 5606 to display graphical information for a GUI on an external input / output device, such as a display 5616 coupled to the high-speed interface 5608. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). Thus, as the term is used herein, where a plurality of functions are described as being performed by “a processor”, this encompasses embodiments wherein the plurality of functions are performed byany number of processors (one or more) of any number of computing devices (one or more). Furthermore, where a function is described as being performed by “a processor”, this encompasses embodiments wherein the function is performed by any number of processors (one or more) of any number of computing devices (one or more) (e.g., in a distributed computing system).

[0353] The memory 5604 stores information within the computing device 5600. In some implementations, the memory 5604 is a volatile memory unit or units. In some implementations, the memory 5604 is a non-volatile memory unit or units. The memory 5604 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0354] The storage device 5606 is capable of providing mass storage for the computing device 5600. In some implementations, the storage device 5606 may be or contain a computer- readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 5602), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine- readable mediums (for example, the memory 5604, the storage device 5606, or memory on the processor 5602).

[0355] The high-speed interface 5608 manages bandwidth-intensive operations for the computing device 5600, while the low-speed interface 5612 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high- speed interface 5608 is coupled to the memory 5604, the display 5616 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 5610, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 5612 is coupled to the storage device 5606 and the low-speed expansion port 5614. The low-speed expansion port 5614, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0356] The computing device 5600 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 5620, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 5622. It may also be implemented as part of a rack server system 5624. Alternatively, components from the computing device 5600 may be combined with other components in a mobile device (not shown), such as a mobile computing device 5650. Each of such devices may contain one or more of the computing device 5600 and the mobile computing device 5650, and an entire system may be made up of multiple computing devices communicating with each other.

[0357] The mobile computing device 5650 includes a processor 5652, a memory 5664, an input / output device such as a display 5654, a communication interface 5666, and a transceiver 5668, among other components. The mobile computing device 5650 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 5652, the memory 5664, the display 5654, the communication interface 5666, and the transceiver 5668, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0358] The processor 5652 can execute instructions within the mobile computing device 5650, including instructions stored in the memory 5664. The processor 5652 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 5652 may provide, for example, for coordination of the other components of the mobile computing device 5650, such as control of user interfaces, applications run by the mobile computing device 5650, and wireless communication by the mobile computing device 5650.

[0359] The processor 5652 may communicate with a user through a control interface 5658 and a display interface 5656 coupled to the display 5654. The display 5654 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 5656 may comprise appropriate circuitry for driving the display 5654 to present graphical and other information to a user. The control interface 5658 may receive commands from a user and convert them for submission to the processor 5652. In addition, an external interface 5662 mayprovide communication with the processor 5652, so as to enable near area communication of the mobile computing device 5650 with other devices. The external interface 5662 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0360] The memory 5664 stores information within the mobile computing device 5650. The memory 5664 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 5674 may also be provided and connected to the mobile computing device 5650 through an expansion interface 5672, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 5674 may provide extra storage space for the mobile computing device 5650, or may also store applications or other information for the mobile computing device 5650. Specifically, the expansion memory 5674 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 5674 may be provide as a security module for the mobile computing device 5650, and may be programmed with instructions that permit secure use of the mobile computing device 5650. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0361] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 5652), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 5664, the expansion memory 5674, or memory on the processor 5652). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 5668 or the external interface 5662.

[0362] The mobile computing device 5650 may communicate wirelessly through the communication interface 5666, which may include digital signal processing circuitry where necessary. The communication interface 5666 may provide for communications under variousmodes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver 5668 using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 5670 may provide additional navigation- and location-related wireless data to the mobile computing device 5650, which may be used as appropriate by applications running on the mobile computing device 5650.

[0363] The mobile computing device 5650 may also communicate audibly using an audio codec 5660, which may receive spoken information from a user and convert it to usable digital information. The audio codec 5660 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 5650. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on the mobile computing device 5650.

[0364] The mobile computing device 5650 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 5680. It may also be implemented as part of a smart-phone 5682, personal digital assistant, or other similar mobile device.

[0365] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0366] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine- readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0367] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0368] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0369] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. Therelationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0370] In some implementations, certain modules described herein can be separated, combined or incorporated into single or combined modules. Any modules depicted in the figures are not intended to limit the systems described herein to the software architectures shown therein. EQUIVALENTS

[0371] Elements of different implementations described herein may be combined to form other implementations not specifically set forth above. Elements may be left out of the processes, computer programs, databases, etc. described herein without adversely affecting their operation. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Various separate elements may be combined into one or more individual elements to perform the functions described herein.

[0372] Throughout the description, where apparatus and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are apparatus, and systems of the present invention that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present invention that consist essentially of, or consist of, the recited processing steps.

[0373] It should be understood that the order of steps or order for performing certain action is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.

[0374] While the invention has been particularly shown and described with reference to specific preferred embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.EMBODIMENTS 1. A method of fiber (e.g., plant fiber, wood fiber) characterization, the method comprising: accessing (e.g., receiving), by a processor of a computing device, an image (e.g., a microscopy image) of a fiber sample comprising one or more fibers; (e.g., automatically) segmenting, by the processor, using a segmentation module, the image to isolate at least one fiber segment of the one or more fibers of the fiber sample image; and (e.g., automatically) characterizing, by the processor, using a classification module, the at least one fiber segment as belonging to at least one fiber class. 2. The method of embodiment 1, wherein the one or more fibers comprise one or more stained fibers (e.g., an iodine-stained fiber). 3. The method of embodiment 1, wherein the one or more fibers are not stained (e.g., using iodine). 4. The method of any one of the previous embodiments, wherein the image is a fiber analyzer (e.g., a MorFi) image. 5. The method of any one of embodiments 1-3, wherein the image is a microscopy image of the fiber sample. 6. The method of embodiment 5, wherein the microscopy image is a high magnification microscopy image (e.g., a microscopy image obtained at 4X, 20X, 40X, 60X, 100X, or higher magnification). 7. The method of any one of the preceding embodiments, wherein the one or more fibers are not manually annotated (e.g., manually characterized by a user). 8. The method of any one of the previous embodiments, wherein the at least one fiber class corresponds to a classification of plant fibers (e.g., tree fibers). 9. The method of embodiment 8, wherein the at least one fiber class is a hardwood fiber or a softwood fiber.10. The method of any one of the previous embodiments, wherein the method comprises obtaining the image of the fiber sample using a microscope or a fiber analyzer. 11. The method of any one of the previous embodiments, wherein the method further comprises characterizing the fiber sample. 12. The method of embodiment 11, wherein the method comprises characterizing an amount, a proportion, and / or a percentage of mass of the at least one fiber class in the sample. 13. The method of any one of the previous embodiments, wherein the method comprises characterizing an amount (e.g., proportion, percentage of mass) of old corrugated cardboard (OCC) and / or unbleached kraft pulp (UKP) in the fiber sample. 14. The method of any one of the previous embodiments, wherein the method comprises classifying the at least one fiber segment as being an anomalous fiber. 15. The method of any one of the previous embodiments, wherein the image segmentation module comprises one or more machine learning algorithms. 16. The method of embodiment 15, comprising training the image segmentation module on a plurality of images of a training dataset (e.g., microscopy images, fiber analyzer images, grayscale images, color images, e.g., RGB color images). 17. The method of embodiment 16, wherein the training dataset comprises a plurality of images of known, substantially pure fibers. 18. The method of embodiments 16 or 17, wherein the method comprises color transforming the plurality of images of the training data set (e.g., color transforming the plurality of images of the training data set from a first color space to a second color space). 19. The method of embodiment 18, wherein color transforming the plurality of images of the training data set comprises transforming the training data set images to LAB color space (CIELAB color space, L*a*b* color space) (e.g., from RGB (red-green-blue) color space to LAB color space). 20. The method of embodiments 16-19, wherein the method comprises mode centering the training dataset images (e.g., subtracting the image mode from each color channel)(e.g., mode centering a color transformed images).21. The method of any one of embodiments 18 to 20, wherein the method comprises performing a principal component analysis (PCA) transformation on the color transformed images. 22. The method of embodiment 21, comprising training at least one of the one or more machine learning algorithms using the PCA transformed images. 23. The method of embodiment 22, wherein the method comprises using the at least one machine learning algorithm trained using the PCA transformed images to eliminate correlations between color channels in the plurality of images of the training data set. 24. The method of embodiment 22 or 23, wherein the method comprises using the at least one machine learning algorithm (e.g., different from the PCA trained machine learning algorithm of embodiment 23) (e.g., a GMM, a BGMM) to segment the at least one fiber segment of the one or more fibers of the fiber sample image. 25. The method of embodiment 24, wherein the at least one machine learning algorithm is a Bayesian Gaussian mixture model (BGMM). 26. The method of any one of embodiments 16 to 25, wherein the method comprises removing fiber-fiber intersections and / or background in each of the plurality of images of the training dataset (e.g., through use of a segmentation mask). 27. The method of any one of the previous embodiments, wherein segmenting the fiber sample images comprises generating a mask (e.g., a binary mask) to isolate the at least one fiber segment of the one or more fibers of the fiber sample image. 28. The method of any one of the previous embodiments, wherein the method comprises performing one or more of a sequence of color space transformations, principal component analysis, mixture clustering (e.g., Bayesian Gaussian mixture clustering), and a morphological operation on the fiber sample image (e.g., to generate a mask). 29. The method of any one of the previous embodiments, wherein segmenting the image comprises transforming (e.g., color transforming) (e.g., a sequence of color transformations) the fiber sample image (e.g., transforming the image from a first color space to a second color space).30. The method of embodiment 29, wherein the method comprises color transforming the fiber sample image to a LAB color space (CIELAB color space, L*a*b* color space) (e.g., from an RGB (red-green-blue) fiber sample image to a LAB fiber sample image) (e.g., from a grayscale fiber sample image to a LAB image). 31. The method of any one of the previous embodiments, wherein the method comprises mode centering the fiber sample image (e.g., mode centering a color transformed fiber sample image). 32. The method of any one of embodiments 29 to 31, wherein transforming the fiber sample image comprises a grayscale transformation. 33. The method of any one of embodiments 29 to 32, wherein transforming the fiber sample comprises binarization of the fiber sample image (e.g., after transforming the fiber sample image to a grayscale image). 34. The method of any one of embodiments 29 to 33, wherein transforming the fiber sample comprises skeletonization of the fiber sample image (e.g., after transforming the fiber sample image to a binary image). 35. The method of any one of the preceding embodiments, wherein the method comprises removing (e.g., by the processor, e.g., using a background removal module, e.g., a machine learning module) one or more small objects from the fiber sample image. 36. The method of embodiment 35, wherein at least one of the one or more small objects are fines from the fiber sample image. 37. The method of any one of the preceding embodiments, wherein the method comprises dividing the fiber sample images (e.g., by the processor) into a plurality of tiles, wherein at least a subset of the tiles each comprise at least a portion of one of the one or more fibers of the fiber sample image. 38. The method of any one of the preceding embodiments, wherein the classification module comprises one or more machine learning modules.39. The method of embodiment 38, wherein at least one of the one or more machine learning modules comprises a neural network with one or more dense layers (e.g., two dense layers, three dense layers, or more). 40. The method of embodiment 39, wherein at least one (e.g., at least two, at least three, or all) of the one or more dense layers comprises at least 5, at least 10, at least 15, at least 20 or more neurons. 41. The method of any one of embodiments 38 to 40, wherein at least one of the one or more machine learning modules comprises a convolutional neural network (CNN). 42. The method of any one of the previous embodiments, wherein the method further comprises characterizing the at least one fiber segment based on fiber width (e.g., mean width), fiber color, fiber length, fiber texture and / or fiber shape. 43. The method of any one of the previous embodiments, wherein the at least one fiber segment comprises a plurality of fiber segments. 44. The method of any one of the previous embodiments, wherein the one or more fibers comprises a plurality of fibers. 45. The method of any one of the previous embodiments, wherein the at least one isolated fiber segment comprises a plurality of isolated fiber segments. 46. The method of any one of the previous embodiments, wherein the at least one characterized fiber segment comprises a plurality of isolated fiber segments 47. A method of characterizing a fiber sample (e.g., plant fiber, wood fiber), the method comprising: accessing (e.g., receiving), by a processor of a computing device, an image (e.g., a microscopy image) of a fiber sample comprising a plurality of fibers; segmenting, by the processor, (e.g., using a segmentation module) the image to isolate a plurality of fiber segments of the plurality of fibers; classifying, by the processor, (e.g., using a classification module) at least a portion of the fiber segments as belonging to at least one fiber class (e.g., a fiber type); anddetermining, by the processor, a composition of the fiber sample based on, at least, the classifications of the portion of the segmented fibers. 48. A system for characterizing a fiber sample, the system comprising: an imaging device (e.g., a microscope, a fiber analyzer, a sensor); a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of embodiments 1 to 47. 49. A system for characterizing a fiber sample, the system comprising: a device configured to access (e.g., receive) one or more fiber sample images; a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of embodiments 1 to 47. 50. A method (e.g., of characterizing fiber samples in a method of manufacturing fiber-based products), the method comprising: performing one or more processing steps to generate (e.g., manufacture) a fiber-based product from a fiber-based input, obtaining a sample of the fiber-based input after undergoing at least one of the one or more processing steps; characterizing (e.g., using a method of embodiments 1 to 47) the processed fiber-based input using at least one machine learning module (e.g., a characterization module, a segmentation module)(e.g., one or more machine learning modules) to obtain one or more fiber properties corresponding to the characterized, processed fiber-based input; and altering the at least one of the one or more processing steps based on, at least, the one or more fiber properties. 51. The method of embodiment 50, wherein the method is a method of papermaking. 52. The method of embodiment 50 or 51, wherein the fiber-based product is a paper product (e.g., toilet paper, paper, cardboard, paper towels, newsprint, etc.).53. The method of any one of embodiments 50-52, wherein the fiber-based input comprises recycled paper products. 54. The method of any one of embodiments 50-53, wherein the at least one processing step comprises a chemical or a mechanical process. 55. The method of any one of embodiments 50-54, wherein the at least one processing step comprises deinking, refining, pulping (repulping), blending, hornification, bleaching, enzymatic treatments, mercerization, regeneration, or handsheeting.

Claims

CLAIMS What is claimed is:

1. A method of fiber characterization, the method comprising: accessing, by a processor of a computing device, an image of a fiber sample comprising one or more fibers; segmenting, by the processor, using a segmentation module, the image to isolate at least one fiber segment of the one or more fibers of the fiber sample image; and characterizing, by the processor, using a classification module, the at least one fiber segment as belonging to at least one fiber class.

2. The method of claim 1, wherein the one or more fibers comprise one or more stained fibers.

3. The method of claim 1, wherein the one or more fibers are not stained.

4. The method of any one of the previous claims, wherein the image is a fiber analyzer image.

5. The method of claim 1, wherein the image is a microscopy image of the fiber sample.

6. The method of claim 5, wherein the microscopy image is a high magnification microscopy image.

7. The method of claim 1, wherein the one or more fibers are not manually annotated.

8. The method of claim 1, wherein the at least one fiber class corresponds to a classification of plant fibers.

9. The method of claim 8, wherein the at least one fiber class is a hardwood fiber or a softwood fiber.

10. The method of claim 1, wherein the method comprises obtaining the image of the fiber sample using a microscope or a fiber analyzer.

11. The method of claim 1, wherein the method further comprises characterizing the fiber sample.

12. The method of claim 11, wherein the method comprises characterizing an amount, a proportion, and / or a percentage of mass of the at least one fiber class in the sample.

13. The method claim 1, wherein the method comprises characterizing an amount of old corrugated cardboard (OCC) and / or unbleached kraft pulp (UKP) in the fiber sample.

14. The method of claim 1, wherein the method comprises classifying the at least one fiber segment as being an anomalous fiber.

15. The method of claim 1, wherein the image segmentation module comprises one or more machine learning algorithms.

16. The method of claim 15, comprising training the image segmentation module on a plurality of images of a training dataset.

17. The method of claim 16, wherein the training dataset comprises a plurality of images of known, substantially pure fibers.

18. The method of claims 16, wherein the method comprises color transforming the plurality of images of the training data set.

19. The method of claim 18, wherein color transforming the plurality of images of the training data set comprises transforming the training data set images to LAB color space (CIELAB color space, L*a*b* color space).

20. The method of claim 19, wherein the method comprises mode centering the training dataset images.

21. The method of claim 18, wherein the method comprises performing a principal component analysis (PCA) transformation on the color transformed images.

22. The method of claim 21, comprising training at least one of the one or more machine learning algorithms using the PCA transformed images.

23. The method of claim 22, wherein the method comprises using the at least one machine learning algorithm trained using the PCA transformed images to eliminate correlations between color channels in the plurality of images of the training data set.

24. The method of claim 1, wherein the method comprises using the at least one machine learning algorithm to segment the at least one fiber segment of the one or more fibers of the fiber sample image.

25. The method of claim 24, wherein the at least one machine learning algorithm is a Bayesian Gaussian mixture model (BGMM).

26. The method of claim 16, wherein the method comprises removing fiber-fiber intersections and / or background in each of the plurality of images of the training dataset.

27. The method of claim 1, wherein segmenting the fiber sample images comprises generating a mask to isolate the at least one fiber segment of the one or more fibers of the fiber sample image.

28. The method of claim 1, wherein the method comprises performing one or more of a sequence of color space transformations, principal component analysis, mixture clustering, and a morphological operation on the fiber sample image.

29. The method of claim 1, wherein segmenting the image comprises transforming the fiber sample image.

30. The method of claim 29, wherein the method comprises transforming the fiber sample image to a LAB color space (CIELAB color space, L*a*b* color space).

31. The method of claim 30, wherein the method comprises mode centering the fiber sample image.

32. The method of claim 29, wherein transforming the fiber sample image comprises a grayscale transformation.

33. The method of claim 29, wherein transforming the fiber sample comprises binarization of the fiber sample.

34. The method of claim 29, wherein transforming the fiber sample comprises skeletonization of the fiber sample image.

35. The method of claim 1, wherein the method comprises removing one or more small objects from the fiber sample image.

36. The method of claim 35, wherein at least one of the one or more small objects are fines from the fiber sample image.

37. The method of claim 1, wherein the method comprises dividing the fiber sample images into a plurality of tiles, wherein at least a subset of the tiles each comprise at least a portion of one of the one or more fibers of the fiber sample image.

38. The method of claim 1, wherein the classification module comprises one or more machine learning modules.

39. The method of claim 38, wherein at least one of the one or more machine learning modules comprises a neural network with one or more dense layers.

40. The method of claim 39, wherein at least one of the one or more dense layers comprises at least 5, at least 10, at least 15, at least 20 or more neurons.

41. The method of claim 38, wherein at least one of the one or more machine learning modules comprises a convolutional neural network (CNN).

42. The method of claim 1, wherein the method further comprises characterizing the at least one fiber segment based on fiber width, fiber color, fiber length, fiber texture and / or fiber shape.

43. The method of claim 1, wherein the at least one fiber segment comprises a plurality of fiber segments.

44. The method of claim 1, wherein the one or more fibers comprises a plurality of fibers.

45. The method of claim 1, wherein the at least one isolated fiber segment comprises a plurality of isolated fiber segments.

46. The method of claim 1, wherein the at least one characterized fiber segment comprises a plurality of isolated fiber segments 47. A method of characterizing a fiber sample, the method comprising: accessing, by a processor of a computing device, an image of a fiber sample comprising a plurality of fibers; segmenting, by the processor, the image to isolate a plurality of fiber segments of the plurality of fibers; classifying, by the processor, at least a portion of the fiber segments as belonging to at least one fiber class; anddetermining, by the processor, a composition of the fiber sample based on, at least, the classifications of the portion of the segmented fibers.

48. A system for characterizing a fiber sample, the system comprising: an imaging device; a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 47.

49. A system for characterizing a fiber sample, the system comprising: a device configured to access one or more fiber sample images; a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 47.

50. A method, the method comprising: performing one or more processing steps to generate a fiber-based product from a fiber- based input, obtaining a sample of the fiber-based input after undergoing at least one of the one or more processing steps; characterizing the processed fiber-based input using at least one machine learning module to obtain one or more fiber properties corresponding to the characterized, processed fiber-based input; and altering the at least one of the one or more processing steps based on, at least, the one or more fiber properties.

51. The method of claim 50, wherein the method is a method of papermaking.

52. The method of claim 50, wherein the fiber-based product is a paper product.

53. The method of claim 50, wherein the fiber-based input comprises recycled paper products.

54. The method of claim 50, wherein the at least one processing step comprises a chemical or a mechanical process.

55. The method of claim 50, wherein the at least one processing step comprises deinking, refining, pulping (repulping), blending, hornification, bleaching, enzymatic treatments, mercerization, regeneration, or handsheeting.

Images