SYSTEM AND METHOD FOR IDENTIFYING SUB-COMPONENT FAILURE IN SEQUENCING EQUIPMENT - Patent application

Abstract

The disclosed technology relates to a system and method for diagnosing and identifying causes of a system malfunction. The method includes applying a preprocessor to time series data. The preprocessor detects time series discontinuities, drift, lack of expected correlation, and trends. The method includes feeding at least a portion of the output of the preprocessor to a trained tree-based classifier for at least one image channel in a sequencing run, and receiving a classification of the particular sequencing run as an anomaly. The anomaly classification may indicate a system malfunction. The method includes feeding at least a portion of the output of the preprocessor for the anomalous sequencing run to an expert rule system. The expert rule system may identify a root cause of the system malfunction to a particular subcomponent that needs to be adjusted or replaced. The method may generate a notification of the particular subcomponent causing the system malfunction.

Description

[Technical field]

[0001] (Priority Application) This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 169,676 (Attorney Docket No. ILLM 1039-2 / IP-2119-PRV), entitled “SYSTEMS AND METHODS FOR IDENTIFYING SUBCOMPONENT FAILURE IN SEQUENCING INSTRUMENTS,” filed April 1, 2021, which is incorporated by reference for all purposes.

[0002] FIELD OF THEINVENTION The disclosed technology relates to the application of machine learning based classifiers and expert systems to identify sub-components causing failures of subsystems in sequencing systems, which can be extended to identify sub-components causing failures in other types of systems. [Background technology]

[0003] The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, it should not be assumed that the problems mentioned in this section, or problems associated with the subject matter provided as background art, have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which as such may also correspond to embodiments of the claimed technology.

[0004] The disclosed technology relates to a sequencing system, including a system that applies a sequencing-by-synthesis technique to sequence nucleotides. A sequencing run to identify nucleotides in a molecule is a lengthy process that takes days to complete. All subsystems of a sequencing machine need to operate without errors so that the resulting base calls are useful for downstream analysis. A subsystem may have multiple subcomponents. Outputs from a sequencing instrument that describe the quality of a sequencing run do not help identify the subcomponent causing the system malfunction. It has been observed that a significant percentage of repair calls result in the replacement of an incorrect subcomponent, thus causing significant losses to operators of such instruments. Summary of the Invention [Means for solving the problem]

[0005] Thus, an opportunity exists to develop systems and methods for identifying subcomponents that are causing a malfunction in sequencing equipment.

[0006] In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosed technology. In the following description, various embodiments of the disclosed technology are described with reference to the following drawings, in which: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 shows an architectural level schematic of a system capable of classifying time series data from sequencing equipment and identifying the root cause of a system malfunction. [Figure 2A] 1 presents a schematic diagram of the subcomponents of the optical subsystem within a sequencing instrument. [Figure 2B] 1 presents a photographic image of the optical subsystem within the sequencing instrument. [Figure 3A] The operation of the z-stage subcomponent for moving the objective lens to adjust the focus for each tile on the surface of the flow cell is presented. [Figure 3B] The operation of the compensator subcomponent to shift the focus from one surface of the flow cell to another is presented. [Figure 4A] Example time series from a good sequencing run and an example time series with discontinuities are presented. [Figure 4B] Example time series of the top and bottom surfaces of a flow cell are presented, showing discontinuities, lack of expected correlation, and trends. [Figure 4C] Example time series of the top and bottom surfaces of a flow cell are presented, showing discontinuities, lack of expected correlation, and trends. [Figure 4D] We present an example time series showing pairs of signal channels drifting apart, indicating a fault in the M3 subcomponent. [Figure 4E] An exemplary time series showing a signal channel with in-process vibration signatures due to a z-stage subcomponent malfunction is presented. [Figure 5A] We present training of a fault detection classifier using labeled training data that includes the primary metrics and the computed features. [Figure 5B] Exemplary data values ​​of the primary metrics and computed features for training the fault detection classifier are presented. [Figure 5C] Exemplary data values ​​of the primary metrics and computed features for training the fault detection classifier are presented. [Figure 5D] Exemplary data values ​​of the primary metrics and computed features for training the fault detection classifier are presented. [Figure 5E] We present a two-step process for generating alerts, which involves classifying production time-series data using a fault detection classifier and identifying malfunctioning subcomponents using expert fault identification rules. [Figure 6A]We present the processing of multiple signal channels classification for one read (Figure 6C), then combined classification across two reads in a sequencing run (Figure 6B), and finally combined classification across multiple sequencing runs (Figure 6A) to identify the subcomponent causing the disorder. [Figure 6B] We present the processing of multiple signal channels classification for one read (Figure 6C), then combined classification across two reads in a sequencing run (Figure 6B), and finally combined classification across multiple sequencing runs (Figure 6A) to identify the subcomponent causing the disorder. [Figure 6C] We present the processing of multiple signal channels classification for one read (Figure 6C), then combined classification across two reads in a sequencing run (Figure 6B), and finally combined classification across multiple sequencing runs (Figure 6A) to identify the subcomponent causing the disorder. [Figure 7A] Provides examples of alerts generated by the system, including malfunctioning subcomponent, severity, alert summary, and alert details. [Figure 7B] Presents different types of alerts corresponding to different severity levels. [Figure 8] FIG. 1 is a simplified block diagram of a computer system that can be used to implement the disclosed techniques. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following discussion is presented to enable those skilled in the art to make and use the disclosed technology and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0009] Introduction Sequencing-by-synthesis (SBS) is one of several common techniques for sequencing nucleotides in DNA or RNA molecules. Machines that perform sequencing are complex systems with sophisticated subsystems with interchangeable subcomponents. These sequencing machines may include subsystems including flow cells, fluidics and reagents, optics and image capture, and processing modules. These sequencing systems apply the SBS technique to base calling cycles in a sequencing run. In an SBS process cycle, complementary nucleotides are added one at a time to a nucleotide sequence fragment (also called a molecule or an insert) from the sequence being processed. Sequencing of the nucleotides in a molecule proceeds in hundreds of cycles. A cycle includes chemical, image capture, and image processing actions. The subsystems and their subcomponents operate during each cycle to identify the tagged complementary nucleotides attached to the molecule.

[0010] The optical subsystem includes, for example, M3, z-stage, and compensator subcomponents used to image the flow cell. The images record the glow from clusters of A, C, G, and T bases with fluorescently tagged complementary nucleotides with fluorescent dyes temporarily attached for the synthesis cycle. The M3 subcomponent includes mirrors used to focus the monochromatic excitation beam or light from a laser or LED source to excite the fluorescent tags. The z-stage subcomponent moves the objective lens up and down along the z-axis for overall focus of the camera that captures the image. Clusters of molecules can grow on both the top and bottom surfaces of the flow cell. The compensator subcomponent provides fine focus between the top and bottom surfaces of the flow cell, which are separated by about a tenth of a millimeter. This accounts for changes in focus from one surface of the flow cell to the other.

[0011] Failure of any of these subcomponents may cause poor focus quality, degradation of image intensity. The information collected to perform sequencing is not designed to identify subcomponent failures. The relationship of some of the information on sequencing quality to subcomponent failure and fault identification is unclear. Technicians who use the overall sequencing confidence score to identify that a machine is malfunctioning often choose the wrong subcomponent for replacement and end up replacing multiple subcomponents when only one is faulty. It has been observed that a significant percentage of repair calls result in the replacement of an incorrect subcomponent, thus causing significant losses to operators of such equipment. It is therefore desirable to develop systems and techniques that reliably identify the subcomponent causing the malfunction of the optical subsystem.

[0012] Using carefully refined ground truth fault identification data that is refined to be better than the performance of a typical technician in the field, an expert system can be trained to recognize signal patterns that characterize faults in a particular subcomponent. In stages, the disclosed technology includes a fault detection classifier and expert fault identification rules. A classifier architecture (such as random forest) that can be trained on a small data set is selected to recognize repairable faults, which are further processed against a hand-crafted set of expert rules. The expert rules suggest which subcomponents should be replaced and present reasoning that is embodied in the rules that are applied. With larger training data sets, the two stages can be combined into a single classifier. A technician guided by the expert system fault identification is more likely to replace the correct subcomponent.

[0013] The raw ground truth data that dictates which subcomponents technicians have decided to replace in the past cannot be trusted due to fault identification error rates. Furthermore, because the sequencing equipment can be trusted, the sample size of equipment repairs is small compared to the number of sequencing runs available. Cleaning the raw ground truth data by correctly labeling subcomponent failures in the collected time series data is more productive than discarding suspect data. This can be appreciated with a sidebar on time series data.

[0014] In a sequencing run, multiple signal channels generate time series data. This is true for SBS and other sequencing techniques. Sequencing a segment of a molecule with 350 nucleotides by SBS can involve more than 300 processing cycles in a sequencing run. A sequencing run can be split into two reads (read 1 and read 2) that start at the 3' and 5' ends of the same sequence fragment and either overlap or leave a gap in the middle of the fragment. Throughout the read, each cycle involves imaging the top and bottom of the flow cell and collecting data. Sequencing instruments can provide data such as full width at half maximum (or FWHM) values ​​that indicate focus quality and image intensity values ​​per sequencing cycle. Some sequencing instruments use nucleotides labeled with four different colored fluorescent dyes corresponding to A, C, G, and T nucleotides. In such sequencing instruments, four signal (or image) channels can be used to distinguish between the four types of nucleotides. Thus, there can be up to eight time series per flow cell surface during a sequencing read (1 read x 2 signal types x 4 signal channels). For a flow cell with two surfaces, there can be up to 16 time series. Such instruments can use one or two laser or LED light sources to excite the fluorescent tags corresponding to A, C, G, T nucleotides. The time series data for each signal channel in a successful sequencing run follows a characteristic signal pattern. Deviation of a given signal channel from the respective signal pattern can indicate a potential problem.

[0015] Some sequencing instruments can distinguish between four types of nucleotides using fewer than four signal channels. For example, some sequencing instruments can detect four types of nucleotides (A, C, G, T) using two signal channels. In such systems, two images are taken per cycle. A first nucleotide type is detected in a first channel, a second nucleotide type is detected in a second channel, a third nucleotide type is detected in both the first and second channels, and a fourth nucleotide type lacking a dye tag label is not detected or is only minimally detected in any channel. The disclosed technology can process a time series of signal channels of four types of nucleotides to detect subcomponent malfunctions.

[0016] Yet another type of sequencing instrument can detect four types of nucleotides using one signal channel. Such a sequencing system can include a patterned flow cell with nanowells fabricated on a CMOS chip. The sequencing instrument can use one fluorescent dye, two chemical steps, and two imaging steps per cycle. In one example, the four types of nucleotides are detected as follows: The first nucleotide (e.g., an "A" base) has a removable label and is labeled only in the first image. The second nucleotide (e.g., a "C" base) is a linker group that can be attached to a label and is labeled only in the second image. The third nucleotide (e.g., a "T" base) can have a permanent fluorescent label and is therefore labeled in both images. The fourth nucleotide (e.g., a "G" base) is permanently dark. The four nucleotides are detected by analysis of the different emission patterns at each base across the two images. The disclosed technology can process the time series of the signal channels of the four types of nucleotides to detect subcomponent malfunctions.

[0017] Time series data from at least one signal channel is preprocessed to prepare training data for a classifier. It is difficult to identify features in the time series data that are associated with a particular subcomponent failure. The disclosed technique was developed using a graphical representation of the time series data to identify features that can identify the subcomponent causing the failure. When feature engineering was applied during signal preprocessing, patterns observed from the faulty subcomponents produced discontinuities, out-of-range signals, poor correlation between two surfaces, and signal trends. When these patterns are detected in the time series data, an abnormal sequencing run is indicated or at least suggested. Each pattern indicates an anomaly caused by a different subcomponent. Some of the training examples are relabeled, and where appropriate, the labels of subcomponents that need to be replaced are revised. The signal patterns are plotted and visually inspected for signal characteristics. Programmed detectors are developed to compute features used to identify the subcomponent causing the malfunction. Examples of programmatic signal pattern detectors include a discontinuity detector, a drift detector, a surface-to-surface correlation detector, and a trend detector. The program detector is applied to time series data from multiple signal channels to generate computed features that are used by the fault detection classifiers and then by the expert fault identification rules. Details of the signal detector are presented below.

[0018] The discontinuity detector identifies large jumps between successive cycles, for example in the focus score. A large jump may indicate a large variation in the time series data between successive cycles. For example, the difference between the focus score values ​​of successive cycles may indicate a large jump when the calculated difference exceeds a predetermined threshold. The surface of the flow cell may not be optically flat across the imaging tile. During the imaging process, the z-stage subcomponent can adjust the height of the objective lens for each tile to accommodate the uneven surface of the flow cell. A faulty z-stage subcomponent may result in inaccurate focus of the image tile resulting in discontinuities (or jumps) in the focus score time series for multiple image channels.

[0019] In each cycle, the z-stage is used to move the objective lens and adjust the optical focus as the LED color changes between green and red. Any malfunction in the z-stage between LED activations will degrade the focus, which is reflected in a sudden increase in the focus score in the FWHM statistics of the tile being imaged. As the beam from each LED activation is split along two paths and detected by at least one camera, the effect of the z-stage malfunction can be observed in the FWHM values ​​from images taken by both cameras. Expert rules identify the z-stage as the likely cause of the malfunction when such a discontinuity is detected.

[0020] The drift detector compares the drift over time of the signals captured by the cameras of the different optical paths to identify malfunctions of components present in one optical path but not the other. The M3 mirror is present on one optical leg in some systems. It has the role of overall focus, and of single leg focus. When the single leg focus is malfunctioning, images from both the red and green actuation of one camera suffer from focus score or FWHM statistical degradation relative to the other camera. Expert rules identify the M3 mirror as the likely cause of malfunction when a differential drift between the signals from each camera is detected.

[0021] The inter-surface correlation detector detects whether a signal channel on the top surface of the flow cell follows a similar signal pattern as a corresponding signal channel on the bottom surface of the flow cell. When the optical subsystem component is operating normally, the corresponding signal channels on the top and bottom surfaces follow a similar signal pattern. A failure of the optical system subcomponent can cause a variation in the signal channel pattern between the top and bottom surfaces. The expert rules can check whether multiple signal channels from one surface (such as the top surface) are classified as anomalous and whether the two surfaces of the flow cell have low correlation. If these conditions are true, the expert identification rules can identify the compensator subcomponent as a likely cause of the system malfunction.

[0022] The trend detector can identify a degraded focus when the focus score values ​​in the last portion of the focus score time series are degraded relative to the focus score values ​​in the first portion of the focus score time series. The trends of the time series corresponding to the multiple signal channels can be given as inputs to the fault detection classifier along with other inputs for classifying the signal channels as normal or anomalous. The trended feature can be used as inputs to one or more expert rules. However, it is primarily used by the first stage random forest classifier to classify the signal channels as normal or anomalous.

[0023] The disclosed technology may also include an in-process vibration detector for detecting features in a focus score time series from at least one signal channel that are indicative of a z-stage sub-component failure. The focus score time series in such a z-stage malfunction may include a middle portion having a discontinuity or spike. The middle portion is located between a first good portion and a second good portion of the focus score time series. The two good portions may include focus score values ​​that are mostly within the normal focus score range. The detector may return a "true" result when it detects an in-process vibration having a higher average focus score (or FWHM) in the focus score time series. The expert fault identification rules engine may use the output from the in-process vibration detector to generate an alert for the z-stage sub-component malfunction.

[0024] environment We describe a system for diagnosing a system malfunction and identifying the cause of the system malfunction among multiple replaceable subcomponents. Sequencing by synthesis (SBS) is a technique for sequencing nucleotides in DNA or RNA molecules. SBS involves adding complementary nucleotides one at a time to nucleotide sequence fragments from the DNA or RNA molecule being sequenced. An optical platform using SBS can sequence billions of clusters of nucleotide sequence fragments, sometimes called molecules, on the surface of a flow cell arranged in multiple lanes with tiles in each lane. The molecular clusters represent clones of molecules. Cloning of molecules amplifies the signal generated during SBS. The system is described with reference to FIG. 1, which shows an architecture level schematic diagram of the system according to an embodiment. Because FIG. 1 is an architecture diagram, certain details have been intentionally omitted to improve clarity of description. The discussion of FIG. 1 is organized as follows: First, the elements of the diagram are described, followed by their interconnections. Then, the use of the elements in the system is described in more detail.

[0025] FIG. 1 includes a system 100. This paragraph names the labeled parts of the system 100. The figure shows a sequencing system 111, a primary metric database 114 for storing metrics collected from the sequencing equipment, a computed feature database 116, a time series pre-processor 130, a raw repair call database 149, a refined ground truth fault identification database 159, a fault detection classifier 183, an expert fault identification rules engine 186, an alert database 189, and a network 155. The time series pre-processor 130 further includes program detectors for computing features from time series data from the reads in the sequencing run. The system includes a discontinuity detector 141, a drift detector 151, a surface-to-surface correlation detector 161, and a trend detector 171. The computed features generated by the time series pre-processor are used by the fault detection classifier and then by the expert fault identification rules engine to identify subcomponents causing malfunctions of the sequencing equipment.

[0026] The disclosed technology is applied to various sequencing systems 111, also called sequencing instruments or sequencing platforms. Some examples of sequencing systems 111 include Illumina's MiSeqDx™, NovaSeq 6000™, HiSeqX™, HiSeq3000™, and HiSeq4000™. These sequencing systems are configured to apply sequencing-by-synthesis (SBS) techniques for base calling. A network 155 connects the sequencing system 111, the primary metric database 114, the computed feature database 116, the time series preprocessor 130, the raw repair call database 149, the refined ground truth fault identification database 159, the fault detection classifier 183, the expert fault identification rule engine 186, and the alert database 189 in communication with each other.

[0027] Before a sequencing cycle begins, a library of molecules to be sequenced is prepared on a slide or flow cell. The molecules are tiled in multiple lanes on one or two surfaces of the flow cell. A cycle includes chemical, image capture, and image processing actions. Subsystems include optical, mechanical, and chemical subsystems that operate in each cycle to identify complementary nucleotides attached to the molecules. A sequencing run can include hundreds of cycles, sometimes performed in two reads proceeding from the 3' and 5' ends of the same sequence fragment. A sequencing run can take multiple days to complete. Sometimes the results of an entire sequencing run are discarded because they do not meet minimum quality requirements for downstream analysis.

[0028] A sequencing instrument produces several outputs (also called primary metrics) after completing scanning and imaging of the flow cell surface. Examples of outputs from the instrument include full width at half maximum (FWHM) values, which indicate focus quality and image intensity values ​​per sequencing cycle. Other examples of primary metrics include the number of reads (e.g., 1 or 2), surface identifier (e.g., 1 or 2), sequencing run identifier, instrument serial number, and run status, which indicates whether the run is "completed" or "stopped" before completion. Primary metrics may also include a quality score for the sequencing run (e.g., Q30 score), percentage of aligned sequences, error rate, maximum number of cycles in a sequencing run, count of tiles on the flow cell surface, etc. Primary metrics provide useful information about the sequencing run, but are not intended to identify subcomponents within the subsystem that are causing malfunctions or failures.

[0029] In SBS, an LED light source or laser is used to illuminate dye-tagged complementary nucleotides attached to molecules in each cycle. A camera takes images of the tile containing the clusters of molecules. The images are then processed to identify the nucleotides (A, C, G, T) attached to the molecules in the clusters. Some sequencing systems use four channels to identify the four types of nucleotides (A, C, G, T) attached to the molecules per cycle. In such systems, four images are produced, each image containing a signal with a single distinct color per image. The four colors correspond to the four possible nucleotides present at a particular position. In another type of sequencing system, two channels are used to identify the four types of nucleotides (A, C, G, T). In such systems, two images are taken per cycle. A first nucleotide type is detected in a first channel, a second nucleotide type is detected in a second channel, a third nucleotide type is detected in both the first and second channels, and a fourth nucleotide type lacking a dye tag label is not detected or is only minimally detected in any channel.

[0030] The optical subsystem includes subcomponents that image the tiles on the surface of the flow cell. A failure in any of these subcomponents can cause poor focus quality, degradation of image intensity. The sequencing data provided by the instrument does not help identify the subcomponent causing the system malfunction. Thus, technicians often select and replace the wrong subcomponent, resulting in replacing multiple subcomponents when only one subcomponent is faulty. Thus, the raw repair call data stored in the raw repair call database 149 is unreliable to use as ground truth. The system cleans the raw ground truth data by correctly labeling the subcomponents causing the malfunction. The system stores the correctly labeled repair call data in the refined ground truth fault identification database 159. For the correctly labeled ground truth data, the system applies the time series preprocessor 130 to generate computed features and stores them in the computed features database along with a label indicating the success or failure of the sequencing run. The disclosed technique may include four types of time series pre-processors that produce respective computed features.

[0031] The system applies a trained fault detection classifier 183 and an expert fault isolation rule engine 186 in a two-step process to identify the subcomponent causing the system malfunction. The classifier (such as a random forest) can be trained using a small training set with clean ground truth labels that indicate repairable faults caused by the subcomponent. The trained fault detection classifier 183 can classify the signal channel as normal or abnormal. The classification and calculated features of the signal channel are further processed against hand-crafted expert rules by the expert fault isolation rule engine 186 to suggest which subcomponent should be replaced and present inferences embodied in the applied rules. The system can generate an alert indicating the subcomponent name and severity level of the alert, such as warning or information. Depending on the severity level of the alert, a repair ticket can be opened for subcomponent maintenance.

[0032] Completing the description of FIG. 1, all of the components of the system 100 described above are communicatively coupled to a network 155. The actual communication paths may be point-to-point over public and / or private networks. Communications may occur across a variety of networks, such as private networks, VPNs, MPLS circuits, or the Internet, and may use appropriate application programming interfaces (APIs) and data exchange formats, such as Representational State Transfer (REST), JavaScript Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java Message Service (JMS), and / or Java Platform Module System. All of the communications may be encrypted. Communications are typically over networks such as local area networks (LANs), wide area networks (WANs), Public Switched Telephone Networks (PSTNs), Session Initiation Protocol (SIP), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, the Internet, including mobile Internet over protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. The engine or system components of FIG. 1 are implemented by software running on various types of computing devices. Exemplary devices are workstations, servers, computing clusters, blade servers, and server farms.Additionally, various authorization and authentication technologies may be used to secure communications, such as username / password, Open Authorization (OAuth), Kerberos, Secured, digital certificates, etc.

[0033] We present an overview of the subcomponents within the optical subsystem of a sequencing instrument. We then provide details of a time series preprocessor that generates features computed from the time series data. We then present a two-stage architecture to classify the time series data of multiple signal channels and use the classification to identify the subcomponents causing the system malfunction.

[0034] Optical Subsystem Subcomponents FIG. 2A presents a diagram of the optical subsystem 205 of the sequencing instrument. We first describe the light path from the radiation source to the imaging device, and then present details of the three subcomponents of the optical subsystem. The optical subsystem comprises an LED assembly 210 with a radiation source including two light emitting diodes (LEDs) 215 and 220 that produce radiation at different wavelengths. For example, a green LED with a wavelength of 532 nanometers and a red LED with a wavelength of 660 nm can be used. The light path of the green LED is shown using dashed lines, and the light path of the red LED is shown using solid lines. The excitation light from the LEDs passes through a green LED collector lens (L6) and a red LED collector lens (L5), respectively. The LED fold mirror reflects the green excitation radiation to a combiner dichroic 225. The green excitation radiation passes through an M5 prism, which reflects the green excitation light through an objective lens 230. The red excitation radiation passes from the LED collector lens L5 to a combiner 225, after which the red excitation radiation follows the same path as the green excitation radiation.

[0035] The objective lens 230 is positioned to collect the emission radiation and direct it to an excitation / emission dichroic (M5 prism), which passes the emission radiation through an F3 beam splitter 235 to image sensors S1 and S2. Samples can be positioned on one or more surfaces of the flow cell. The surface of the flow cell is not optically flat. A z-stage subcomponent 250 is coupled to at least a portion of the objective lens and configured to translate the objective lens 230 up and down (along the z-axis) to adjust the position of the objective lens when an image of a tile on the surface of the flow cell is captured. An aperture (labeled "A1 Apert") can combine the green and red LED outputs to form a single illumination beam.

[0036] Samples can be positioned on two opposing inner surfaces of the flow cell (surface 1 and surface 2) as shown in Figure 2A. A compensator 240 can be included in the optical path or removed from the optical path to shift the focus of the objective lens from one surface to the other. For example, a compensator (such as a lens) positioned in the optical path can shift the focus of the objective to surface 2, and when removed from the optical path, the focus of the objective shifts to surface 1. A mechanical assembly can be coupled to the compensator.

[0037] An exemplary optical subsystem at 205 uses two LEDs of different wavelengths to illuminate the fluorescently tagged nucleotides. Two cameras S1 and S2 each capture images after illuminating the sample using one LED. For example, when a red excitation laser illuminates the sample, the images from cameras S1 and S2 can be used to detect C and A bases, respectively. When a green excitation laser illuminates the sample, the images from cameras S1 and S2 can be used to detect T and G bases, respectively. The F3 beam splitter splits the emission radiation into two optical legs, labeled leg 1 and leg 2. The emission radiation in each leg passes through focusing lenses L1 and L2, respectively, and subsequently through F1 and F2 emission filters.

[0038] In a four-dye system, four separate colors can be used to call DNA bases. In such a system, each of the four nucleotides can be labeled with a separate dye. In one cycle, the radiation signals from these dyes are imaged by using two excitation lasers, a red 660 nm frequency laser and a green 532 nm frequency laser. When the sample is illuminated by the red excitation laser, the emission radiation is split into two legs, with half of the emission light directed to each camera. Emission filters F1 and F2 are selected to detect emission wavelengths corresponding to C and A bases by cameras S1 and S2, respectively. When the green excitation laser illuminates the sample, a separate set of emission filters F1 and F2 can be selected to detect T and G bases. Each image corresponds to a specific combination of excitation lasers and filters. In one example, the emission filters include bandpass filters at 557±11 nm (to detect signal channel G), 615±40 nm (to detect signal channel T), 684±11 nm (to detect signal channel A), and 740±50 nm (to detect signal channel C). The disclosed technology can process signal channels from other systems, including one dye, two dye, or three dye systems.

[0039] Figure 2B presents a photographic image 270 of an example optical subsystem within a sequencing instrument. The locations of at least some of the subcomponents from the schematic diagram of Figure 2A are labeled on the photographic image of Figure 2B.

[0040] M3 Mirror Subcomponents Referring to the schematic diagram of FIG. 2A, one of the two optical legs may have an M3 focus mirror 260 positioned in the optical path. For example, in the optical subsystem 205, the M3 focus mirror is positioned in leg 1. The M3 mirror has the role of global focus and single leg focus. The M3 mirror can be actuated and rotated between two mechanical positions. In the imaging mode, the M3 mirror is positioned in its "imaging position" and acts as a 45 degree folding mirror in the optical path of the camera S1. In the focus mode, the M3 mirror is positioned in its "focus position" and induces a horizontal focus gradient across the image. During the focus mode, lasers of different wavelengths (red, green, and blue) are reflected off the M3 mirror and the z-stage moves up and down to obtain the best focus image. When the single leg focus is malfunctioning, the images from both the red and green actuation of one camera suffer from focus score or FWHM statistical degradation relative to the other camera. Expert rules are able to identify the M3 mirror as the likely cause of a malfunction when a differential drift between the signals from the respective cameras is detected.

[0041] Z Stage Subcomponent FIG. 3A illustrates the operation of the z-stage subcomponent 250. Schematic diagram 311 illustrates the movement of the objective lens along the z-axis. Samples are positioned on two inner surfaces of the flow cell, namely surface 1 313 and surface 2 315. Surface 1 is positioned toward the top end of the flow cell and surface 2 is positioned toward the bottom end of the flow cell. The distance between the two surfaces is small, for example, about 0.72 millimeters (mm). A z-stage (not shown) can be coupled to the objective lens to move the objective lens along the z-axis perpendicular to the surface of the flow cell. The movement of the z-stage can be created by energizing a piezoelectric crystal in a mechanical enclosure. The movement of the z-stage along the z-axis is about 300 micrometers. The top-most position is −300 micrometers from the bottom-most position (labeled 0). A cluster of samples inside the bottom surface of the flow cell is located 0.072 mm below the top surface.

[0042] The surface of the flow cell may not be optically flat. During the imaging process, an objective lens is positioned over each tile in the flow cell to capture an image of the sample. Fig. 351 shows two tiles (tile 1 and tile 2) such that tile 1 is closer to object 230 compared to tile 2. The z-stage subcomponent maintains focus between tiles by moving the objective lens 230 further or closer to the surface of the flow cell to accommodate surface bumps as shown in Fig. 355. A faulty z-stage subcomponent may result in a degradation of the focus score of the image of the tile.

[0043] Compensator Subcomponents FIG. 3B illustrates the operation of the compensator subcomponent 240. Without positioning the compensator in the optical path, the objective lens 230 can focus and detect an image from the bottom surface (or surface 2) of the flow cell, as shown in FIG. 370. When the compensator 240 is positioned in the optical path, the focus of the objective lens 230 shifts to the top surface (or surface 1) of the flow cell, as shown in FIG. 380. The objective lens can then detect an image from the top surface. The compensator subcomponent may include a mechanism to move the compensator in and out of the optical path, as shown in FIG. 3B. Inaccurate positioning of the compensator may cause degradation of the focus score.

[0044] The compensator function can be implemented by using several design alternatives. For example, a compensation color can be positioned in the light path and adjusted between binary states. For example, a first state may correspond to a situation where the objective is focused on the bottom surface of the flow cell and detects an image from the bottom surface of the flow cell. In a second state of the color, the objective may be focused on the top surface of the flow cell and detect an image from the top surface of the flow cell. A malfunction of the compensator may be caused by an inaccurate focus shift between the two surfaces. A low focus score for the image from one surface and a lack of expected correlation between the focus scores of the corresponding signal channels from the two surfaces may indicate a malfunction of the compensator subcomponent.

[0045] Examples of inaccurately labeled repair records It has been observed that a significant percentage of repair calls result in the replacement of an incorrect subcomponent, thus causing considerable losses to the operators of such equipment. Technicians who use the overall sequencing confidence score to identify that a machine is malfunctioning often pick the wrong subcomponent for replacement, resulting in replacing multiple subcomponents when only one is faulty.

[0046] A repair technician may indicate in the "Parts Replaced" section of a repair ticket (or repair record) one or more subcomponents that are not actually replaced or repaired. In some cases, the repair technician selects multiple or all subcomponents associated with a subsystem, such as the optical subsystem, in the parts replacement section. When a technician repairs a device, only one or a few subcomponents may be replaced. The "Work Performed" section of a repair record may identify the subcomponents that have been replaced or repaired in some repair tickets. The disclosed technology may apply text mining techniques to analyze the "Work Performed" section of a repair ticket. The "Work Performed" section includes free-form text as shown in the following repair ticket example. The new labels extracted using text mining are then used to update the labels of the training data example. The accurately labeled training data may be used to generate rules for an expert fault identification rules engine.

[0047] In the following four repair calls (Ticket 1-Ticket 4), the M3 subcomponent was selected by the repair technician in the part replacement section, but after analyzing the description of the work performed, our technique identified that the M3 subcomponent was not the cause of the system malfunction, and that the z-stage subcomponent was likely repaired or replaced by the repair technician. We present below the text from the work performed section of the repair records:

[0048] Ticket 1: "The z-stage controller was replaced following MISEQ field repair guide pn:15032181" Ticket 2: "Per field repair guide, replaced chiller, checked chiller temperature. Completed all optical tests." Ticket 3: "Replaced main valve, manifold, green line syringe. 7 / 6 unit static problem pr2 line. Line went through vcl. 7 / 7 replaced pr2 line. Ran flow test. Passed." Ticket 4: "Engineer needs to replace / fix compensator. Equipment contract is ending and customer will update it. rr 270916 - checked images of faulty run and it doesn't seem to be compensator focus issue, blurry edges. tested and seems to work properly anyway. optical inspection checked z height top surface -0.13. adjusted along optics. boltpac passed. also tested fluid temp work. updated calibration file with z height."

[0049] In the following three repair calls (Ticket 5-Ticket 7), the z-stage subcomponent was selected by the repair technician within the part replacement section, but after analyzing the description of the work performed, our technique identified that the z-stage subcomponent was not the cause of the system malfunction.

[0050] Ticket 5: "Replaced y stage, m3 mirror motor, mirror, and valve manifold." Ticket 6: "1. Replaced m3 motor m3 motor sensor. 2. Optical path adjustment completed. 3. Cleaning fluid part vcl test. -> Boltpac / vcl test passed." Ticket 7: "· Cracks were found in the manifold so it was replaced and the optics were adjusted."

[0051] Some of the information about sequencing quality and its relationship to subcomponent failures and fault identification is unclear. Technicians who use the overall sequencing confidence score to confirm that a machine is malfunctioning often pick the wrong subcomponent for replacement and end up replacing multiple subcomponents when only one is faulty. Thus, the disclosed technique cleans the raw ground truth data by correctly labeling the collected time series data with subcomponent failures. We then present a time series preprocessor that can be used to extract features from the correctly labeled time series data. We present an exemplary text mining technique that was used to assign subcomponent labels to repair tickets.

[0052] Text Mining Techniques We applied text mining techniques to label faulty subcomponents in the Work Done section of the repair ticket. The input to the text mining techniques is free-form text from the "Work Done" section of the repair ticket. The text strings from the "Work Done" section are pre-processed before the text mining algorithm is applied. Steps 1-4 below are pre-processing steps. Step 5 involves text mining and step 6 produces the output.

[0053] Step 1 involves normalizing the text to the lower case alphabet. Step 2 involves removing stop words from the string, e.g. "the", "and", etc. These are redundant words that do not provide any useful information to the analysis. Step 3 involves removing frequently occurring words that are not interesting for our analysis. Words of interest for this analysis might include "m3", "mirror", "optical", "replaced", etc. Step 4 involves lemmatizing or stemming the words, for example for verbs the root of the verb is used instead of the past or future tense of the verb.

[0054] After preprocessing, text mining is performed on the text string. The disclosed technology can apply various available text mining techniques. For example, in one approach, a rule-based technique can be used to identify subcomponents and completed operations such as "exchange". In another approach, topic modeling can be applied. However, topic modeling and testing requires larger labeled data for training. In our case, we did not have enough labeled data to test all cases with topic modeling. As more labeled data becomes available, this technique can be applied. Other techniques that can be applied include text classification and name entity recognition (NER). NER considers the context around an entity when analyzing text. We present an exemplary rule-based algorithm for identifying M3 subcomponents in text.

[0055] Step 5 involves using the following text mining algorithm to identify the subcomponents: Similar rule-based algorithms can be used to identify the z-stage and compensator subcomponents to replace in the work performed section of the repair ticket.

[0056] 1:def classify_m3_replaced(): 2:print('m3 exchange classification.') 3: training_data = data_processed.copy() # Preprocessed data 4:training_data['label_code']=0 5:for index,row in training_data.iterrows(): 6:work=row['WorkPerformed'] 7:sentenceList=[x.strip()for x in work.split(’。’)] 8:for sentence in sentenceList: 9:if ’replace’ in sentence and ’m3’ in sentence: 10:training_data.at[index,’label_code’]=1 11:if ’replace’ in sentence and ’mirror’ in sentence: 12:training_data.at[index,’label_code’]=1 13:if ’change’ in sentence and ’m3’ in sentence: 14:training_data.at[index,’label_code’]=1 15:if ’order’ in sentence and ’m3’ in sentence: 16:training_data.at[index,’label_code’]=1 17:if ’install’ in sentence and ’m3’ in sentence: 18:training_data.at[index,’label_code’]=1 19:training_data.to_csv(output_path)

[0057] The "classify_m3_replaced" method receives the preprocessed text string at line 3, as shown above. The method parses the text string in a loop starting at line 5. Rules are applied at lines 9, 11, 13, 15, and 17. For example, at line 9, logic checks whether "replace" and "m3" are present in the text string. If the condition is true, meaning both of these terms are present in the text string, the label "1" is assigned to the training data example. The method presented above assigns the training data example a "true" (or "1") label if any one of the rules at lines 9, 11, 13, 15, and 17 is true. If none of the rules evaluate to "true," the training data is labeled as "false" (or "0"). Since the method is testing the M3 subcomponent, a "true" label means work performed by a repair technician includes replacing or repairing the M3 subcomponent. The system may include similar methods for the z-stage and compensator subcomponents, so that training data examples can be assigned one or more subcomponent labels using the "true" outputs from the corresponding methods.

[0058] Step 6 is an output step which involves assigning one or more labels to the subcomponents as determined using the rule-based algorithm described above.

[0059] Time Series Preprocessor The sequencing instrument can generate primary metrics including focus scores and intensity values ​​for multiple signal channels. The time series data from the multiple signal channels is pre-processed to prepare training data for the classifier. It is difficult to identify features in the time series data that are associated with a specific subcomponent failure. The disclosed technology was developed using a graphical representation of the time series data to identify features that can identify the subcomponent causing the failure. Two types of time series data, focus scores and image intensity, are plotted.

[0060] The focus score is defined as the average full width at half maximum (FWHM) of the molecular clusters. In one example, the minimum and maximum contrast values ​​are each 10 th ~99.5 th The minimum contrast value is in the 15th percentile. th Percentile or 5 th Similarly, the maximum contrast value can be the 90th percentile. th Percentile or 95 th The selected column may be a percentile. The selected column may be a particular tile or lane of a flow cell. The process of determining the intensity value of each cluster in the template for a given sequencing image is called intensity extraction. To extract the intensity, a background is calculated for the cluster using a portion of the image that contains the cluster. The background signal is subtracted from the cluster signal to determine the intensity.

[0061] We present details of four program detectors that can compute features that are used to identify the subcomponents causing the malfunction.

[0062] Discontinuity Detector The discontinuity detector identifies, for example, large jumps between consecutive cycles in the focus score. A discontinuity in the focus score time series may indicate a failure of the z-stage subcomponent to adjust the position of the objective lens for each tile on the surface of the flow cell, which may not be optically flat. A defective z-stage subcomponent may result in an imprecise focus of the image tile resulting in discontinuities (or jumps) in the focus score time series for multiple signal channels. The discontinuity detector detects jumps in the focus score time series that may indicate a failure of the z-stage subcomponent. It has been observed that a defective z-stage component may cause more discontinuities in the focus score time series of the top surface of the flow cell compared to the bottom surface of the flow cell. As a result, the time series of corresponding signal channels from the two surfaces may not follow similar signal patterns and therefore lack the expected correlation.

[0063] FIG. 4A presents a graph diagram 405 showing a focus score time series 406 and an image intensity time series 407 for a good sequencing run. The focus score and intensity values ​​are plotted for the cycles (along the x-axis) of the sequencing run. As mentioned above, some sequencing runs may have two reads from opposite ends. In such cases, the time series of the two reads can be plotted separately or together in one graph. The focus score and intensity value time series are plotted separately for the A, C, T, and G signal channels. In a good sequencing cycle, the focus scores for the four signal channels are smooth, with focus score values ​​between 2.5 and 3.5 (graph 406). The intensity value time series gradually increases from the beginning of the sequencing run, with around 150 for the A, G, T signal channels and around 300 for the C signal channel. The intensity gradually decreases towards the end of the sequencing run. The focus score and image intensity time series for a good sequencing run do not have large fluctuations or large jumps.

[0064] FIG. 415 presents time series data of focus score (graph 417) and image intensity (graph 422) for a sequencing run with a z-stage subcomponent malfunction. The focus score time series for the G and T signal channels show discontinuities or large jumps (graph 419). The discontinuity detector can calculate the magnitude of the difference between the focus scores in consecutive cycles in the focus score time series for each image channel in the sequencing run. The focus score time series can be smoothed prior to the calculation of the difference. The discontinuity detector includes logic to detect the number of consecutive cycles in which the magnitude of the difference between consecutive cycles exceeds a predefined threshold. This calculated feature can be used as one of the input features in the training data to train a fault detection classifier to detect the anomalous signal channel. An expert fault identification rule can check this calculated feature along with other conditions to identify the z-stage subcomponent as the cause of the system malfunction.

[0065] Discontinuity detection can use the "big_jump" method below to compute a "big jump" feature of the focus score time series of a signal channel. The method computes the difference between a point and a previous point in the focus score time series for the channel and compares this difference to a threshold. The threshold can be a magnitude or a positive and negative difference value. For positive and negative differences, an exemplary threshold pair is two positive or one negative standard deviation in the difference. Other thresholds can be set larger or smaller than the exemplary thresholds mentioned above. For magnitude, an exemplary threshold is two standard deviations. Other thresholds larger or smaller than two standard deviations can be used. The two computed features num_fs_max_bigJump and num_fs_min_bigJump are stored in a computed features database to train the fault detection classifier. In the big jump method presented below, comments are included for each line of code beginning with a "#" symbol that describes the logic implemented in the line. Lines are numbered using the line number followed by a colon ":" symbol.

[0066] How to make a big jump 1:measures=focus_scores #Store the focus scores for the signal channels in a list data structure 2: std = std(measures) # Calculate the standard deviation of a time series 3:diffs=diff(measures) #Find the difference between each point in the time series and the previous point 4: max_indexes = np.where(diffs>2*std) # Identify indexes whose diffs are greater than 2xstd 5: min_indexes = np.where(diffs ≤ -std) # Identify indexes whose diffs are less than -1xstd 6: channel + "_num_fs_max_bigJump" = len(measures) / length(max_indexes) # Divide the sum 7: Divide the number of data points in the focus score time series by the count of max_indexes 8: channel + "_num_fs_min_bigJump" = len(measures) / length(min_indexes) # Divide the sum 9: Divide the number of data points in the focus score time series by the count of min_indexes

[0067] FIG. 4B presents the time series of the bottom and top surfaces of the flow cell for all signal channels. In the bottom surface (graph 430), the focus score time series (labeled 431) for the A, C, G, and T signal channels appear normal within the range of 2.5-3.5, with no major discontinuities present within the signal channels. In the top surface (graph 435), the focus score time series (labeled 436) for the A, C, G, and T signal channels appear abnormal with large discontinuities or large jumps. The values ​​of the data points within the time series are also higher than the normal range (2.5-3.5), indicating poor focus quality. It has been observed that a defective z-stage component may cause more discontinuities within the focus score time series for the top surface of the flow cell compared to the bottom surface of the flow cell. Expert fault identification rules can use and check these conditions to identify z-stage subcomponent malfunctions as possible causes of system failure.

[0068] Drift Detector The drift detector compares the drift over time of the signals captured by the cameras of the different optical paths to identify malfunctions of components present in one optical path but not the other. In some sequencing systems, the M3 mirror subcomponent is positioned on one optical leg and not on the other, and thus the signature calculated from the drift detector can identify M3 subcomponent failures. When the M3 subcomponent malfunctions, the images from both the red and green actuation of one camera suffer a focus score of FWHM degradation relative to the other camera.

[0069] FIG. 4D presents an example of drift caused by an M3 subcomponent malfunction. FIG. 480 presents a graphical plot of focus score time series 486 for four signal channels. Two signal channels A and G detected by camera S2 drift together in close proximity to each other. Similarly, C and T signal channels detected by camera S1 drift together following a similar signal pattern. This situation may indicate a possible M3 subcomponent failure. The M3 detector can implement logic to capture the signal channel drift phenomenon. In one embodiment, the M3 drift method presented below detects the drift of two signal time series from one camera.

[0070] How to Drift the M3 #Applying a smoothing filter 1:a_np=smooth(A_read[0].focus_scores) 2:c_np=smooth(C_read[0].focus_scores) 3:g_np=smooth(G_read[0].focus_scores) 4:t_np=smooth(T_read[0].focus_scores) #Calculate the difference between each cycle and the same cycle in another channel 5:ac_diff_abs=absolute(subtract(a_np, c_np)) 6:gt_diff_abs=absolute(subtract(g_np, t_np)) 7:ag_diff_abs=absolute(subtract(a_np, g_np)) 8:ct_diff_abs=absolute(subtract(c_np, t_np)) # Split the time series into two parts (we want to see if the second half of the signal is drifting from the first half) 9:window=round(length(A_read[0].focus_scores) / 2) #Calculate the average difference between the second half and the first half 10:dif_mean_ac=absolute(mean(ac_diff_abs[window:])-mean(ac_diff_abs[:window])) 11:dif_mean_ag=absolute(mean(ag_diff_abs[window:])-mean(ag_diff_abs[:window])) 12:dif_mean_ct=absolute(mean(ct_diff_abs[window:])-mean(ct_diff_abs[:window])) 13:dif_mean_gt=absolute(mean(gt_diff_abs[window:])-mean(gt_diff_abs[:window])) #If (A and G) or (T and C) drift together, it is an M3 drift 14:if dif_mean_ac > dif_mean_ag and dif_mean_gt > dif_mean_ct : #Other values ​​of the threshold can be set so that more instances of M3 drift are analyzed. 15:if dif_mean_ac > 0.2: 16:num_of_drifts=num_of_drifts+1 17:if dif_mean_gt > 0.2: 18:num_of_drifts=num_of_drifts+1 19: if num_of_drifts == 2: 20:return True 21:else: return False

[0071] In the first block of the M3 drift method (lines 1-4), a smoothing filter (e.g., Savgol filter, moving average filter, kernel smoother, etc.) is applied to remove noise.

[0072] The second block of the M3 drift method (lines 5-8) calculates the difference between the focus score values ​​of pairs of signal channels. The pairs for which the difference is calculated are selected using the camera pair channel and the LED pair channel. For example, the pair AC and GT are based on the same LED light source. A red LED light source excites the dye to detect the A and C signal channels, and a green LED light source excites the dye to detect the G and T signal channels. The signal channel pairs AG and CT are based on detection by the same camera. Figure 2A shows that the C and T signals are detected by camera S1, and the A and G signals are detected by camera S2. The difference between the signal channels of these pairs can help identify possible drift of the channels. If the signal channels of a pair from the same camera are drifting, a possible M3 drift can be identified because the M3 subcomponent is present in the light path of one camera (S1) but not the light path of the other camera (S2).

[0073] In the third block of the M3 drift method (line 9), the focus score time series is split into two parts (or windows) and the focus scores in the second part are compared with the focus scores in the first part to detect drift.

[0074] In the fourth block of the M3 drift method (lines 10–13), the average difference between the first and second halves of the four pairs of signal channels presented above (second block) is calculated.

[0075] In the fifth block of the M3 drift method (lines 14-20), the average difference between pairs of signal channels is compared. Two camera pair channels (AG and CT) are compared with two LED pair channels (AC and GT) to check if the difference of the camera pair signal channel is smaller than the LED pair signal channel (line 14). If this condition is true, the average difference of each LED pair AC and GT is compared with a threshold (0.2) in lines 15 and 17, respectively, to ensure that they are above 0.2. The threshold can be set above or below 0.2 based on the analysis of more M3 failure cases in sequencing equipment. If both conditions are true, two counters are incremented in lines 16 and 19, respectively. If the counters reach 2 (line 19), the M3 method returns "true" (line 20), which means that M3 drift is present in the observed signal channel. Otherwise, the method returns "false" in line 21. The calculated M3 drift features may be stored in a calculated features database 116.

[0076] Surface correlation detector The surface-to-surface correlation detector detects whether a signal channel on the top surface of the flow cell follows a similar signal pattern as a corresponding signal channel on the bottom surface of the flow cell. When the optical subsystem components are operating normally, the corresponding signal channels on the top and bottom surfaces follow a similar signal pattern. A failure of the optical system subcomponents can cause a variation in the signal channel pattern between the top and bottom surfaces.

[0077] FIG. 4B shows a focus score time series 431 for a signal channel from the bottom surface of the flow cell and a focus score time series 436 for a corresponding signal channel from the top surface. The multiple time series from the top surface contain discontinuities or large jumps, while the multiple corresponding time series from the bottom surface are smooth. The inter-surface correlation detector may calculate the statistical correlation using, for example, the Pearson correlation coefficient. The correlation coefficient may range from -1 to 1. A correlation coefficient value closer to 1 indicates a higher correlation between the time series of each signal channel from the two surfaces. A separate correlation coefficient is calculated for the channels between the top and bottom surfaces. Thus, the inter-surface correlation detector may generate four calculated features, "A_correlation", "C_correlation", "G_correlation", and "T_correlation". A lack of expected correlation between the abnormal signal channel 436 and the normal signal channel 431 may indicate a failure of the z-stage or compensator subcomponent. The calculated features may be stored in the calculated features database 116.

[0078] FIG. 4C includes time series of four signal channels from the bottom surface 460 and the top surface 470 of the flow cell. In this example, the focus score time series 461 from the bottom surface has discontinuities, but the corresponding signal channel 471 from the top surface is in the range of 2.5 to 3.5, which indicates good focus quality. The surface-to-surface correlation detector can calculate the correlation coefficient values ​​of the four signal channels. The expert fault identification rule can check whether all signal channels from the bottom surface are abnormal and whether at least some signal channels from the top surface are normal. These two conditions may indicate a possible compensator subcomponent malfunction.

[0079] Trend Detector The trend detector can identify focus degradation when the focus score values ​​in the last portion of the focus score time series are higher than the focus score values ​​in the first portion of the focus score time series. For example, the focus score time series 436 of the signal channel from the top surface (in FIG. 4B) has relatively higher focus score values ​​in the last portion of the time series compared to the first portion of the time series. In one embodiment, the trend detector can calculate the trend using the logic presented in the trend detection method below.

[0080] Trend Detection Methods 1:first_mean = the mean value of the first window 2:last_mean = last window mean 3:slope=(last_mean-first_mean) / (len(focus_scores)-2*window_size)

[0081] The focus score time series is divided into 10 windows, one for each signal channel. The size of the window is equal to the number of focus score values ​​in the window. Fewer or more than 10 windows can be used to partition the focus score time series. The trend detection method calculates two averages for the focus score time series. The first average (line 1) is for the first window of the time series, and the second average (line 2) is for the last window of the focus score time series. It should be understood that other windows (or partitions) of the time series data can be used to determine the first and last averages. For example, the mean value of the second window can be used to determine the "first_mean" in line 1 of the trend detection method, and the mean value of the penultimate window can be used to determine the "last_mean" in line 2 of the trend detection method. Data from other windows can be used to calculate the "first_mean" and "last_mean" values.

[0082] In the third line of the trend detection method, the ratio of the difference between the average of the last window and the average of the first window and the number of focus score values ​​between the first and last windows is calculated. A higher value of the slope may indicate a signal trend, and an increasing value of the focus score indicates focus degradation towards the end of the sequencing run. The slope features (A_fs_delta_slope, C_fs_delta_slope, G_fs_delta_slope, T_fs_delta_slope) for the four focus score times per surface of the read are stored in the calculated feature database 116.

[0083] The primary metric and the calculated features can be fed to a fault detection classifier to classify the signal channel as normal or abnormal. We present the training of an exemplary fault detection classifier and application of the trained classifier to classify production time-series data.

[0084] In-process vibration detector The in-process vibration detector processes the focus score (FWHM) time series from at least one signal channel of the read and includes logic to identify signal channels that are normal in the beginning and end but contain irregular spikes with higher average focus score (or FWHM) between the two normal or good parts. For example, FIG. 4E shows an intensity value time series 490 (top) and a focus score time series 492 (bottom) for a signal channel from a sequencing instrument. The focus score time series includes two goods parts 494 and 498 at the beginning and end of the sequencing read. However, the middle part 496 shows discontinuities (or spikes) in all four signal channels. This in-process vibration feature in the focus score time series indicates a z-stage problem that may be caused by tight engagement of the cap screws used to secure the z-stage subcomponents. We present an exemplary in-process vibration detection method below, followed by a description of the method.

[0085] In-process vibration detection method 1:def is_the_jump(measures, std): 2:diffs = np.diff(measures) 3:max_indexes=np.where(diffs>std) 4:min_indexes=np.where(diffs≦std) 5:if (len(max_indexes[0]) ==0) or (len(min_indexes[0])==0): 6: return False 7:firs_max=max_indexes[0][0] 8: The first good part 9:good_part=measures[0:firs_max] 10: if len(good_part)==0: 11: return False 12:good_avg=sum(good_part) / len(good_part) 13:#Middle binding 14: bad_part = measures [max_indexes [0] [0]: min_indexes [0] [-1]] # First maximum value 15: To the last minimum value (From the first rise to the last fall 16: if len(bad_part) == 0: 17: return False 18:bad_avg=sum(bad_part) / len(bad_part) 19: #The last good part 20:good_part2=measures[min_indexes[0][-1]:-1]#Final descent before the end 21: if len(good_part2) == 0: 22: return False 23:good_avg2=sum(good_part2) / len(good_part2) 24:if (bad_avg - good_avg) > 0.1 and (bad_avg - good_avg2) > 0.1: 25:print(str(bad_avg-good_avg)+``,''+str(bad_avg-good_avg2)) 26: return True

[0086] The "is_the_jump" method presented above can detect in-process oscillations in a focus score time series, as shown in FIG. 4E. In line 1 of the method presented above, the method receives the focus score time series (as the "measure" parameter) and the standard deviation (as the "std" parameter). In line 2, the method calculates "diff", which is the difference between the focus scores of consecutive cycles in the lead. In lines 3 and 4, the method determines the index of the element in the "diff" data structure where the difference value between consecutive cycles exceeds a threshold, such as one standard deviation. Other thresholds greater than one standard deviation can be used. The threshold can be a magnitude or a positive and negative difference value. For positive and negative difference, an exemplary threshold pair is one positive or one negative standard deviation in the difference. In lines 5 and 6, the results in the "max_indexes" and "min_indexes" data structures are checked to see if they are not empty. If any one of the two data structures is empty, the method returns a "false" output indicating that the time series does not contain any oscillatory features in the process.

[0087] The method includes logic to detect the first element in the time series where a jump occurs. This element may be the first data point in the time series (or one of the first three or five data points) at the beginning of the middle part shown as 496 in the time series of FIG. 4E. The method stores the index of the first data point as "firs_max" in line 7. The method stores the part of the time series before "firs_max" as "good_part" in line 9. The good part represents the first part of the time series indicated by label 494. The average of the data points in the first good part is calculated in line 12 and saved in the "good_ave" variable.

[0088] The method detects the bad portion of the time series in line 14, which corresponds to the portion of the time series labeled 496 in FIG. 4E. The “bad_avg” variable stores the average focus score during the bad portion of the time series in line 18.

[0089] The method finds a second good portion of the time series in line 20. The second good portion of the time series is labeled as 498 in FIG. 4E. The average of the second good portion is stored in the “good_avg2” variable in line 23.

[0090] The method compares the difference between the bad average and the two good averages in line 24. If the difference between the bad average and the good average is greater than "0.1" for both the first and second good portions, the method returns "true" indicating that the time series contains an in-process vibration feature. Other values ​​of the threshold greater than or less than "0.1" may be used to compare the averages in line 24. The presence of an in-process vibration feature in the focus score time series may indicate a z-stage failure. The results from the above method may be evaluated by an expert rule. If the method returns "true," the expert rule may generate an alert for a z-stage subcomponent malfunction. The expert rule may also present a description of the alert, such as "z-stage malfunction caused by tight engagement of cap screws."

[0091] Fault Detection Classifier The disclosed technique applies fault detection classifiers and expert fault identification rules in a stepwise manner. In a first step, a classifier is applied to classify the signal channel time series as normal or abnormal. The abnormal classification of the signal channel may indicate a repairable fault and is further processed with a hand-crafted expert rule set to identify the subcomponent causing the malfunction. Random forest is one example of a classifier applied to classify the signal channel. Other examples of classifiers that may be applied for fault detection include decision tree classifiers, eXtreme Gradient Boosting (XGBoost) classifiers, gradient boosted decision trees (GBDT) classifiers, AdaBoost classifiers, etc. In other implementations, classifiers such as support vector machines (SVMs), logistic regression, etc. may be applied. Deep neural networks may also be applied for fault detection classification using large, labeled training data sets. In the following sections, we present the training of a random forest classifier and the application of the trained model to production data.

[0092] Random forest classifiers (also called random decision forests) are an ensemble machine learning technique. Ensemble techniques or algorithms combine more than one technique of the same or different kinds to classify objects. A random forest classifier consists of multiple decision trees operating as an ensemble. Each individual decision tree in the random forest acts as a base classifier and outputs a class prediction. The class with the most votes becomes the prediction of the random forest model. The basic idea behind random forests is that a large number of relatively uncorrelated models (decision trees) operating as a committee will favor one of the individual constituent models.

[0093] Classifier Training Architecture FIG. 5A presents an exemplary training architecture of the fault detection classifier 183. The training data consists of a time series of signal channels from a sequencing run. The training data includes primary metrics 514 and calculated features 516. The training data examples 525 include primary metrics and calculated features from one read per surface of the flow cell. There may be two reads per sequencing run, and the flow cell may have two surfaces containing samples, so there may be up to four examples (or rows) of training data per sequencing run. A sequencing run may have fewer than four rows if the run is stopped before completion or if data is collected for one read. The ground truth training data is prepared by first cleaning the data and then applying labels. The training data can be cleaned by applying a smoothing filter to remove noise. For example, a Savgol (Savitzky-Golay) filter can be applied to the time series data points to smooth the data, i.e., to increase the accuracy of the data without distorting the signal trends. The example time series data is labeled as normal (labeled as "0") or abnormal (labeled as "1"). Labeling is accomplished by reviewing time series graphs such as those shown in Figures 4A-4C and identifying sub-component failures.

[0094] In one example, the model was trained using about 300 labeled training data examples. The size of the training data will increase as more fault data is collected from users of the sequencing system. We next present examples of input features for training the classifier. The input features include the primary metric and the computed features.

[0095] The primary metrics in the primary metric database 514 may include the following data: Example training data 525 is broken down into smaller tables labeled "1" through "12" in Figures 5B, 5C, and 5D. The primary metrics are provided by the instrument. Figures 5B, 5C, and 5D present example data values ​​525 for the primary metrics and calculated features. In Figure 5B, the tables labeled "1", "2", and "3" present the primary metrics and the table labeled "4" presents the calculated features. The table headers present the feature names, and three rows of data from the example training data are shown in Figures 5B, 5C, and 5D for illustrative purposes. We have listed the primary metrics below.

[0096] Surface identifier (1 or 2), Number of leads (1 or 2), -Start date and time of the sequencing run, - an identifier for the sequencing run; -Serial number of the sequencing device, Sequencing run status (completed or stopped), the size (e.g., in bytes) of the data output by the sequencing instrument for a sequencing run; Q30 quality score, The proportion of aligned sequences, Error rate, which refers to the proportion of incorrectly called bases, and can be calculated from reads aligned against the PhiX control in a sample The maximum number of cycles in a sequencing run, a tile count, indicating the number of tiles imaged in the sequencing run; A concatenation of the sequencing run identifier, the read number, and the surface identifier

[0097] The computed features in the computed feature database 516 may include the following computed features: Figure 5B (Table 4), Figure 5C (Tables 5, 6, 7, 8), and Figure 5D (Tables 9, 10, 11, 12) provide example values ​​for the computed features. The top row of the table lists the feature name. Three example values ​​for each computed feature are shown. Examples of computed features are listed below: Average strength of leads per surface, Mean focus score for leads per surface, Intensity variability per signal channel calculated as the absolute intensity difference between each cycle (A_intensity_change, C_intensity_change, G_intensity_change, T_intensity_change), · The variability of focus scores per channel calculated as the absolute focus score difference between each cycle (A_fs_change, C_fs_change, G_fs_change, T_fs_change), Gradient features per image channel calculated by partitioning the focus score time series into 10 windows and comparing the focus score from the last window to the first window, where the gradient features may be calculated using the trend detection method presented above (A_fs_delta_slope, C_fs_delta_slope, G_fs_delta_slope, T_fs_delta_slope); Two big jump features per signal channel identifying discontinuities in the time series calculated by applying the M3 drift method presented above, (A_num_fs_max_bigJump, (A_num_fs_min_bigJump, C_num_fs_max_bigJump, C_num_fs_min_bigJump, G_num_fs_max_bigJump, G_num_fs_min_bigJump, T_num_fs_max_bigJump, T_num_fs_min_bigJump), Correlation features calculated per channel using the Pearson correlation coefficient, where the correlation is calculated between the focus score time series for the two surfaces per image channel (A_correlation, C_correlation, G_correlation, T_correlation) M3 drift features (true or false) calculated using the M3 drift method presented above, The minimum and maximum focus score values ​​per image channel per surface within the lead (A_fs_min, A_fs_max, C_fs_min, C_fs_max, G_fs_min, G_fs_max, T_fs_min, T_fs_max), The difference between the maximum and minimum focus score values ​​per image channel per surface within the lead (A_fs_min_max_diff=A_fs_max-A_fs_min, C_fs_min_max_diff=C_fs_max-C_fs_min, G_fs_min_max_diff=G_fs_max-G_fs_min, T_fs_min_max_diff=T_fs_max-T_fs_min), The focus score between the two surfaces by calculating the difference between channel_fs_min_max_diff (as calculated above) for each channel of the two surfaces and taking the average (channels = A, C, G, T)

[0098] The example training data 525 includes input features including the primary metric and feature values ​​calculated for one read per surface of the sequencing run. A random forest classifier with 100 decision trees and a depth of 12 performed well. It is understood that a range of 100-300 decision trees or 100-400 decision trees and a random forest classifier with a depth range of 10-20 or 8-22 are expected to provide good results for this embodiment. The hyperparameters are tuned using randomized search cross-validation. Increasing the number of trees may increase the performance of the model, but may also increase the time required for training.

[0099] Decision trees are prone to overfitting. To overcome this problem, bagging techniques can be used to train decision trees in random forests. Bagging is a combination of bootstrapping and aggregation techniques. In bootstrapping, during training, row samples are taken from the training database and used to train each decision tree in the random forest. For example, a subset of selected row features can be used to train decision tree 1. Hence, the training data for decision tree 1 can be referred to as row sample 1 with column sample 1, or RS1+CS1. The columns or features can be selected randomly. Decision tree 2 and subsequent decision trees in the random forest are trained in a similar manner by using a subset of the training data. Note that the training data for decision trees is generated with replacement, i.e., the same row data can be used in training multiple decision trees.

[0100] The second part of the bagging technique is the aggregation part that is applied during production. Each decision tree outputs a classification for each class. For binary classification, it can be a "1" indicating a bad signal channel or a "0" indicating a good signal channel. The output of the random forest is an aggregation of the outputs of the decision trees in the random forest with the majority of votes selected as the output of the random forest. By using votes from multiple decision trees, the random forest reduces the high variance in the decision tree results, thus resulting in good prediction results. By using row and column sampling to train individual decision trees, each decision tree becomes an expert on the training records with the selected features.

[0101] During training, the output of the random forest is compared to the ground truth labels and the prediction error is calculated. During backpropagation, the model weights are adjusted such that the prediction error is reduced. The parameters (e.g., weights) of the trained random forest classifier are stored for use in fault detection classification of production signal channels during inference.

[0102] We next describe the classification of production signal channels from sequencing runs using the trained fault detection classifier.

[0103] Classifier Production Architecture 5E illustrates a two-stage classification of sequencing runs, using a fault detection classifier 183 in the first stage and expert fault isolation rules 186 in the second stage. In the first stage, input to the fault detection classifier may include primary metrics and calculated features as described above.

[0104] The fault detection classifier 183 classifies the signal channels (per lead per surface) as normal or abnormal. Output from the classifier may include the normal / abnormal classification of the signal channel and a confidence score of the prediction. Since there are four signal channels per surface, two surfaces per lead, and two reads in a sequencing run, there may be 32 outputs from the fault detection classifier (including output labels and confidence scores per label) per sequencing run (4 signal channels x 2 surfaces x 2 reads).

[0105] The signal channel classification along with the calculated features 116 are provided as input to the expert fault isolation rules 186. The input to the expert fault isolation rules may also include some primary metrics such as a sequencing run identifier, read count, and surface count. The expert fault isolation rules suggest which subcomponents to replace and present reasoning that is embodied in the rules that are applied. We present details of the fault isolation rules in the following sections.

[0106] Expert fault identification rules The signal channel classification is processed against a manually created expert rule set to identify the malfunctioning subcomponent. Figures 6A, 6B, and 6C present the processing of the classification of multiple signal channels for one read (Figure 6C), then the classification combined across two reads in a run (Figure 6B), and finally the classification combined across multiple sequencing runs (Figure 6A) to identify the malfunctioning subcomponent.

[0107] Figure 6A presents a high level process flow chart for identifying subcomponents by processing multiple sequencing runs from a sequencing instrument. Figure 6A presents the process at the sequencing run level and is further elaborated in the flow charts of the following figures.

[0108] In step 601, a two-step process of classification and expert rule application is applied to the sequencing run. The system applies a trained classifier to multiple signal channels from the sequencing run. The results of the classification are further processed by expert fault identification rules to identify the subcomponent causing the malfunction. In step 605, the output from the expert fault identification rules is checked to see if an alert is generated. If no alert is detected in step 605, the process may be repeated for the next sequencing run (615). If an alert is detected in step 605, the process of step 601 is repeated for the previous sequencing run in process step 620. In one embodiment, the immediately preceding sequencing run is evaluated in step 620. In another embodiment, a sequencing run earlier than the immediately preceding sequencing run may be evaluated in step 620. In the continuous process of sequencing runs, the system can store the results for processing alerts for a sequencing run in memory or a persistent data store and access the stored results for any of the previously processed sequencing runs to check whether an alert was generated.

[0109] Decision step 625 may result in two branches of process steps. If no alert is detected at step 625, the first branch (branch 1) is followed. This may indicate two alerts for two sequencing runs in a row. If no alert is detected at step 625, the second branch (branch 2) is followed. When an alert is first generated, it may be an informational alert, and when an alert is generated for two sequencing runs, one after the other, a warning or critical alert may be generated.

[0110] If an alert is detected in step 625, the process proceeds to branch 1 to generate a warning alert. If the expert fault attribution rules generate an alert (625), the system may apply further expert rules to determine the subcomponent (or part) causing the malfunction (645). The severity level of the alert may be determined by the expert fault attribution rules engine (650). The severity of the alert may determine the follow-up action or workflow initiated from the alert.

[0111] If no alert is detected in step 625, the process proceeds to branch 2, generating an informational alert. The quality score for the sequencing run may be checked to see if it is above the minimum required quality score (630). If the quality score is good, no alert is generated (615). If the quality score is not good or is below the required threshold, the system may generate an informational alert about the suspect subcomponent 660. An informational alert may not generate a case or ticket for the maintenance team. It may be provided on a graphical user interface (GUI) as an informational alert to the maintenance team and / or the operator of the sequencing equipment.

[0112] FIG. 6B presents details of processing a sequencing run by evaluating data from two reads in the sequencing run. For example, the process steps in FIG. 6B can be performed in each of process sequencing run process steps 601 or 620. In this process flow chart, the two reads in the sequencing run are processed by applying a two-step process of a classifier followed by an expert-identified rule (steps 671 and 673). If both process steps 671 and 673 identify the same subcomponent that is causing the failure (step 675), the system determines the severity level of the alert in step 679. If the two process steps 671 and 673 do not identify the same subcomponent that is causing the system malfunction, an additional expert rule can be applied in step 679 to identify the subcomponent. The process then follows the steps described above to generate an alert.

[0113] 6C presents an example of applying expert-specific rules to classify multiple signal channels for two surfaces in a read to identify the subcomponent causing the system malfunction. The process starts at step 685, when the fault detection classifier 183 is called to classify all signal channels for two surfaces in a read of a sequencing run. There are four signal channels corresponding to the four types of nucleotides A, C, G, and T. Since images can be captured from two surfaces of the flow cell within one read, therefore up to eight signal channels can be classified by the fault detection classifier 183 for each read.

[0114] The classification results are evaluated by the fault identification rules engine 186 to identify the subcomponent causing the malfunction. Figure 6C presents an example of a set of four rules that may be used to identify the subcomponent.

[0115] The first exemplary rule checks whether the classification of all signal channels from the surface is anomalous (687).

[0116] A second example rule checks if the classification of all signal channels from the bottom surface is anomalous (689). The fault detection classifier provides a confidence score of the classification of the signal channels. The expert identification rule engine can use the confidence score from the classifier when identifying the subcomponent that is causing the system malfunction. We will explain the use of the confidence score in identifying the subcomponent when presenting an example expert rule (Expert Rule 4).

[0117] A third exemplary rule, in FIG. 6C, checks whether the classifications of corresponding signal channels from the top and bottom surfaces are highly correlated (691).

[0118] The fourth exemplary rule checks whether the sequencing run included a PhiX control library (693). The PhiX control is a reliable library that can be sequenced alone and serves as a calibration control. Due to its balanced nucleotide composition, the PhiX library can be used for sequencing run quality monitoring (e.g., cluster generation, sequencing, and alignment). The library provides a balanced fluorescent signal at each cycle. If the PhiX library is used in a sequencing run, it can help identify sample quality issues. If the cluster generation from the PhiX library is consistent with the expected results, the sample is likely not responsible for any quality issues in base calling.

[0119] The results of the four rules are further evaluated by other expert-specific rules in step 695 to identify subcomponents that may be causing the system malfunction.

[0120] We now present some detailed examples of fault isolation rules, including identification of the subcomponent causing the fault.

[0121] Fault Identification Rule Examples We present four rules as examples illustrating the identification of subcomponents causing system malfunctions.

[0122] Rule 1-M3 Alert If M3-drift == True on both surfaces Then read alert = M3

[0123] Rule 1 tests the value of the "M3 Drift" calculated feature. If the value is true for both surfaces of the lead, the expert identification rules engine determines an M3 subcomponent malfunction as the cause of the system failure and generates an alert for the M3 subcomponent.

[0124] Rule 2-Z Stage Alert If top and bottom surfaces are not correlated and all signal channels from top surface are classified as abnormal and not all signal channels from bottom surface are classified as abnormal Then read alert = Z-stage warning

[0125] Rule 2 tests three conditions before generating an alert. The first condition tests whether the top and bottom surfaces are uncorrelated. The system may check the values ​​of all channel correlation calculated features. If the value of the channel correlation feature is less than a predefined threshold, the first condition may be true. The second condition tests whether all signal channels from the top surface are classified as abnormal. The third condition tests whether at least one or more signal channels from the bottom surface are classified as normal. If the three conditions are true, the rule generates a z-stage subcomponent alert that identifies the z-stage as a possible cause of the system malfunction. The rule sets the severity level of the alert as "Warning," which is more severe than an informational alert.

[0126] Rule 3 – Compensator Alert If top and bottom surfaces are not correlated and all signal channels from top surface are classified as abnormal and not all signal channels from bottom surface are classified as abnormal Then read alert = Z-stage warning

[0127] Rule 3 also tests three conditions before generating an alert. The first condition checks whether the top and bottom surfaces are uncorrelated. The second condition tests whether at least one or more signal channels from the top surface are classified as normal. The third condition tests whether all signal channels from the bottom surface are classified as abnormal. If the three conditions are true, the rule generates a Compensator subcomponent alert that identifies the compensator as a possible cause of the system malfunction. The rule sets the severity level of the alert as "Warning."

[0128] Rule 4-M3 and Z stage evaluation 1:m3_pair_bad, cam_confident=camera_pair_bad(evaluation_result) 2:if m3_pair_bad: 3:m3_votes+=1 4:if cam_confident: m3_votes+=0.5 5:led_pair_bad, led_confident=LED_pair_bad(evaluation_result) 6:if led_pair_bad: 7:zstage_votes+=1 8:if led_confident: 9:zstage_votes+=0.5 10:if m3_votes - zstage_votes >= 1.5 : #If it does not have high phi percentiles and is within spec #Good run, likely due to sophisticated optics 11:if m3_votes - zstage_votes >= 1.5 and bad_nonPhix: 12:read_df['class']=AlertDetail.M3_W.name(warning,) 13:read_df['detail']='Problem detected with one camera. Q30 out of spec' 14:else: 15:read_df['class']=AlertDetail.M3.name(info alert) 16:read_df['detail'] = 'Problem detected with one camera.' #If it does not have high phi percentiles and is within spec #Good run, likely due to sophisticated optics 17:elseif zstage_votes-m3_votes≧1.5: 18:if zstage_votes - m3_votes >= 1.5 and bad_nonPhix: 19:read_df['class']=AlertDetail.ZStage_W.name 20:read_df['detail'] = 'Problem detected in one LED color pair. Q30 is out of spec.' 21:else: 22:read_df['class']=AlertDetail.ZStage.name 23:read_df['detail'] = 'Problem detected in one LED color pair.' 24:else: 25:read_df['class']='NA' 26:read_df['detail']=''

[0129] Rule 4 can generate alerts for the M3 and z stage subcomponents by evaluating the results received from two methods. The first method is called "camera_pair_bad" and the second method is called "LED_pair_bad". Both methods return two Boolean values.

[0130] The "camera_pair_bad" method returns two Boolean values. A first Boolean value indicating whether a pair of M3 channels, i.e., pair "AG" and pair "CT", is abnormal. Each pair of signal channels is detected by the same camera, i.e., signal channels A and G are detected by camera S2, and signal channels C and T are detected by camera S1. A deterioration in the focus score in both pairs of signal channels may indicate a malfunction of the M3 subcomponent. A second Boolean value returned by the "camera_pair_bad" method indicates the confidence score of the classifier. If the confidence score for the classification of the signal channels is above a threshold, the "cam_confident" variable can be set as true, otherwise it can be set as false.

[0131] The "LED_pair_bad" method also returns two Boolean values. The first Boolean value indicates whether the pair of LED channels, i.e., pair "AC" (for red LEDs) and pair "GT" (for green LEDs), is abnormal. Each pair of signal channels is detected by illuminating the sample from excitation from the same LED. At each cycle, the z-stage is used to move the objective lens and adjust the optical focus as the LED color changes between green and red. Malfunctions in the z-stage between LED activations will degrade the focus. Deterioration of the focus score in both pairs of signal channels may indicate a malfunction of the z-stage subcomponent. The second Boolean value returned by the "LED_pair_bad" method indicates the confidence score of the classifier. If the confidence score for the classification of the signal channels is above a threshold, the "led_confident" variable can be set as true, otherwise it can be set as false.

[0132] Lines 1-9 of the computer program code for Rule 4 evaluate these Boolean values ​​for the two methods and increment counters for M3 and z stage voting.

[0133] In line 10, the difference between the M3 vote and the z stage vote is calculated. If the difference is greater than "1.5", then the logic embodied in the "if" block (lines 10-23) is executed, otherwise the "else" block (lines 24 and 25) is executed. It is understood that other thresholds greater or less than 1.5 may be used when comparing the M3 and z stage votes.

[0134] If the condition in line 10 is true, control enters an if block to check another condition in line 11. In the if statement in line 11, two conditions are tested in a logic "AND" combination. The first condition is that the difference between the M3 vote and the z-stage vote is compared to the value "1.5", and the second condition is "bad_nonPhix", which means that the sequencing run did not have a high percentage of PhiX controls and the Q30 quality score was poor. If both conditions in the statement in line 11 are true, lines 11-13 generate an M3 subcomponent warning alert. The expert fault identification rule also generates a description for the repair technician: "Problem detected with one camera. Q30 is out of spec." The alert description can help the repair technician debug the system to identify the possible root cause of the failure.

[0135] If the condition statement on line 11 is false, the blocks (lines 15 and 16) are executed. These lines of code generate an information alert for the M3 subcomponent. The information alert also contains the description "A problem was detected with one camera." Information alerts can be less severe than warning alerts.

[0136] The logic implemented in lines 17-23 implements similar logic as described above, except in this case the z-stage vote is greater than M3 votes. Therefore, the system generates an alert for the z-stage subcomponent with an explanation. For example, in the if block (lines 19 and 20), the system generates a z-stage alert with the explanation "Problem detected with one LED pair. Q30 is out of spec." In the else block (lines 22 and 23), the system generates a z-stage alert "Problem detected with one LED color pair."

[0137] If the condition statement on line 10 is false, control passes to the else block (lines 25 and 26) and no alert is generated.

[0138] FIG. 7A presents examples of alerts for M3, z-stage, and compensator subcomponents. Two or three types of alerts may be generated: an “info” (or information) alert, a warning alert, and optionally, a critical alert (as in FIG. 7B). The warning (or critical) alert may have a higher severity level compared to the information alert. The system also provides an alert summary. For example, the first alert is an example of an “info” alert for the M3 subcomponent. The alert also includes “info” alert summary for the M3 subcomponent: “Potential optical problem. M3 is likely the culprit.” The alert also includes “additional alert details.” For example, for the first M3 alert, the alert details may be “Problem detected with one camera.” Thus, as shown in FIG. 7A, the alerts generated by the expert fault identification engine 186 may provide guidance to a repair technician and reduce incorrect subcomponent replacement. FIG. 7A presents examples of “info” and warning alerts for three subcomponents.

[0139] FIG. 7B presents different types of alerts that may be generated by the system. In some implementations, the “critical” and “warning” alerts may be combined into one category and labeled as a “warning” alert. When an alert is generated, different types of actions may be automatically generated. For example, a critical alert 710 may trigger a workflow to generate a case (or ticket) and a work order for a repair technician, so that the technician can address the system malfunction without delay. A warning alert 720 may also generate a case and a work order, which are queued for evaluation before a repair technician is dispatched to address the system malfunction. An informational (or info) alert 730 may be provided to the equipment operator or user, and no case or work order is generated.

[0140] Specific Implementations The disclosed technology relates to diagnosing system malfunctions and isolating the causes of the system malfunctions among multiple replaceable subcomponents.

[0141] The disclosed technology can be implemented as a system, a method, a device, a product, a computer-readable medium, or an article of manufacture. One or more features of the embodiments can be combined with the base embodiment. Non-mutually exclusive embodiments are taught as combinable. One or more features of the embodiments can be combined with other embodiments. The present disclosure will periodically inform users of these options. The omission from some embodiments of the enumeration of repeating these options should not be interpreted as limiting the combinations taught in the preceding section. These descriptions are incorporated herein by reference into each of the following embodiments.

[0142] The disclosed technology may be practiced as a method for diagnosing a system malfunction and identifying a cause of the system malfunction among a plurality of replaceable subcomponents. The method includes applying a preprocessor to time series data from at least one sequencing run. The preprocessor may detect time series anomalies. The anomalies may include discontinuities, drift, lack of expected correlation, and trends. The preprocessor may detect time series discontinuities, detect drift, detect lack of expected correlation, and detect trends. The method includes feeding at least a portion of the output of the preprocessor to a trained tree-based classifier for at least one image channel in one sequencing run. The preprocessor may process time series of two or four image channels. The method includes receiving a classification of the at least one sequencing run as an anomaly indicative of a system malfunction. The method includes feeding at least a portion of the output of the preprocessor for the anomaly sequencing run to an expert rule system. The method includes receiving a root cause identification of the system malfunction to a particular subcomponent requiring adjustment or replacement. The subcomponent may be among a plurality of replaceable subcomponents. The method includes generating a notification of a particular subcomponent that is causing a system malfunction.

[0143] Methods and other embodiments of the disclosed technology may include one or more of the following features. For purposes of brevity, combinations of features disclosed in this application are not individually recited and are not repeated for each base set of features. Features applicable to methods, systems, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how features identified in this section can be readily combined with basic features in other statutory classifications.

[0144] In one embodiment, processing the abnormal sequencing run with the detected time series discontinuities further includes using sequencing run time series data generated from an optical subsystem using the first and second LED illumination frequencies. The method includes detecting the time series discontinuities in a first time series from the first LED illumination frequency but not in a second time series from the second LED illumination frequency. The method includes generating a notification of a malfunction of a z-stage subcomponent used to precisely position the imaging device relative to the flow cell for display on a graphical user interface.

[0145] In one embodiment, the method includes detecting discontinuities in focus scores by applying a smoothing filter to a focus score time series for each image channel in the sequencing run. The method includes calculating a magnitude of the difference between smoothed focus scores in consecutive cycles in the focus score time series for each image channel in the sequencing run. The method includes detecting a number of consecutive cycles in which the magnitude of the difference between consecutive cycles exceeds a predefined threshold. The method includes generating a notification in response to detecting a malfunction of a z-stage subcomponent used to precisely position the imaging device relative to the flow cell. The notification may be displayed on a graphical user interface.

[0146] In one embodiment, processing the anomalous sequencing run with the detected time series drift further includes using sequencing run time series data generated from an optical subsystem using a first camera and a second camera. The first camera can detect two image channels and the second camera can detect two other image channels. The method includes detecting the drift in the time series corresponding to the two image channels from the first camera but not in the time series corresponding to the two other image channels from the second camera. The method includes generating a notification of the malfunction of the M3 subcomponent. The notification can be displayed on a graphical user interface.

[0147] In one embodiment, the method includes detecting drift in focus scores by applying a smoothing filter to a focus score time series for each image channel in the sequencing run. The method includes calculating differences between focus scores for each cycle across pairs of image channels in a plurality of image channels in the sequencing run. The method includes dividing the calculated differences into at least an early time series and a late time series for each pair of image channels. The method includes calculating, for each pair of image channels, an average difference value between the calculated differences for the early time series and the late time series for each pair of image channels. The method includes comparing, for a particular pair of image channels, the average difference value to average difference values ​​for other pairs of image channels. The method includes generating a notification of a malfunction of the M3 subcomponent when the average difference value for the particular pair of image channels is greater than the average difference values ​​for other pairs of image channels by a predetermined threshold.

[0148] In one embodiment, the method further includes detecting the expected lack of correlation by calculating a correlation between a focus score time series for a first specific image channel in the multiple image channels in a sequencing run for a first surface and a focus score time series for a second specific image channel in the multiple image channels in a sequencing run for a second surface. The first specific image channel and the second specific channel detect the same nucleotide in the sequencing run. The method includes detecting the expected lack of correlation between the two surfaces using the calculated correlation when the calculated correlation is below a predetermined threshold indicative of a compensator or z-stage subcomponent failure. Corresponding signal channels from the two surfaces are expected to have the same signal pattern, and when a compensator or z-stage subcomponent fails, the corresponding signal channels do not have a correlation, which may be referred to as an expected lack of correlation.

[0149] In one embodiment, the method includes detecting trends in focus scores by applying a smoothing filter to the focus score time series for each image channel in the sequencing run. The method includes partitioning the focus score time series into a predetermined number of partitions for each image channel. The method includes averaging focus scores in the first and last partitions. The method includes calculating a gradient between the last partition and the first partition of the focus score time series for each image channel using a difference between the averaged focus score values ​​between the first and last partitions and the number of focus score values ​​per partition. The method includes detecting trends such as whether a gradient between the last partition and the first partition of the focus score time series exceeds a predetermined threshold. The calculated features including trend features may be used to train a fault detection classifier. The trained fault detection classifier may then classify the signal channel as anomalous using the calculated features provided as input to the classifier. The classification and calculated features of the signal channel may be used by an expert fault localization rules engine to indicate a sub-component failure. Multiple expert rules may be applied in parallel or one after the other to identify a z-stage or compensator subcomponent as causing a malfunction.

[0150] In one embodiment, the method further includes identifying a root cause of the malfunction in the two or more sequencing runs prior to generating a notification of the subcomponent malfunction.

[0151] A sequencing run includes multiple cycles, where a cycle may include chemical treatment, image capture, and image processing actions.

[0152] The time series data can include a focus score for each image channel. Determining the focus score includes calculating a time series of focus scores based on an average FWHM per cycle of the flow cell from an average FWHM per image tile.

[0153] The time series data may include intensity values ​​for each image channel. Determining the intensity values ​​may include calculating a time series of intensity values ​​from an average of the intensity values ​​for each image tile based on an average intensity per cycle of the flow cell.

[0154] The notification of the particular subcomponent causing the system malfunction may include a description, including an identified possible defect in the particular subcomponent causing the system malfunction.

[0155] In one embodiment, receiving the classification of at least one sequencing run as an anomaly further includes indicating a system malfunction.

[0156] Root cause identification of a system malfunction to a specific subcomponent may indicate the particular subcomponent that needs adjustment or replacement among multiple replaceable subcomponents.

[0157] Malfunctioning of the z-stage sub-components used to precisely position the imaging device relative to the flow cell can result in poor focus quality of the captured image.

[0158] Some expert rules may be successfully executed without using anomaly classification from the fault detection classifier. For example, an expert rule for identifying the M3 subcomponent may use an output from a drift detector. If the output from the drift detector detects drift in at least one signal channel, the expert fault isolation rule engine may generate an alert for the M3 subcomponent. In one embodiment, if the output from the drift detector detects drift in all four signal channels, the expert fault isolation rule engine may generate an alert for the M3 subcomponent. Similarly, the expert fault isolation rule engine may use an output from an in-process vibration detector to identify a z-stage subcomponent malfunction. A second method embodiment of the present technology includes applying a preprocessor to time series data from at least one sequencing run. The preprocessor may detect time series anomalies. The anomalies may include discontinuities, drift, lack of expected correlation, and trends. The preprocessor may detect time series discontinuities, detect drift, detect lack of expected correlation, and detect trends. The method includes providing at least a portion of the output of the preprocessor to an expert rule system. The method includes receiving a root cause identification of a system malfunction to a particular subcomponent that requires adjustment or replacement, the subcomponent being among a plurality of replaceable subcomponents, and generating a notification of the particular subcomponent that is causing the system malfunction.

[0159] Other embodiments consistent with the methods may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform any of the methods described above. Yet another embodiment may include a system having a memory loaded with program instructions for performing any of the methods described above from a computer readable storage medium. The system may be loaded from either a transitory or a non-transitory computer readable storage medium.

[0160] Aspects of the disclosed technology may be embodied as a system including one or more processors coupled to a memory. The memory is loaded with computer instructions for diagnosing a system malfunction and identifying a cause of the system malfunction among a plurality of replaceable subcomponents. The system includes logic for applying a preprocessor to time series data from at least one sequencing run. The preprocessor can detect time series discontinuities, detect drift, detect lack of expected correlation, and it detects trends. The system includes logic for feeding at least a portion of the output of the preprocessor to a trained tree-based classifier for at least one image channel in one sequencing run. The preprocessor can process time series of two or four image channels. The system includes logic for receiving a classification of at least one sequencing run as an anomaly indicating a system malfunction. The system includes feeding at least a portion of the output of the preprocessor for the anomaly sequencing run to an expert rule system. The system includes receiving a root cause identification of the system malfunction to a particular subcomponent among the plurality of replaceable subcomponents that requires adjustment or replacement. The system includes logic that generates a notification of the particular subcomponent that is causing the system malfunction.

[0161] A second system embodiment of the present technology includes one or more processors coupled to a memory. The memory is loaded with computer instructions for diagnosing a system malfunction and identifying a cause of the system malfunction among a plurality of replaceable subcomponents. The system includes logic for applying a preprocessor to the time series data from at least one sequencing run. The preprocessor can detect time series discontinuities, detect drift, detect lack of expected correlation, which detects trends. The preprocessor can process two or four image channel time series. The system includes providing at least a portion of the output of the preprocessor to an expert rule system. The system includes receiving a root cause identification of the system malfunction to a particular subcomponent among the plurality of replaceable subcomponents that requires adjustment or replacement. The system includes logic for generating a notification of the particular subcomponent causing the system malfunction.

[0162] The computer-implemented system may incorporate any of the method features described immediately above or throughout this application that apply to the method implemented by the system. For the sake of brevity, alternative combinations of system features are not individually listed. Features applicable to systems, methods, and articles of manufacture are not repeated for each statutory classification set of basic features. The reader will understand how features identified in this section for one statutory class can be readily combined with basic features in other statutory classifications.

[0163] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform the functions of the above-described system. Yet other embodiments may include a method for performing the functions of the above-described system.

[0164] As an article of manufacture rather than a method, program instructions executable by a processor may be stored on a non-transitory computer readable medium (CRM). The program instructions, when executed, perform the computer-implemented methods described above. Alternatively, the program instructions may be stored on a non-transitory CRM and, when combined with appropriate hardware, may become one or more components of a computer-implemented system that practices the disclosed methods.

[0165] Each of the features discussed in this specific embodiment section for the method embodiment applies equally to the CRM and embodiment. As noted above, all method features are not repeated here and should be considered by reference.

[0166] Computer Systems 8 is a simplified block diagram of a computer system 800 that may be used to implement the disclosed techniques. The computer system typically includes at least one processor 872 that communicates with a number of peripheral devices via a bus subsystem 855. These peripheral devices may include, for example, a storage subsystem 810 that includes a memory subsystem 822 and a file storage subsystem 836, user interface input devices 838, user interface output devices 876, and a network interface subsystem 874. The input and output devices enable user interaction with the computer system. The network interface subsystem provides an interface to external networks, including interfaces to corresponding interface devices in other computer systems.

[0167] In one embodiment, the fault detection classifier 183 is communicatively linked to a storage subsystem and a user interface input device.

[0168] User interface input devices 838 can include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, scanners, touch screens integrated into displays, audio input devices such as voice recognition systems and microphones, as well as other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into a computer system.

[0169] The user interface output devices 876 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem can also provide non-visual displays such as an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system to a user or to another machine or computer system.

[0170] The storage subsystem 810 stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by the processor alone or in combination with other processors.

[0171] The memory used in the storage subsystem may include several memories, including a main random access memory (RAM) 832, for storing instructions and data during program execution, and a read only memory (ROM) 834 with fixed instructions stored therein. The file storage subsystem 836 provides persistent storage for program and data files and may include a hard disk drive, associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem in the storage subsystem or in other machines accessible by the processor.

[0172] The bus subsystem 855 provides a mechanism for allowing the various components and subsystems of the computer system to communicate with each other as intended. Although the bus subsystem is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0173] The computer system itself can be of various types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a mainframe, a server farm, a loosely distributed set of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the description of the computer system depicted in Figure 8 is intended only as a specific example for purposes of illustrating the disclosed technology. Many other configurations of computer systems are possible having more or fewer components than the computer system depicted in Figure 8.

[0174] The computer system 800 includes a GPU or FPGA 878. It may also include machine learning processors hosted by machine learning cloud platforms such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rackmount solutions such as the GX4 Rackmount Series, GX8 Rackmount Series, NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's Intelligent Processor Unit (IPU), Qualcomm's Zeroth platform with Snapdragon processors, NVIDIA's Volta, NVIDIA's DRIVE PX, NVIDIA's JETSON TX1 / TX2 MODULE, Intel's Nirvana, Movidius VPU, Fujitsu DPI, ARM's DynamicIQ, IBM TrueNorth, and others. [Explanation of symbols]

[0175] 1~4 Steps (Ticket) 5~7 Tickets 8GX 15 Maximum 30 Q 100 Systems 111 Sequencing System 114 Primary Metric Database 116 Features Database 130 Time Series Preprocessor 141 Discontinuity Detector 149 Databases 151 Drift Detector 155 Network 159 Ground Truth Fault Identification Database 161 Surface correlation detector 171 Trend Detector 183 Fault Detection Classifier 186 Expert Fault Identification Rule Engine 189 Alert Database 205 Optical Subsystem 210 LED Assembly 215 and 220 Light emitting diodes (LEDs) 225 Combiner Dichroic 230 Objective Lens 235 Beam Splitter 240 Compensator Subcomponents 250 z-stage subcomponents 260 Focus Mirror 270 Photo images 311 Schematic diagram 405 Graph 406 Focus Score Time Series 407 Image Intensity Time Series 431 Focus Score Time Series 436 Abnormal Signal Channel 460 Bottom 461 Focus Score Time Series 470 Top surface 471 Signal Channel 486 Focus Score Time Series 490 Intensity Value Time Series 492 Focus Score Time Series 494 and 498 Goods section 496 Middle part 514 Primary Metric Database 516 feature database 532 Green 601 Process Sequencing Run Process Step 620 Process Sequencing Run Process Step 671 and 673 Process Steps 710 Critical Alert 720 Warning Alert 730 Information (or info) alert 800 Computer Systems 810 Storage Subsystem 822 Memory Subsystem 832 Random Access Memory (RAM) 834 Dedicated memory (read only memory, ROM) 836 File Storage Subsystem 838 User Interface Input Devices 855 Bus Subsystem 872 processor 874 Network Interface Subsystem 876 User Interface Output Devices 878 FPGA

Claims

1. 1. A method for diagnosing a system malfunction and identifying a cause of the system malfunction among a plurality of replaceable subcomponents, the method comprising: applying a pre-processor to the time series data from at least one sequencing run to compute features that detect time series anomalies, the time series anomalies including at least one of a discontinuity, a drift, a lack of expected correlation, or a trend; providing at least a portion of the calculated features as an output of the pre-processor to a trained tree-based classifier for at least one image channel in a sequencing run, and receiving a classification of the at least one sequencing run as an anomaly; providing at least a portion of the calculated features of the anomaly sequencing runs and the classification to an expert rule system configured to apply a defined set of rules to the at least a portion of the calculated features of the anomaly sequencing runs or the classification to identify a root cause attribution of the system malfunction to a particular subcomponent of the plurality of replaceable subcomponents causing the time series anomaly; generating a notification of the particular subcomponent causing the system malfunction.

2. processing the abnormal sequencing run with the detected time series discontinuities; using the sequencing run time series data generated from an optical subsystem using first and second LED illumination frequencies; detecting the time series discontinuity in a first time series from the first LED illumination frequency but not in a second time series from the second LED illumination frequency; 10. The method of claim 1, further comprising: generating the notification of a malfunction of a z-stage subcomponent used to precisely position an imaging device relative to a flow cell for display on a graphical user interface.

3. Discontinuities in the focus score, applying a smoothing filter to the focus score time series for each image channel in the sequencing run; calculating, for each image channel in the sequencing run, a magnitude of the difference between smoothed focus scores for successive cycles in the focus score time series; detecting a number of consecutive cycles in which the magnitude of the difference between consecutive cycles exceeds a predefined threshold; The method of claim 1 or 2, further comprising: generating, in response to the detection, a notification of a malfunction of a z-stage subcomponent used to precisely position an imaging device relative to a flow cell for display on a graphical user interface.

4. processing the anomalous sequencing runs with detected time series drift; using time series data of the sequencing run generated from an optical subsystem using a first camera detecting two image channels and a second camera detecting two other image channels; detecting the drift in time series corresponding to the two image channels from the first camera, but not in time series corresponding to the two other image channels from the second camera; The method according to any one of claims 1 to 3, further comprising: generating said notification of a malfunction of an M3 subcomponent.

5. Drift in focus score, applying a smoothing filter to the focus score time series for each image channel in the sequencing run; calculating a difference between focus scores for each cycle across pairs of image channels within a plurality of image channels in the sequencing run; splitting the calculated differences into at least an early time series and a late time series for each pair of image channels; calculating an average difference value between the calculated differences of the first half time series and the second half time series for each pair of image channels for the pair of image channels, and comparing the average difference value for a particular pair of image channels to the average difference values ​​for other pairs of image channels; and generating the notification of a malfunction of an M3 subcomponent when the average difference value for the particular image channel pair is greater than the average difference values ​​for other image channel pairs by a predetermined threshold.

6. The expected lack of correlation, calculating a correlation between a focus score time series for a first particular image channel within a plurality of image channels in the sequencing run for a first surface and the focus score time series for a second particular image channel within a plurality of image channels in the sequencing run for a second surface, wherein the first particular image channel and the second particular channel detect the same nucleotide in the sequencing run; 6. The method of claim 1, further comprising: detecting a lack of the expected correlation between two surfaces using the calculated correlation when the calculated correlation is below a predetermined threshold indicative of a compensator sub-component failure.

7. Trends within the Focus Score applying a smoothing filter to the focus score time series for each image channel in the sequencing run; partitioning the focus score time series into a predetermined number of partitions for each image channel; averaging the focus scores in the first partition and the last partition; calculating, for each image channel, a gradient between the last partition and the first partition of the focus score time series using the difference between the averaged focus score values ​​between the first partition and the last partition and a number of focus score values ​​per partition; 7. The method of claim 1, further comprising: detecting the trend indicative of a z-stage failure or a compensator failure if the gradient between the last partition and the first partition of the focus score time series exceeds a predetermined threshold.

8. The method of claim 1 , further comprising identifying the root cause of a malfunction in two or more sequencing runs prior to generating the notification of a sub-component malfunction.

9. 9. The method of claim 1, wherein the sequencing run comprises a plurality of cycles, a cycle of the plurality of cycles comprising chemical treatment, image capture, and image processing actions.

10. The time series data includes a focus score for each image channel, and determining the focus score includes:

10. The method of claim 9, comprising calculating a time series of focus scores based on an average FWHM per cycle of the flow cell from an average FWHM per image tile.

11. The time series data includes, for each image channel, intensity values, and determining the intensity values ​​comprises:

10. The method of claim 9, comprising calculating a time series of intensity values ​​based on an average intensity per cycle of the flow cell from the average intensity values ​​per image tile.

12. 6. The method of claim 1, wherein the notification of the particular subcomponent causing the system malfunction includes a description including an identified possible defect in the particular subcomponent causing the system malfunction.

13. The method of claim 1 , wherein receiving the classification of the at least one sequencing run as an anomaly further comprises indicating a system malfunction.

14. The method of claim 1 , wherein the notification indicates that the particular subcomponent among the plurality of replaceable subcomponents requires adjustment or replacement.

15. The method of claim 1 or 2, wherein the malfunction of the z-stage sub-component used to precisely position an imaging device relative to a flow cell causes the discontinuity.

16. 1. A system including one or more processors coupled to a memory, the memory being loaded with computer instructions for diagnosing a system malfunction and identifying a cause of the system malfunction among a plurality of replaceable subcomponents, the instructions, when executed on the processor, comprising: applying a pre-processor to the time series data from at least one sequencing run to compute features that detect time series anomalies, the time series anomalies including at least one of a discontinuity, a drift, a lack of expected correlation, or a trend; providing at least a portion of the calculated features as an output of the pre-processor to a trained tree-based classifier for at least one image channel in a sequencing run, and receiving a classification of the at least one sequencing run as an anomaly; providing at least a portion of the calculated features of the anomaly sequencing runs and the classification to an expert rule system configured to apply a defined set of rules to the at least a portion of the calculated features of the anomaly sequencing runs or the classification to identify a root cause attribution of the system malfunction to a particular subcomponent of the plurality of replaceable subcomponents causing the time series anomaly; generating a notification of the particular subcomponent causing the system malfunction.

17. processing the abnormal sequencing run with the detected time series discontinuities; using the sequencing run time series data generated from an optical subsystem using first and second LED illumination frequencies; detecting the time series discontinuity in a first time series from the first LED illumination frequency but not in a second time series from the second LED illumination frequency; The system of claim 16 , further performing an action including: generating the notification of a malfunction of a z-stage subcomponent used to precisely position an imaging device relative to a flow cell for display on a graphical user interface.

18. Discontinuities in the focus score, applying a smoothing filter to the focus score time series for each image channel in the sequencing run; calculating, for each image channel in the sequencing run, a magnitude of the difference between smoothed focus scores for successive cycles in the focus score time series; detecting a number of consecutive cycles in which the magnitude of the difference between consecutive cycles exceeds a predefined threshold; The system of claim 16 or 17, further performing actions including detecting by: generating a notification of a malfunction of a z-stage sub-component used to precisely position the imaging device relative to the flow cell in response to the detection for display on a graphical user interface.

19. processing the anomalous sequencing runs with detected time series drift; using time series data of the sequencing run generated from an optical subsystem using a first camera detecting two image channels and a second camera detecting two other image channels; detecting the drift in time series corresponding to the two image channels from the first camera, but not in time series corresponding to the two other image channels from the second camera; The system of any one of claims 16 to 18, further performing actions including: reporting a malfunction of the M3 subcomponent.

20. Drift in focus score, applying a smoothing filter to the focus score time series for each image channel in the sequencing run; calculating a difference between focus scores for each cycle across pairs of image channels within a plurality of image channels in the sequencing run; splitting the calculated differences into at least an early time series and a late time series for each pair of image channels; calculating an average difference value between the calculated differences of the first half time series and the second half time series for each pair of image channels for the pair of image channels, and comparing the average difference value for a particular pair of image channels to the average difference values ​​for other pairs of image channels; The system of any one of claims 16 to 19, further performing an action including detecting by reporting a malfunction of an M3 subcomponent that focuses on multiple tiles when the average difference value for the particular image channel pair is greater than the average difference value for other image channel pairs by a predetermined threshold.

21. The expected lack of correlation, calculating a correlation between a focus score time series for a first particular image channel within a plurality of image channels in the sequencing run for a first surface and the focus score time series for a second particular image channel within a plurality of image channels in the sequencing run for a second surface, wherein the first particular image channel and the second particular channel detect the same nucleotide in the sequencing run; 21. The system of claim 16, further comprising: detecting a lack of the expected correlation between two surfaces using the calculated correlation when the calculated correlation is below a predetermined threshold indicative of a compensator sub-component failure; and detecting a lack of the expected correlation between two surfaces using the calculated correlation when the calculated correlation is below a predetermined threshold indicative of a compensator sub-component failure.

22. Trends within the Focus Score applying a smoothing filter to the focus score time series for each image channel in the sequencing run; partitioning the focus score time series into a predetermined number of partitions for each image channel; averaging the focus scores in the first partition and the last partition; calculating, for each image channel, a gradient between the last partition and the first partition of the focus score time series using the difference between the averaged focus score values ​​between the first partition and the last partition and a number of focus score values ​​per partition; 22. The system of claim 16, further comprising: detecting the trend indicative of a z-stage failure or a compensator failure if the gradient between the last partition and the first partition of the focus score time series exceeds a predetermined threshold.

23. The system of any one of claims 16 to 22, further performing an action including identifying the root cause of a malfunction in two or more sequencing runs prior to generating the notification of a sub-component malfunction.

24. The system of any one of claims 16 to 23, wherein the sequencing run comprises a plurality of cycles, a cycle of the plurality of cycles comprising chemical treatment, image capture, and image processing actions.

25. The time series data includes a focus score for each image channel, and determining the focus score includes:

25. The system of claim 24, further performing the action of calculating a time series of focus scores based on an average FWHM per cycle of the flow cell from the average FWHM per image tile.

26. The time series data includes, for each image channel, intensity values, and determining the intensity values ​​comprises:

25. The system of claim 24, further performing the action of calculating a time series of intensity values ​​based on an average intensity per cycle of the flow cell from the average intensity values ​​per image tile.

27. 21. The system of claim 16, wherein the notification of the particular subcomponent causing the system malfunction includes a description including an identified, possible defect in the particular subcomponent causing the system malfunction.

28. 20. The system of claim 16, wherein receiving the classification of the at least one sequencing run as an anomaly further performs an action including indicating a system malfunction.

29. The system of claim 16 , wherein the notification indicates that the particular subcomponent among the plurality of replaceable subcomponents is in need of adjustment or replacement.

30. 18. The system of claim 16 or 17, wherein the malfunction of the z-stage sub-component used to precisely position an imaging device relative to a flow cell causes the discontinuity.

31. 1. A non-transitory computer readable storage medium having stored thereon computer program instructions for diagnosing a system malfunction and identifying a cause of the system malfunction from among a plurality of replaceable subcomponents, the instructions, when executed on a processor, comprising: applying a pre-processor to the time series data from at least one sequencing run to compute features that detect time series anomalies, the time series anomalies including at least one of a discontinuity, a drift, a lack of expected correlation, or a trend; providing at least a portion of the calculated features as an output of the pre-processor to a trained tree-based classifier for at least one image channel in a sequencing run, and receiving a classification of the at least one sequencing run as an anomaly; providing at least a portion of the calculated features of the anomaly sequencing runs and the classification to an expert rule system configured to apply a defined set of rules to the at least a portion of the calculated features of the anomaly sequencing runs or the classification to identify a root cause attribution of the system malfunction to a particular subcomponent of the plurality of replaceable subcomponents causing the time series anomaly; and generating a notification of the particular subcomponent causing the system malfunction.

32. processing the abnormal sequencing run with the detected time series discontinuities; using the sequencing run time series data generated from an optical subsystem using first and second LED illumination frequencies; detecting the time series discontinuity in a first time series from the first LED illumination frequency but not in a second time series from the second LED illumination frequency; 32. The non-transitory computer-readable storage medium of claim 31 , further comprising: generating the notification of a malfunction of a z-stage sub-component used to precisely position an imaging device relative to a flow cell for display on a graphical user interface.

33. Discontinuities in the focus score, applying a smoothing filter to the focus score time series for each image channel in the sequencing run; calculating, for each image channel in the sequencing run, a magnitude of the difference between smoothed focus scores for successive cycles in the focus score time series; detecting a number of consecutive cycles in which the magnitude of the difference between consecutive cycles exceeds a predefined threshold; 33. The non-transitory computer-readable storage medium of claim 31 or 32, further comprising: detecting by: generating, in response to the detection, the notification of a malfunction of a z-stage sub-component used to precisely position an imaging device relative to a flow cell for display on a graphical user interface.

34. Trends within the Focus Score applying a smoothing filter to the focus score time series for each image channel in the sequencing run; partitioning the focus score time series into a predetermined number of partitions for each image channel; averaging the focus scores in the first partition and the last partition; calculating, for each image channel, a gradient between the last partition and the first partition of the focus score time series using the difference between the averaged focus score values ​​between the first partition and the last partition and a number of focus score values ​​per partition; Detecting the trend indicative of a z-stage failure or a compensator failure if the gradient between the last partition and the first partition of the focus score time series is above a predetermined threshold.

35. 32. The non-transitory computer-readable storage medium of claim 31, wherein the notification indicates that the particular subcomponent among the plurality of replaceable subcomponents requires adjustment or replacement.

36. 33. The non-transitory computer readable storage medium of claim 31 or 32, wherein the malfunction of the z-stage sub-component used to precisely position an imaging device relative to a flow cell causes the discontinuity.

Images