Error recovery system for substrate processing systems

Abstract

An error recovery system for a substrate processing tool includes a processor and non-transitory memory storing instructions. The system displays a recovery assistant button on a GUI that displays an error in the substrate processing tool; and in response to activation of the recovery assistant button, retrieves data relevant to the error from a database; generates a diagnostic report indicating a root cause of the error by analyzing the data; displays the diagnostic report indicating the root cause of the error on the GUI; generates a set of steps to recover from the error based on the root cause; display the set of steps on the GUI; executes at least portions of the set of steps automatically, interactively, or using a combination thereof on the GUI; and executes a recovery option from a plurality of recovery options to place the substrate processing tool in normal operating condition.

Description

ERROR RECOVERY SYSTEM FOR SUBSTRATE PROCESSING SYSTEMSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 675,769 filed on July 26, 2024. The entire disclosure of the above application is incorporated herein by reference.FIELD

[0002] The present disclosure relates generally to substrate processing systems and more particularly to an error recovery system for substrate processing systems.BACKGROUND

[0003] The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0004] A substrate processing system (also called a tool) typically comprises one or more processing chambers. Each processing chamber comprises a plurality of processing modules in which processes such as deposition, etching, and other treatments are performed on substrates such as semiconductor wafers. Examples of processes that may be performed on a substrate comprise a chemical vapor deposition (CVD) process, a chemically enhanced plasma vapor deposition (CEPVD) process, a plasma enhanced chemical vapor deposition (PECVD) process, a sputtering physical vapor deposition (PVD) process, atomic layer deposition (ALD), and plasma enhanced ALD (PEALD). Additional examples of processes that may be performed on a substrate comprise etching (e.g., chemical etching, plasma etching, reactive ion etching, etc.) and cleaning processes.

[0005] During processing, a substrate is arranged on a substrate support such as a pedestal in a station. During deposition, gas mixtures comprising one or more precursors are introduced into the station, and plasma may be optionally struck to activate chemical reactions. During etching, gas mixtures comprising etch gases are introduced into the station, and plasma may be optionally struck to activate chemical reactions. A computer-controlled robot typically transfers substrates from one process module to another in a sequence in which the substrates are to be processed.SUMMARY

[0006] An error recovery system for a substrate processing tool comprises a processor and non-transitory memory storing instructions configured to cause the processor to display a recovery assistant button on a graphical user interface (GUI). The GUI is configured to display an error in the substrate processing tool. The instructions are configured to cause the processor to, in response to activation of the recovery assistant button, retrieve data relevant to the error from a database; generate a diagnostic report indicating a root cause of the error by analyzing the data; display the diagnostic report indicating the root cause of the error on the GUI; generate a set of steps to recover from the error based on the root cause; display the set of steps on the GUI; execute at least portions of the set of steps automatically, interactively, or using a combination thereof on the GUI; and execute a recovery option from a plurality of recovery options to place the substrate processing tool in normal operating condition.

[0007] In additional features, the GUI is configured to display the error on a first page of the GUI, and wherein the instructions are configured to cause the processor to display the recovery assistant button on the first page of the GUI, display the diagnostic report on a second page of the GUI, display the set of steps on the second page of the GUI, and execute the set of steps on the second page of the GUI.

[0008] In additional features, the instructions are configured to cause the processor to automatically execute the recovery option.

[0009] In additional features, the GUI is configured to display the plurality of recovery options, and upon executing the set of steps on the GUI, the recovery option is interactively selectable for execution on the GUI.

[0010] In additional features, the GUI is configured to display the plurality of recovery options on the first page of the GUI, and upon executing the set of steps on the second page of the GUI, the recovery option is interactively selectable for execution on the first page of the GUI.

[0011] In additional features, the instructions are configured to cause the processor to determine at least one of (i) an amount of information to display in the diagnostic reportand (ii) whether to execute the steps automatically or interactively based on a type of user using the GUI.

[0012] In additional features, the instructions are configured to cause the processor to execute the set of steps by switching between executing the steps automatically and interactively.

[0013] In additional features, the instructions are configured to cause the processor to, during interactive execution of a step in the set of the steps, display information on the GUI and receive input through the GUI.

[0014] In additional features, the instructions are configured to cause the processor to, during interactive execution of a step in the set of the steps, display information on the second page of the GUI and receive input through the second page of the GUI.

[0015] In additional features, the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, automatically perform (i) one or more operations of the step, (ii) access and process relevant data from the database, and (iii) display information about the step adjacent to the step on the GUI.

[0016] In additional features, the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, while on the second page of the GUI, automatically perform (i) one or more operations of the step, (ii) access and process relevant data from the database, and (iii) display information about the step adjacent to the step on the second page of the GUI.

[0017] In additional features, the one or more operations comprise performing a diagnostic procedure on a component of the substrate processing tool.

[0018] In additional features, the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, interactively display information on the GUI, receive input through the GUI, and resume the automatic execution of the step on the GUI in response to receiving the input.

[0019] In additional features, the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, interactively display information on the second page of the GUI, receive input through the second page of the GUI, and resume the automatic execution of the step on the second page of the GUI in response to receiving the input.

[0020] In additional features, the instructions are configured to cause the processor to interactively allow at least one of adding, deleting, replacing, and omitting one or more steps in the set of steps.

[0021] In additional features, the instructions are configured to cause the processor to automatically at least one of add, delete, replace, and omit one or more steps in the set of steps during the execution of the set of the steps.

[0022] In additional features, the instructions are configured to cause the processor to select the data relevant to the error from at least one of an error log and diagnostic data stored in the database based on a correlation of the data with the error.

[0023] In additional features, the instructions are configured to cause the processor to convert a data format of at least one of the error log and the diagnostic data retrieved from the database into a format adapted for ingesting and processing the data to generate the diagnostic report and the set of steps.

[0024] In additional features, the instructions are configured to cause the processor to generate a portion of the diagnostic data by performing a diagnostic procedure on a component of the substrate processing tool after the error.

[0025] In additional features, the instructions are configured to cause the processor to formulate one or more hypotheses for determining the root cause of the error by correlating the error to the data retrieved from the database, and generate the diagnostic report comprising the one or more hypotheses.

[0026] In additional features, each of the hypotheses comprises information about a component related to the error and a corrective action to recover from the error.

[0027] In additional features, the hypotheses are independent of each other to eliminate redundant checking of inter-related or overlapping causes of the error.

[0028] In additional features, the instructions are configured to cause the processor to, using a set of rules designed for the error, analyze the error, collect and process relevant data based on the analysis, and formulate and filter the hypotheses to determine the root cause of the error based on the processing.

[0029] In additional features, the instructions are configured to cause the processor to test each of the hypotheses based on relevant data, filter a first hypothesis from the hypotheses that indicates a less likely cause for the error, and select a second hypothesisfrom the hypotheses that indicates a most likely cause for the error as a hypothesis indicating the root cause of the error.

[0030] In additional features, the instructions are configured to cause the processor to filter the first hypothesis by performing a diagnostic procedure on a component indicated by the first hypothesis as being related to the error.

[0031] In additional features, the instructions are configured to cause the processor to filter the first hypothesis in response to determining that a component indicated by the first hypothesis as being related to the error has not caused another error similar to the error.

[0032] In additional features, the instructions are configured to cause the processor to confirm that the second hypothesis indicates the root cause of the error by identifying correlations in historical data of a component indicated by the second hypothesis as being related to the error.

[0033] Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

[0035] FIG. 1 shows an example of a substrate processing system comprising a controller that provides a graphical user interface (GUI) for interacting with subsystems of substrate processing system;

[0036] FIGS. 2A-2F show an example of using a typical GUI to resolve an error in the substrate processing system;

[0037] FIGS. 3A and 3B show an example of an interface provided on the GUI by an error recovery module embedded in the controller according to the present disclosure;

[0038] FIGS. 4 and 5A-5C show an example of using the GUI comprising the interface of FIGS. 3A and 3B to resolve errors in the substrate processing system; and

[0039] FIGS. 6A and 6B show a flowchart of a method employed by the error recovery module to provide the interface of FIGS. 3A and 3B to resolve errors in the substrate processing system.

[0040] In the drawings, reference numbers may be reused to identify similar and / or identical elements.DETAILED DESCRIPTION

[0041] A substrate processing system (called a tool) typically comprises multiple process modules (PMs) in which substrates can be processed. For example, the substrates can be processed using various deposition and etch processes. The tool comprises one or more computer-controlled robots that can transfer substrates in and out of the tool and between the PMs. The tool comprises gas boxes to supply various gases to the PMs. The tool comprises pumps to exhaust residual material from the PMs and to maintain pressure (e.g., vacuum) in the PMs. The tool comprises cooling systems to cool components such as pedestals and showerheads in the PMs. The tool comprises power supplies that supply radio frequency (RF) and direct current (DC) power to various components of the tool and the PMs. The tool comprises many sensors such as pressure sensors, temperature sensors, optical sensors, cameras, and so on. The sensors are installed at various locations in the tool and in the PMs. The tool comprises a system controller that controls various operations performed by the components of the tool and the PMs based on inputs received from the sensors. In general, the robots, the gas boxes, the exhaust system, the cooling system, the power supplies, and so on, together with the associated sensors can be called respective subsystems of the tool.

[0042] The system controller comprises diagnostics that can detect faults in one or more subsystems of the tool. The diagnostics can be executed during runtime of the tool (e.g., during substrate processing, cleaning processes, and so on) and during preventive maintenance. The diagnostics can interpret data received from the sensors and from the outputs received from the subsystems (e.g., during tool powerup, substrate processing, and cleaning of the PMs). On detecting a fault condition based on the data, the diagnostics can output an alarm on a graphical user interface (GUI) provided on a display coupled to the system controller. The alarm may indicate a fault condition occurring in the tool. The diagnostics can also output options on the GUI from which a field service engineer (FSE) can select an option to resolve the fault condition.

[0043] Typically, the alarm comprises a lot of data associated with the fault condition. The data in the alarm can be undecipherable, insufficient, or both for the FSE to make an informed decision about selecting the correct option to resolve the fault condition. Selecting an incorrect option can cause the tool to enter into an undesirable state, which can disable or damage a component that may not be faulty. The data in the alarm may include an error code along with related data that requires further analysis to find the root cause of the error code and the alarm. Accordingly, fully understanding and interpreting the data in the alarm and deciding exactly how to respond can be difficult for the FSE.

[0044] Further, the data in the alarm alone may be insufficient to identify the root cause of the error code and the alarm. Additional information not included in the alarm may be needed to determine the root cause of the error code and the alarm. The alarm typically does not provide pointers to the additional information that may be needed to determine the root cause of the error code and the alarm. Accordingly, the FSE may not know or may be unable to decide what additional information is needed to select the correct option from the options provided on the GUI of the system controller. Furthermore, after identifying the root cause for the alarms and before selecting the correct option from the options provided on the GUI of the system controller, the FSE may need to perform one or more corrective actions to put the tool in an operable condition. The FSE may not know the exact steps to perform and / or how to perform the steps.

[0045] The present disclosure provides an error recovery system (called a recovery assistant) that is built into the system controller and that assists the FSE in determining the root cause of the alarm and in correctly selecting a response to the alarm. The recovery assistant augments the diagnostics by analyzing the data in the alarm and identifying any additional information needed by the FSE to select the correct response to the alarm. The recovery assistant helps the FSE understand the data in the alarm and leads the FSE to any additional information needed to correctly respond to the alarm. The recovery assistant reads the data in the alarm and establishes a series of steps that are easy to follow for the FSE without requiring the FSE to interpret and analyze the data in the alarm. Instead, the recovery assistant handles the interpretation and analysis of the data in the alarm. The recovery assistant walks the FSE through the steps.

[0046] The recovery assistant can perform some of the steps automatically without any input or intervention from the FSE. The recovery assistant can also perform some of the steps interactively by interacting with the FSE. For example, in a step, the recoveryassistant can output additional data on the display of the system controller and ask the FSE to respond to a question requiring a simple yes or no answer. The recovery assistant can receive an input from the FSE in response to the additional displayed data. The recovery assistant can select the next step based on the response from the FSE. Accordingly, in one of the steps, the recovery assistant can display additional data, receive an input from the FSE based on the additional data, and select the next step.

[0047] For example, the additional data may include an image captured by a camera in the tool and a question about the image, and the input form the FSE may include a simple yes or no answer. In other examples, the question and answer may be about selecting whether an operation required in a step should be performed automatically by the tool controller or should be performed manually by the FSE. In another example, the question and answer may be about deciding whether the subsystem being analyzed should be allowed to operate while being analyzed or should be isolated from the rest of the tool. The steps are executed until the root cause of the alarm is determined and until the tool is in a state to continue normal operation. Having determined the root cause, the FSE can confidently select the correct option displayed on the GUI of the system controller.

[0048] In some instances, before selecting the correct option, the FSE may need to perform a corrective action to put the tool in an operable condition. The recovery assistance provides the information needed to perform the action. The recovery assistant provides step-by-step instructions for performing the action. After the action is performed, with the tool in an operable state and having determined the root cause, the FSE can confidently select the correct option displayed on the GUI of the system controller. Selecting the correct option resets the alarm and the tool resumes normal operation.

[0049] Thus, the recovery assistant not only interprets the data in the alarm but also leads the FSE to any other information that may be needed and walks the FSE through steps to isolate the root cause for the alarm. Accordingly, the FSE can make an informed decision and select the correct option displayed on the GUI of the system controller to resolve the fault condition indicated in the alarm. The recovery assistant also assists the FSE in performing any corrective action needed before selecting the correct option in response to the alarm.

[0050] A selectable option (e.g., a button) is added to the alarm page on the GUI to allow the FSE to navigate from the alarm page to the recovery assistant. The recovery assistant is displayed on a page on the GUI that is separate and different from the pageon which the alarm is displayed on the GUI. The information for the alarm (e.g., alarm text, alarm issuer, recovery options, etc.) is copied from the alarm page to the recovery assistant page. Once on the recovery assistant page, the recovery assistant performs all the operations including providing data to the FSE, the question and answer (Q&A) with the FSE, instructions for performing any corrective action, and eventually allowing the FSE to select the correct recovery option to respond to the alarm, all on the same recovery assistant page. Thus, the FSE does not need to leave the recovery assistant page and does not need to explore what information to obtain and where to obtain the information throughout the diagnostic and recovery procedure performed on the recovery assistant page on the GUI. The recovery assistant provides all the information for fixing the fault on a single page.

[0051] The recovery assistant can coach or train the FSE through recovery steps as well as diagnostic steps as follows. The FSE selects an alarm on the alarm page on the GUI and starts the recovery assistant. On the recovery assistant page on the GUI, the recovery assistant automates diagnostic steps (e.g., compares outputs of sensors) and / or coaches the FSE through the diagnostic steps (e.g., displays a camera image and asks the FSE a yes / no question). The recovery assistant determines the root cause for the alarm. The FSE may need to perform actions to put the tool into an operable state before selecting a recovery option on the alarm page. For example, in an active wafer centering (AWC) procedure, the FSE may need to vent a vacuum transfer module (VTM) or move the wafer by hand to center the wafer on the end effector of the robot and then pump down the VTM. The recovery assistant guides the FSE through these actions. For example, the recovery assistant provides instructions to the FSE to begin VTM venting from within the recovery assistant rather than the FSE having to go to a different page on the GUI. The recovery assistant also shows the FSE instructions for centering the wafer by hand. After thus placing the tool is an operable state, the FSE selects the correct recovery option on the alarm page (e.g., continue moving the wafer), which clears the error and starts the tool operating normally again.

[0052] Thus, the recovery assistant simplifies the workflow of the FSE by analyzing the data in the alarm, by performing some of the diagnostic stems automatically, by walking the FSE through diagnostic and recovery steps, and by providing step-by-step instructions to the FSE. All these steps are performed within the recovery assistant itself to determine the root cause for the alarm, to place the tool in an operable state beforeselecting a recovery option, and to select the correct recovery option for the tool to resume normal operation.

[0053] The present disclosure is not limited to a system for helping FSEs troubleshoot the tool. Rather, additionally, the present disclosure provides an improved error recovery module (called the recovery assistant) for semiconductor manufacturing equipment (called a tool). Specifically, the recovery assistant comprises a frontend and a backend. The frontend provides a novel information display given the limitations of the current GUI as described below. Additionally, the backend provides novel solution modeling that generates more accurate resolution of alarms given the numerous related / un related error codes that occur in the tools.

[0054] More specifically, the frontend simplifies the GUI by providing all diagnostic and recovery procedures on a single page of the GUI (called the recovery assistant page), which eliminates the need for the FSE to navigate through multiple pages of the GUI to recover from an alarm. The backend performs the diagnostic and recovery procedures by analyzing the alarm data, providing a diagnostics report for the alarm including hypothesized potential solutions to determine the root cause of the alarm, and providing steps customized to recover from the alarm, all on a single page of the GUI.

[0055] The backend collects data from various sources (e.g., sensors, current and past diagnostics and error logs, past solutions to the current alarm and related alarms, etc.) to provide the diagnostics report and to formulate the customized recovery steps. For some alarms, the diagnostics report alone may be sufficient for selecting a correct recovery option to recover from the alarm. For other alarms, in addition to providing the diagnostics report and the root cause of the alarm, the backend formulates the recovery steps to recover from the alarm. The backend guides the FSE to select a correct recovery option to recover from the alarm, after which the tool resumes normal operation. These and other features of the error recovery module of the present disclosure are described below in detail.

[0056] The present disclosure is organized as follows. An example of a tool is shown and described with reference to FIG. 1. A series of actions and operations that are typically performed using a GUI by an operator of the tool or an FSE to recover from an alarm without the recovery assistant of the present disclosure are shown and described with reference to FIGS. 2A-2F. These figures illustrate the difficulties involved and encountered by the operator or the FSE in the process of recovering from the alarm. Therecovery assistant of the present disclosure, which alleviates these difficulties, is shown and described with reference to subsequent figures. Specifically, the novel features of the GUI (the frontend), which simplify the process of recovering from the alarm, are shown and described with reference to FIGS. 3A-5C. The solution modeling (the backend) of the recovery assistant that provides the novel features of the GUI is shown and described with reference to FIGS. 6A and 6B.

[0057] Throughout the following description, an alarm caused by an error in an active wafer centering (AWC) procedure performed by a wafer transfer robot is used as an example. For example, the alarm indicates an error associated with the wafer transfer robot. However, the recovery assistant of the present disclosure is not limited to the described example. Rather, the recovery assistant of the present disclosure can simplify recovering from any alarm that includes an error related to any subsystem of a tool. An example of the tool and its subsystems is described below.EXAMPLE OF A TOOL

[0058] FIG. 1 shows an example of a tool 100. For example, the tool 100 comprises a plurality of process modules (PMs) 102-1 , 102-2, ..., 102-8 (collectively called the PMs 102 and individually called the PM 102). While eight PMs 102 are shown, the tool 100 can comprise any number of PMs 102. Further, while the PMs 102 are shown arranged in two rows, the PMs 102 can be arranged in other ways (e.g., in a circular manner).

[0059] The tool 100 comprises various subsystems. For example, a gas box, one or more power supplies (e.g., RF and DC power supplies), and one or more cooling assemblies are provided for each PM 102. A plurality of gas boxes, power supplies, and cooling systems are generally shown at 1 10-1 , 1 10-2, ..., 110-8 for the PMs 102, respectively. The gas boxes supply various gases (e.g., process gases, purge gases, cleaning gases, etc.) to the PMs 102 through respective gas distribution devices such as showerheads for performing various processes such as deposition, etching, and cleaning in the PMs 102. The power supplies supply RF power (e.g., to showerheads or coils of the PMs 102) to strike plasma in the PMs 102 when plasma is used in the processes performed in the PMs 102. The power supplies also supply DC power to components such as heaters and clamping electrodes used in the pedestals in the PMs 102. The cooling systems supply one or more coolants to different components of the tool 100 (e.g., to showerheads and pedestals in the PMs 102, around the power supplies, and so on).

[0060] A vacuum transfer module (VTM) 104 comprises one or more robots to transfer substrates into and out of the tool 100 and between the PMs 102. For example, two robots R1 and R2 are shown. The VTM 104 is arranged between the PMs 102 such that the robots can transfer the substrates to and from all the PMs 102. When two or more robots are used, a buffer 108 is arranged between the robots to facilitate and coordinate the transfer of substrates between the robots. The VTM 104 comprises a plurality of cameras (not shown) to monitor the robots and wafer transfers.

[0061] Airlocks 120-1 , 120-2 are provided at each end of the VTM 104. Substrates may be loaded into the tool 100 via a load port module (LPM) 130 comprising one or more front opening unified pods (FOUPs) 132. The substrates are then transported through an equipment front end module (EFEM) 134 and the airlock 120-1 to the PMs 102 using the VTM 104. The tool 100 also comprises a power lock out and tag out (LOTO) assembly 138 that distributes power to various components of the tool 100 and provide LOTO functions.

[0062] The tool 100 also comprises an exhaust subsystem (not shown). The exhaust subsystem is coupled to the PMs 102. The exhaust subsystem comprises one or more pumps that evacuate reaction byproducts from the PMs 102. The tool 100 further comprises one or more pumps (not shown) that maintain vacuum in the PMs 102 and the VTM 104.

[0063] The subsystems of the tool 100 described above comprise many sensors. For example, the PMs 102 comprise temperature sensors, pressure sensors, optical sensors (e.g., dynamic alignment or DA sensors). For example, the temperature sensors are disposed in showerheads and pedestals in the PMs 102. The gas boxes comprise flow sensors and valves to control flow of gases supplied by the gas boxes to the PMs 102. The power supplies comprise voltage and current sensors. The cooling systems comprise flow sensors, valves, and leak sensors. The VTM 104 comprises various sensors and motors associated with the robots in addition to the cameras.

[0064] The tool 100 comprises a system controller 150. The system controller 150 communicates with and controls the subsystems and other components of the tool 100 described above. The system controller 150 controls the subsystems and other components based on feedback received from the sensors. The system controller 150 can also communicate with one or more servers in a cloud (not shown) via suitable networks. The system controller 150 can upload data from the tool 100 to the one or moreservers and can download data (e.g., updates to the recovery assistant) from the one or more servers. Similarly, other tools can also exchange data with the one or more servers. For example, the data can comprise diagnostic data, performance data, and so on of the tools. The data can be used to build and update the recovery assistant as described below.

[0065] The system controller 150 is connected to a display 152. For example, the display 152 comprises a touch-screen. The system controller 150 comprises an error recovery module (also called the recovery assistant) 160 of the present disclosure. The recovery assistant 160 is embedded in the system controller 150. Accordingly, the recovery assistant 160 can access all the subsystems, sensors, and other components of the tool 100 described above via the system controller 150. The system controller 150 presents information (e.g., the alarm page, the recovery assistant page, etc.) on a GUI 170 on the display 152 as described below.

[0066] All examples of GUI described below are presented on the GUI 170 on the display 152. In the description of the recovery assistant below (description of FIGS. 3A onwards), the recovery assistant option (button) on the GUI 170 is shown as element 300. When the recovery assistant 300 is selected on the GU1 170, the recovery assistant 160 embedded in the system controller 150 is activated. When activated, the recovery assistant 160 opens a recovery assistant page on the GUI 170 and performs the operations of the recovery assistant 300 as described below in detail. Before describing the recovery assistant 300, a typical alarm recovery procedure performed using the GUI 170 without the recovery assistant 300 is described.TYPICAL ALARM RECOVERY PROCEDURE

[0067] FIGS. 2A-2F schematically show examples of screens of a GUI displayed on the display 152 of the system controller 150 without the recovery assistant 300. The figures show the GUI comprising a series of actions and operations that are typically performed by an operator of the tool or an FSE to recover from an alarm without using the recovery assistant 300. The figures show examples of pages of the GUI that the FSE needs to navigate after reading the alarm text on the alarm page and before correctly selecting a recovery option on the alarm page. The figures illustrate the difficulties involved and encountered by the operator or the FSE in the process of recovering from the alarm without the recovery assistant 300.

[0068] Throughout the present disclosure, in all the figures showing a GUI, left and bottom portions of the GUI pages comprise standard GUI features that comply with a standard prescribed and followed by semiconductor equipment manufacturing industry (SEMI). The standard is generally known as the SEMI standard. The standard GUI features of the SEMI standard, including the alarm page, which is also part of the SEMI standard, are not altered by the recovery assistant of the present disclosure except for the following.

[0069] As shown in FIG. 3A onwards, a recovery assistant option (button) is added to the alarm page in the space not allocated to the GUI features of the SEMI standard. For example, the recovery assistant button is provided in bottom right portion of the alarm page above the GUI features of the SEMI standard located at the bottom of the alarm page (e.g., see FIG. 3A). For example, the recovery assistant button is located under the recovery options on the alarm page. Further, since the recovery assistant can also automatically select the correct recovery option without requiring the FSE to make the selection, the only other change to the alarm page, in addition to adding the recovery assistant button, is removal of the recovery options from above the recovery assistant button on the alarm page (see FIG. 3B). A typical alarm recovery procedure performed using the GUI 170 without the recovery assistant 300 is now described.

[0070] FIG. 2A shows an example of a standard alarm page 200 without the recovery assistant 300. The alarm page 200 includes a first portion 202 on the left-hand side of the alarm page 200 that includes information about the tool 100 and the PMs 102 according to the SEMI standard. The first portion 202 also includes some standard commands such as Pause System to interact with and operate the tool 100 according to the SEMI standard. The alarm page 200 includes a second portion 204 at the bottom of the alarm page 200 that includes additional standard commands to interact with and operate the tool 100 according to the SEMI standard. For example, the additional commands include Operate, Diagnose, Maintain, Setup, Alarm, etc. Any command in the first and second portions 202, 204 can be selected by clicking on a respective button.

[0071] The alarm page 200 can display one or more alarms. Examples of the alarms are shown as Alarm 1 210-1 and Alarm 2 210-2 (collectively called the alarms 210 and individually called the alarm 210). The alarm 210 includes alarm text comprising a description of the alarm (e.g., nature of a detected error or fault), an error code, and additional information about the alarm.

[0072] On the alarm page 200, an operator (e.g., typically an FSE) reads the alarm text of the alarm 210. For example, the alarm 210 indicates a wafer transfer error in the AWC procedure (e.g., a robot interface error). The alarm page 200 includes options 212 to search, filter, and delete alarms 210. The alarm page 200 also provides recovery options 220 to recover from the alarm 210.

[0073] The recovery options 220 differ depending on the alarm 210. For example, the recovery options 220 for the alarm 210 associated with the wafer transfer error include Acknowledge, Retry, or Kill. Retry involves verifying acceptable wafer position at source before retrying. Kill involves killing all process jobs in the PM. Killing the jobs also sends the transfer robot to home position, and disables the transfer chamber and the airlock.

[0074] Before selecting one of the recovery options 220, the FSE needs to determine the root cause of the alarm 210. The FSE can determine the root cause of the alarm 210 by performing the following procedures, which include performing various operations and at times some manual actions as well.

[0075] Typically, the first action in response to the alarm 210 is to pause the system by clicking on a pause button (not shown) in the first portion 202 of the alarm page 200. Then the FSE selects (clicks on) an Operate command in the second portion 204 of the alarm page 200 to operate the tool camera system in an effort to determine the root cause of the alarm 210. Selecting the Operate command opens a new page (not shown) on the GUI and displays various cameras located in the transfer chamber (VTM) of the tool 100. The FSE may be unable to decide which and how many cameras to use / view to determine the root cause for the alarm 210.

[0076] One way to proceed is to go to the Maintain and Diagnostics pages of the GUI and examine dynamic alignment (DA) sensors, which are used to align the wafer, as follows. The FSE selects (clicks on) a Maintain command in the second portion 204 of the alarm page 200. Selecting the Maintain command opens yet another new page on the GUI shown in FIG. 2B. On the new page, schematics of the VTM is displayed in a center portion of the page as shown at 205. On the new page, sub-options of the Maintain command (e.g., module, wafer, diagnostics, etc.) are also displayed above the second portion 204 of the GUI as shown at 206. The sub-options shown at 206 may also be according to the SEMI standard. For example, the FSE can select Module in the suboptions shown at 206. Thus, the FSE can select Maintain and then select Module.

[0077] On the new page shown in FIG. 2B, options such as Go To Maintenance, Reinitialize, etc. are shown on the top of the page as shown at 208. The FSE can select Go To Maintenance from the options shown at 208 to place the selected Module (VTM) in maintenance mode. Additionally, the new page shown in FIG. 2B includes options (buttons) for selecting and operating components such as Airlocks, Vent, Pump to Base, etc. as shown at 214. Other commands are shown on the right side of the page at 214. With Maintain, Module, and Go To Maintenance selected, the FSE can select Vent command from the options shown at 214 to vent the VTM as shown in FIG. 2C. If the venting operation does not complete successfully, the FSE can select and operate one or more valves by clicking on the valves displayed in the schematics shown in center portion of the page (at 205) to ensure that the VTM is vented successfully.

[0078] In FIG. 2D, to diagnose the DA sensors, the FSE can select a Diagnose command in the second portion 204 and can select I / O from the sub-options shown at 206. The status of various DA sensors (shown in FIG. 2D as data for selected option) is displayed in the center of the page as shown at 207. Additionally, the FSE can select the Maintenance command from the options shown at 204, which opens another new page shown in FIG. 2E.

[0079] In FIG. 2E, the FSE can select Diagnostics from the sub-options shown at 206. The FSE can then select DA sensors from the options shown at the top of the page at 218 to diagnose the DA sensors. The diagnostic data of various DA sensors (shown as additional data for selected option in FIG 2E) is displayed in the center portion of the page as shown at 209. For example, the diagnostic data can include intensities of the DA sensors and respective thresholds. The FSE can analyze the data to determine if the DA sensors are working properly. At this point, the FSE can manually clean the DA sensors if needed based on the diagnostic data and can manually adjust the wafer.

[0080] In FIG. 2F, the FSE can then select the Maintenance command from the options shown at 204, select Module from the sub-options of the Maintenance command shown at 206, and select a Pump to Base command from the options shown at 214 to return the VTM to vacuum. The FSE can also select one or more valves from the schematic of the VTM shown at 205 (if toggled earlier to vent the VTM) to return the VTM to vacuum.

[0081] At this point, the FSE can be sure that the VTM and the DA sensors are operating properly, that the wafer has been manually adjusted, and that the retry recovery option can be selected for the tool 100 to resume normal operation. The FSE selects the Alarmcommand from the options shown at 204, which opens the alarm page on which the FSE can select the correct recovery option 220 (e.g., retry).RECIVERY ASSISTANT - GUI ASPECT (FRONT END)

[0082] Instead of going through the extensive procedure to diagnose the root cause and recover from the alarm as described above, the present disclosure adds an option (a clickable button) of the recovery assistant 300 on the alarm page 200. The addition of the recovery assistant 300 on the alarm page 200 does not otherwise alter the setup of the alarm page 200 that complies with and provides the options required by the SEMI standard. On the alarm page 200, the operator (e.g., the FSE) reads the alarm text and the recovery options 220, and selects (clicks on) the recovery assistant 300. The GUI opens a new page, which is called the recovery assistant page 302. On the recovery assistant page 302, the recovery assistant 300 determines the root cause of the alarm 210 and guides the FSE to select the correct recovery option by automatically (and at times, depending on the nature of the alarm 210, interactively) performing the following operations on the recovery assistant page 302.

[0083] Specifically, the recovery assistant 300 first provides a diagnostics report 310 to determine the root cause for the alarm 210 and then provides the steps to arrive at the correct recovery option to recover from the alarm 210 as described below in detail. In some examples, in addition to determining the root cause, the recovery assistant 300 may perform all the recovery steps automatically without any input from the FSE. Additionally, the recovery assistant 300 can also automatically select the correct recovery option, which fully automates the alarm recovery process as described below in detail.

[0084] FIGS. 3A and 3B show two examples of the alarm page 200 with the addition of the recovery assistant option (button) 300. FIG. 3A shows the alarm page 200 with the added recovery assistant 300 and shows the recovery options 220. FIG. 3B shows the alarm page 200 with the added recovery assistant 300 without the recovery options 220.

[0085] In FIG. 3A, the FSE clicks on the recovery assistant 300 and is taken to the recovery assistant page (see FIG. 4) 302. On the recovery assistant page 302, the recovery assistant 300 determines the root cause of the alarm 210 caused by an error in the AWC procedure and which steps to perform to recover from the alarm 210 as described below in detail. Then the FSE can decide which of the recovery options 220 on the alarm page 200 can be selected to recover from the alarm 210. The FSE returns to the alarm page 200 and selects the correct recovery option 220 to recover from thealarm 210. After the FSE selects the correct recovery option 220, the tool 100 can resume normal operation.

[0086] In FIG. 3B, the FSE clicks on the recovery assistant 300 and is taken to the recovery assistant page (see FIG. 4) 302. On the recovery assistant page 302, the recovery assistant 300 determines the root cause of the alarm 210 caused by an error in the AWC procedure and which steps to perform to recover from the alarm 210 as described below in detail. In addition, the recovery assistant 300 determines which of the recovery options 220 is correct. The recovery assistant 300 automatically selects the correct recovery option 220 to recover from the alarm 210. The FSE does not have to return to the alarm page and select the correct recovery option 220 to recover from the alarm 210. At the end of the procedures performed by the recovery assistant 300, the tool 100 can resume normal operation.

[0087] The recovery assistant 300 comprises two components. A first component of the recovery assistant 300 is a diagnostics report 310. The diagnostics report 310 comprises hypotheses (described below) to determine the root cause of the alarm 210. In some embodiments, the diagnostics report 310 may comprise displaying the root cause determined by the recovery assistant 300 without displaying the hypotheses. A second component of the recovery assistant 300 is the steps to recover from the alarm 210. Both components are now described below in detail.

[0088] FIG. 4 shows the first component - the diagnostics report 310 - of the recovery assistant 300. The diagnostics report 310 is displayed on the recovery assistant page 302 after the recovery assistant 300 is selected on the alarm page 200. Specifically, after the FSE selects the recovery assistant 300 on the alarm page, the recovery assistant 300 presents the recovery assistant page 302 on the GUI 170 on the display 152 shown in FIG. 1 . The recovery assistant 300 displays the diagnostics report 310 on the recovery assistant page 302. The diagnostics report 310 comprises identifying information such as the error code of the alarm 210, time stamp, error description, component ID of the component that likely failed, the command in the AWC procedure that failed and caused the alarm 210, etc.

[0089] The diagnostics report 310 comprises a plurality of hypotheses for determining the root cause for the alarm 210. For example, the hypotheses comprise information about components used in the AWC procedure that are likely to be defective and proposed solutions (e.g., comments or suggestions regarding corrective actions toperform). To formulate the hypotheses, the recovery assistant 300 analyzes the information in the alarm 210. The recovery assistant 300 identifies information in past error logs and in past and current diagnostic results that is relevant to the error in the alarm 210.

[0090] For example, the past error logs may comprise corrective actions performed (e.g., by the FSE and / or by the recovery assistant 300) to recover from similar or related errors and alarms. For example, the diagnostic results may comprise results of diagnostics performed (e.g., by the FSE and / or by the recovery assistant 300) on sensors related to the alarm 210 (e.g., DA sensors and cameras used in the AWC procedure).

[0091] For other alarms, the diagnostic results may comprise results of diagnostics performed on other sensors such as temperature and pressure sensors for diagnosing the other alarms. In other examples (i.e., for other alarms), the diagnostic results may comprise results of diagnostics performed on other components (e.g., voltage and current sensors, valves, pumps, motors, etc. in the subsystems of the tool 100).

[0092] For example, the diagnostics may have been performed periodically by the system controller 150 before the occurrence of the error in the alarm 210. The diagnostics may also be additionally performed currently by the recovery assistant 300 after the occurrence of the error in the alarm 210. These error logs and diagnostic results are stored in one or more databases (e.g., in a server locally or in cloud) that are coupled to the system controller 150. The recovery assistant 300 retrieves the relevant information from these error logs and diagnostic results from the one or more databases.

[0093] The recovery assistant 300 correlates the error in the alarm with the above data mined from the error logs and diagnostic results. The recovery assistant 300 selects only that information from these error logs and diagnostic results that is relevant to the alarm 210, analyzes the selected information, and formulates the hypotheses for determining the root cause for the alarm 210. The recovery assistant 300 uses logic tables and rules for formulating and filtering (eliminating) the hypotheses. Examples of logic tables and rules are described below later.

[0094] The hypotheses depend on the nature of the alarm 210. The type and amount of the information selected by the recovery assistant 300 to formulate the hypotheses can be both vast and diverse. For example, the recovery assistant 300 may select the information based on the relevance of the alarm 210 to other errors that have occurred in the past. Based on the information in the alarm 210, the recovery assistant 300 selectsthe relevant information about related components (e.g., DA sensor, arms and / or end effectors of the robots, etc.). The recovery assistant 300 selects the relevant information about alarms already caused by the related components and about the actions that have been performed to resolve the errors in the alarms.

[0095] Further, the recovery assistant 300 may select the time window within which to select the information to formulate the hypotheses. For example, if another component under consideration (i.e., a possible candidate for causing the current alarm 210) has been recently replaced and is operating correctly (e.g., as indicated by respective diagnostic results), the recovery assistant 300 can exclude the information about such components and can eliminate the component as a possible cause of the current alarm 210.

[0096] The recovery assistant 300 tests each hypothesis and filters (eliminates) those hypotheses that are less likely to be the root cause for the alarm 210. For example, the recovery assistant 300 can eliminate a hypothesis that the DA sensors may be the root cause for the alarm 210 if the intensities and thresholds of the DA sensors collected by running diagnostics (i.e., currently, after the occurrence of the alarm 210; or recently, before the occurrence of the alarm 210) are within normal ranges.

[0097] As another example, the recovery assistant 300 may check if any alarms have occurred that can cause the wafer to shift on the end effector to formulate a hypothesis that the end effector may be causing the alarm 210. If the end effector has caused any alarms, then that information is relevant to the alarm 210 indicating a wafer alignment error. If the end effector has not caused any alarms, then the recovery assistant 300 filters out (eliminates) the hypothesis that the end effectors may be the root cause for the alarm 210.

[0098] Since the alarm 210 relates to wafer misalignment that caused the alarm 210 in the AWC procedure, the recovery assistant 300 may also check historical data about wafer offsets (positioning data relative to the center of the robot), which are relevant to the alarm 210. The recovery assistant 300 checks for any correlation between the wafer types and the robot arms used during previously performed wafer transfers. Depending on the historical data and any correlation, the recovery assistant 300 can determine if a particular type of wafer is causing the alarm 210; or if the robot arm, regardless of the type of wafer, is causing the alarm. Based on the determination, the recovery assistant300 can formulate, test, and confirm a hypothesis that a particular type of wafer or the robot arm is causing the alarm 210.

[0099] Thus, the recovery assistant 300 formulates the hypotheses, tests each hypothesis based on relevant data, filters (eliminates) some of the hypotheses, and determines the root cause for the alarm 210 based on the hypothesis that indicates the most likely cause (the root cause) for the alarm 210. Each hypothesis is independent of the other hypotheses. The independence of the hypotheses eliminates many redundant checks of seemingly inter-related or overlapping likely causes for the alarm 210 that might otherwise need to be performed. Thus, the independent hypotheses make it possible for the recovery assistant 300 to quickly isolate the root cause for the alarm 210.

[0100] The diagnostics portion of the recovery assistant 300 tests all hypotheses because in some instances two hypotheses may be true. For example, maybe the root cause of a DA error was due to de-chucking issues in the PM. Additionally, a portion of the substrate may be booking the DA sensors. The diagnostics portion of the recovery assistant 300 will indicate that there was a de-chucking issue and that there is a DA error.

[0101] Notably, without the recovery assistant 300, the FSE may not know where to find the voluminous and diverse information that is relevant to the alarm 210, which diagnostics results to consider, which diagnostics to run, and which time frame(s) to select within which the information is relevant and useful to determine the root cause for the alarm 210. Instead, the recovery assistant 300 uses logic tables or rule tables that are designed to analyze specific alarms, to collect and analyze the relevant information for recovering from the specific alarms, and to formulate and filter the hypotheses to determine the root cause for the specific alarms. Examples of the logic tables and rules are described below.

[0102] Further, the recovery assistant 300 uses the selected past (historical) information as feedback to formulate the hypotheses and to then propose steps to recover from the alarm 210 based on the hypotheses. Additionally, the recovery assistant 300 can also propose (forecast) new hypotheses and new recovery steps (e.g., variations of feedbackbased hypotheses and recovery steps) to eliminate some of the hypotheses and recovery steps for quickly determining the root cause of the alarm 210 and quickly recovering from the alarm 210.

[0103] After determining the root cause of the alarm 210 using the hypotheses, the recovery assistant 300 can formulate steps to recover from the alarm 210. For example,the recovery assistant 300 can propose the recovery steps by correlating the error in the alarm 210 with corrective actions that were performed to correct similar errors in prior occurrences of the alarm 210. The recovery assistant 300 can further narrow down the relevant data about the past corrective actions mined from the error logs based on the root cause identified by the hypothesis to suggest only those steps that are needed to recover from the specific error in the current instance of the alarm 210.

[0104] Based on the diagnostics report, with the root cause and the recovery steps for the alarm 210 determined by the recovery assistant 300, the FSE can decide which steps to perform to recover from the alarm 210. From the recovery steps suggested by the recovery assistant 300, the FSE can decide if a step can be skipped. The FSE can also add a step to the set of recovery steps suggested by the recovery assistant 300. For example, the FSE can replace a step suggested by the recovery assistant 300 with another step. The recovery assistant 300 also provides an option to execute the steps automatically or interactively. Further, depending on the amount of service (e.g., parts and / or time) needed to recover from the alarm 210, the FSE can also take a PM offline and perform the steps while the rest of the tool (e.g., other PMs) can continue to operate normally.

[0105] Further, the recovery assistant 300 can also display a summarized version of the diagnostics report 310 depending on the experience level of the user. For example, the experience level can be inferred from login information (e.g., if the recovery assistant 300 is invoked by an operator or an FSE). For example, the full diagnostics report (as shown in FIG. 4) can be shown to an FSE who is more experienced in troubleshooting and servicing the tool 100. Alternatively, an abbreviated version of the diagnostics report may be shown to an operator of the tool 100 who is typically less experienced than the FSE in troubleshooting and servicing the tool 100.

[0106] For example, the full diagnostics report can also show additional data that can be interpreted by the FSE but not by the operator of the tool 100. The additional data can be filtered out and not displayed to the operator of the tool 100. Additionally, in the second component of the recovery assistant 300 (described below), automatic versus manual options to perform the steps to recover from the alarm 210 can be useful to both the FSE and the operator of the tool 100.

[0107] FIGS. 5A-5C show examples of the steps to recover from the alarm 210 and other information (e.g., data (e.g., an image), questions, etc.) presented on a recoveryassistant page 302, which is the second component of the recovery assistant 300. The information is dynamically presented on the recovery assistant page 302. The information is customized (i.e., is relevant and tailored) to recover from the specific alarm 210. That is, the information will differ depending on the nature of the alarm 210. The backend of the recovery assistant 300, which formulates the hypotheses and provides the diagnostics report in the first component of the recovery assistant 300, also manages all the information presented on the recovery assistant page 302 including the recovery steps and any information presented during the performance of the recovery steps as described below.

[0108] FIG. 5A shows an example of the recovery assistant page 302. Some of the information such as the alarm 210 itself is copied from the alarm page 200 onto the recovery assistant page 302. As in the alarm page 200, the first and second portions 202, 204 of the recovery assistant page 302 conform to the SEMI standard and are unchanged (i.e., are not altered). In a center portion 304 of the recovery assistant page 302, the recovery assistant 300 displays the steps that need to be performed to recover from the alarm 210. For example, the steps are shown as Step 1 , Step 2, Step 3, ..., Step N. For example, the steps for the alarm 210 may include start recovery, pause system, check wafer position, home robot, vent VTM, manual wafer reposition, pump VTM, resume system, retry failed action, and confirm recovery.

[0109] The recovery assistant 300 displays a start recovery button on the recovery assistant page 302 to initiate the steps. The steps are formulated and sequenced based on the root cause for the alarm 210 determined by the recovery assistant 300 and based on the diagnostics report described above. For example, the steps to recover from the alarm 210 may be based on trends in DA sensor data; the health of the DA sensors, which the recovery assistant 300 can derive from raw sensor signals; history of past posts for the alarm 210 (which can include similar steps performed in response to the occurrence of the alarm 210 in the past); part number and firmware version of the robot; the wafer type; and so on.

[0110] FIGS. 5B and 5C show examples of how the steps progress on the recovery assistant page 302. For example, in FIG. 5B, Step 1 (e.g., Start Recovery) is completed. Step 2 (e.g., Pause System) requires input and is displayed with details or instructions such as check jobs running on the tool 100, pause the system, and confirm paused status. Step 2 also provides an option to perform the remaining steps automatically (i.e.,by the recovery assistant 300 without any input from the FSE) or manually (i.e., interactively, where the recovery assistant 300 provides additional information, asks questions, and receives answers during performance of some steps). The FSE can select one option (e.g., automatic or manual) and can click on execute button after which the recovery assistant 300 performs the steps according to the selection. The recovery assistant 300 also displays a stop recovery button on the recovery assistant page 302, which the FSE can use at any time.

[0111] FIG. 5C shows an example of a step that is interactively performed by the recovery assistant 300 on the recovery assistant page 302. For example, Step 1 and Step 2 are completed. Step 3 may be Check Wafer Position. Step 3 may require manual input. In Step 3, for example, an image 306 of the wafer captured by one of the cameras is displayed on the recovery assistant page 302 for the FSE to view. Additionally, the recovery assistant 300 displays details such as which cameras are on along with the image 306. Next to the image 306, the recovery assistant 300 asks a question such as “Can you see the wafer?” and provides Yes or No (or Confirm) as options for the FSE to select and respond with after viewing the image 306.

[0112] Depending on the alarm and the root cause, the type of information displayed, questions asked, and responses prompted on the recovery assistant page 302 may differ. Additionally, to further automate the recovery steps, showing images and asking questions on the recovery assistant page 302 can be obviated by employing image processing techniques with which the recovery assistant 300 can automatically analyze an image and confirm status of an object (e.g., wafer position) without interacting with the FSE.

[0113] After the FSE responds to the question, the recovery assistant 300 performs the remaining steps automatically or manually as initially selected. Also, after the interactive step, the recovery assistant 300 can again provide an option to execute the remaining steps automatically or manually for the FSE to select. Examples of the remaining steps may include Home the Robot, Vent the VTM, Manual Wafer Reposition, Pump the VTM, Resume the System, Retry the Failed Action, and Confirm Recovery. Even if the automatic mode is selected for performing the recovery steps, the recovery assistant 300 can interactively display data and ask a question as needed during any of the recovery steps. Further, depending on the response to a question, the recovery assistant 300and / or the FSE can automatically add or remove one or more steps from the sequence of recovery steps.

[0114] After the recovery is confirmed by the recovery assistant 300, the FSE can return to the alarm page 200 and select the correct recovery option (e.g., Retry), and the tool 100 can resume normal operation. Alternatively, the recovery assistant 300 itself can select the correct recovery option (e.g., Retry), and the tool 100 can resume normal operation without requiring the FSE to make the selection.RECIVERY ASSISTANT - SOLUTION MODELING ASPECT (BACK END)

[0115] The system controller 150 records (stores) various types of data throughout the operation of the tool 100. For example, the system controller 150 records error logs, diagnostic results of many components and subsystems of the tool 100, etc. during the operation of the tool 100. Different applications such as error loggers and diagnostic programs may store data in different formats. To alleviate the complexities involved in processing of the data in different formats by the recovery assistant 300 when an alarm occurs, the recoded data needs to be reformatted or converted into a format that is suitable for the recovery assistant 300 to ingest and process to provide the diagnostics report 310 and the recovery steps as described above.

[0116] For example, the recoded data can be reformatted or converted by the recovery assistant 300 after retrieving the recoded data. For example, the retrieved data can be prepended with a line of code that provides the relevant data to the recovery assistant 300 in a format that is suitable for the recovery assistant 300. The format is suitable for the recovery assistant 300 to ingest and process the data and to provide the diagnostics report 310 and the recovery steps as described above.

[0117] For example, the code can convert the data into JSON format, which is an open standard file format and data interchange format that uses readable text to store and transmit data objects comprising attribute-value pairs and arrays. The recovery assistant 300 can ingest and process the data in JSON format and can provide the diagnostics report 310 and the recovery steps as described above. Alternatively, a format other than JSON format can be selected, and a mechanism such as a data formatter or converter can be used instead of the code that is prepended to the data recoded.

[0118] In the above description of the frontend component of the recovery assistant 300 (i.e., the diagnostic report 210 and the hypotheses used to determine the root cause),some of the operations performed by the recovery assistant 300 at the backend are already described to explain how the frontend of the recovery assistant 300 is generated and how the frontend works. Specifically, the mining of the error logs and the diagnostic results, determining relevance of the mined data, formulating hypotheses based on the relevant data to determine the root cause, and formulating the recovery steps based on the root cause by the recovery assistant 300 are already described above. The feedbackbased formulation and forecasting of new hypotheses and new recovery steps is also described above. Therefore, these descriptions are not repeated for brevity.

[0119] An example of a method comprising the procedures performed by the recovery assistant 300 at the backend to provide the outputs (i.e., the diagnostics report 310 and the recovery steps) on the recovery assistant page 302 are now described below. Subsequently, examples of logic tables used to formulate the hypotheses are described.

[0120] FIGS. 6A and 6B show a method 400 performed by the recovery assistant 300 to provide the outputs on the recovery assistant page 302 described above. Again, an AWC error is used as an example, and the general methodology described below (and above) is applicable to any other error or alarm. For example, the error recovery module 160 embedded in the system controller 150 of the tool 100 as shown in FIG. 1 performs the method 400 and executes the recovery assistant 300 as follows.

[0121] At 402, for example, upon occurrence of an alarm indicating an AWC error, the error recovery module 160 displays the button for the recovery assistant 300 on the alarm page 200. Upon the selection of the recovery assistant 300 on the alarm page 200, at 404, the error recovery module 160 executes the recovery assistant 300. The recovery assistant 300 analyzes the error and attempts to determine the root cause of the error (e.g., by providing the diagnostics report 310 as described above). If the root cause is known or likely (e.g., determined based on the hypotheses as described above), the method 400 proceeds to 406. If the root cause is unknown (e.g., cannot be determined based on the hypotheses), the method 400 proceeds to 408.

[0122] At 406, if the root cause is known or likely, the recovery assistant 300 determines, for this root cause, whether it is possible to perform recovery steps to allow selecting (clicking) the retry / continue option from the recovery options 320 on the alarm page 200 instead of selecting the kill option from the recovery options 320 on the alarm page 200. If it is not possible to perform recovery steps to allow selecting the retry / continue option from the recovery options 320, the method 400 proceeds to 410, where the FSE canselect the kill option from the recovery options 320 on the alarm page 200. The option to kill the process jobs may be inappropriate for some errors such as AWC errors. The recovery assistant 300 also provides a preview of what actions need to be performed after selecting the kill option.

[0123] If it is possible to perform recovery steps to allow selecting the retry / continue option from the recovery options 320, the diagnostics report 310 of the recovery assistant 300 outputs the root cause to the recovery assistant page 302. The recovery assistant 300 also outputs what is the check that the root cause was fixed to the recovery assistant page 302, and the method 400 proceeds to 408.

[0124] At 408, if the root cause is unknown at 404, or if it is possible to perform recovery steps to allow selecting the retry / continue option from the recovery options 320 at 406, the recovery assistant 300 determines if the wafer and the robot are functioning correctly. For example, the recovery assistant 300 obtains and checks if the wafer offset is less than a maximum value to allow for automatic recovery by performing the recovery steps to allow selecting the retry / continue option. For example, the recovery assistant 300 may also show an image on the recovery assistant page 302 and ask a question on the recovery assistant page 302 regarding the status of the wafer.

[0125] Alternatively, instead of showing the image and asking a question, the recovery assistant 300 may use image processing techniques to automatically determine the wafer status and determine whether the wafer and / or the robot is / are not functioning correctly without any interaction with the FSE. If the wafer and / or the robot is / are not functioning correctly, the method 400 proceeds to 410, where the FSE selects the kill option from the recovery options 320 on the alarm page 200. The recovery assistant 300 also provides a preview of what actions need to be performed after selecting the kill option. If the wafer and / or the robot is / are functioning correctly, the method 400 proceeds to 412.

[0126] At 412, the recovery assistant 300 determines whether an action needs to be performed to make sure that the wafer and / or the robot is / are functioning correctly before selecting the retry / continue option from the from the recovery options 320 on the alarm page 200. If no action needs to be performed to make sure that the wafer and / or the robot is / are functioning correctly, the method 400 proceeds to 414. At 414, the FSE selects the retry / continue option from the recovery options 320 on the alarm page 200. If an action needs to be performed to make sure that the wafer and / or the robot is / are functioning correctly, the method 400 proceeds to 416.

[0127] At 416, the recovery assistant 300 determines if there is a procedure available for the scenario such as the wafer being centered / not centered. If a recovery procedure is not available, the method proceeds to 430. At 430, the FSE can exit the recovery assistant 300 by selecting (clicking) the stop recovery button on the recovery assistant page 302 shown in FIGS. 5B and 5C. The FSE can select an option from the recovery options 320 on the alarm page 200 based on the knowledge of the root cause provided by the recovery assistant 300. The recovery assistant 300 may also provide recommended actions if applicable. The FSE can exit the recovery assistant 300 at any time by selecting (clicking) the stop recovery button on the recovery assistant page 302 shown in FIGS. 5B and 5C. The method 400 ends after 430.

[0128] If a recovery procedure is available, the recovery assistant 300 can display the recovery procedure on the recovery assistant page 302 with a preview of steps and estimated time for performing the steps. This allows the FSE to decide whether to take the resource (e.g., the PM 102, the airlock 120, etc.) offline. For example, the FSE can decide to take the resource offline if the steps are numerous, part are needed, and / or the estimated time is long.

[0129] At this point, if a procedure is available, the method 400 can proceed to 418 or 420 as follows. For example, the available procedure may be a full procedure for fixing the root cause and centering the wafer, with an option to take the resource offline. Alternatively, the available procedure may be only for centering the wafer and not fixing the root cause, with an option to take the resource offline. In either case, the method 400 proceeds to 418. Alternatively, if the procedure is available but it is not relevant or feasible to take the resource offline, the method 400 proceeds to 420.

[0130] At 418, the recovery assistant 300 provides an option to the FSE to decide whether to take the resource offline instead of fixing the root cause. If the FSE decides not to take the resource offline instead of fixing the root cause, the method 400 proceeds to 420. If the FSE decides to take the resource offline instead of fixing the root cause, the method 400 proceeds to 422.

[0131] At 422, the recovery assistant 300 determines if the wafer still needs to be centered. If the wafer does not need to be centered, the method 400 proceeds to 414, where the FSE selects the retry / continue option from the recovery options 320 on the alarm page 200. Alternatively, if the wafer still needs to be centered, the method 400 proceeds to 420.

[0132] At 420, if the FSE decides not to take the resource offline instead of fixing the root cause at 418, or if the wafer still needs to be centered at 422, the recovery assistant 300 provides an option to the FSE to decide whether to start the available procedure now. The FSE can decide to wait instead, in which case, the method proceeds to 430. Alternatively, if the FSE decides to start the available procedure, the method 400 proceeds to 424.

[0133] At 424, the recovery assistant 300 determines if the FSE completed the procedure. Again, the FSE can decide to wait instead, in which case, the method proceeds to 430. Alternatively, if the FSE completed the procedure, the method 400 proceeds to 426.

[0134] At 426, the recovery assistant 300 determines if, after completing the procedure, the check passed for what was fixed by completing the procedure. If the check did not pass, the recovery assistant 300 recommends clicking back (i.e., returning to perform the check) or provides an option to the FSE to decide to wait instead, in which case, the method proceeds to 430. If the check passed, the method 400 proceeds to 414, where the FSE selects the retry / continue option from the recovery options 320 on the alarm page 200.

[0135] Examples of logic tables and rules used by the recovery assistant 300 to formulate the hypotheses described above are now described. As described above, the hypotheses are specific for each error. For example, for one type of error (e.g., the AWC error described in the example above), the data relevant to the error may comprise intensities and thresholds for the DA sensors. The logic employed to determine whether the DA sensor is the root cause of the error in the alarm 210 may comprise formulating two hypotheses be as follows.

[0136] A first hypothesis may be to check if the data for a DA sensor is out of a specified range and determine a status of the DA sensor. This procedure is performed for each DA sensor. For example, the results of the procedure may be as follows: DA sensor 1 is blocked, at risk, or working properly; and DA sensor 2 is blocked, at risk, or working properly. If a DA sensor is blocked or at risk, the DA sensor is likely to be the root cause of the error. If the DA sensors are working properly, the DA sensors are not the root cause of the error, and the first hypothesis is eliminated. Further, if the DA sensor is blocked or at risk, one of the recovery steps may be to clean or replace the DA sensor. Alternatively, the step can be omitted if the DA sensors are working properly.

[0137] A second hypothesis may be to check whether the wafer is still on the end effector of the robot. For example, if the subsystem being checked is not a PM, the recovery assistant 300 asks the FSE to confirm if the wafer is still on the end effector of the robot. The recovery assistant 300 determines whether the root cause is related to the robot based on the outcome of the second hypothesis.

[0138] For another type of error, in addition to the first hypothesis described above, a different second hypothesis may be formulated as follow. The relevant data to formulate the hypothesis may comprise, in addition to the error code and error description, time stamp of the error, wafer ID, subsystem (e.g., PM number), firmware version of the robot, robot arm (e.g., arm A or arm B), the command (e.g., pick) that triggered the error, and so on.

[0139] The hypothesis may be to check historical wafer offset, which may be further categorized or partitioned by the wafer type and the robot arm used for wafer transfer. A two-tiered analysis may be performed to isolate the root cause of the error. In Tier 1 , the recovery assistant 300 can check wafer offset against a warning limit. For example, only some type of wafers picked up from a PM may show a history of offsets greater than the warning limit, in which case a recipe used in the PM, wafer usage in the PM, and hardware in the PM should be checked (i.e., the robot arm may not be the root cause of the error). As another example, only an arm (e.g., arm A or arm B) of the robot in a PM may show a history of offsets greater than the warning limit, in which case the behavior (operation) of that arm and hardware in the PM should be checked (i.e., an arm of the robot may be the likely root cause of the error). In another example, wafers picked from a PM may show a history of offsets greater than the warning limit, but there may be no correlation between the offsets and the wafer type or the arm used, in which case a recipe used in the PM, wafer usage in the PM, and the behavior (operation) of the arm should be checked (i.e., the wafer type and the robot arm may not be the root cause of the error).

[0140] In Tier 2, in addition to Tier 1 , the recovery assistant 300 can check trends in wafer offsets based on DA sensor data. For example, offsets for only some types of wafers picked with an arm (e.g., arm A or arm B) of the robot from a PM may be trending to be greater than the warning limit. Alternatively, some types of wafers picked with an arm (e.g., arm A or arm B) of the robot from a PM may show no trend in DA readings beyond the warning limit. Based on the combination of Tier 1 and Tier 2 analyses, the recovery assistant 300 can determine whether the wafer type, an arm of the robot, a DAsensor, hardware of a PM, or a recipe used in a PM is the root cause of the error. Accordingly, the recovery assistant 300 can devise appropriate recovery steps to resolve the error using the methodology described above. If a recipe is the root cause, the recovery assistant 300 can provide the information to FSE, who can conclude that the tool 100 is operating normally and does not need to be serviced. The operator of the tool 100 can handle the recipe issues separately.

[0141] Thus, although the generic methods for diagnosing alarms using a GUI exist, these methods are inefficient, whereas the recovery assistant 300 is directed to a particular manner of formulating and presenting only those hypotheses that can be used to determine the root cause of the alarm and then presenting only those steps that are needed to recover from the alarm based on the root cause, which makes alarm resolution efficient.

[0142] Specifically, the recovery assistant 300 directly retrieves only that information from the error logs and diagnostics results that is relevant to the alarm. The information retrieval is triggered by selecting the recovery assistant button added to the alarm page 200 that is otherwise unmodified to conform to the SEMI standard. In addition to determining the root cause of the alarm based on the retrieved relevant data, the recovery assistant 300 also lists only the necessary recovery steps, which, when executed (automatically or manually), launch related diagnostics, or retrieve related data as needed to perform the steps, without requiring the user to know which diagnostics to perform and where to locate the data related to the steps.

[0143] Further, during execution of a recovery step, the recovery assistant 300 provides additional information such as an image directly within the same recovery assistant page 302 where the steps are being executed, which again eliminates the need for the user to navigate through multiple pages of the GUI, select a specific camera, and so on. Also, the recovery assistant 300 displays only limited data such as the image that is needed for the step being executed on the recovery assistant page 302, which restrains the amount of data displayed on the recovery assistant page 302.

[0144] Thus, the recovery assistant 300 displays a limited set of information in a specific manner on a single GUI page which is in stark contrast to the conventional GUI, which does not point the user in the right direction and does not retrieve and present only relevant information but rather requires the user to locate the information in different areas by navigating through many GUI pages. Accordingly, the recovery assistant 300provides many specific improvements over prior GUI systems, resulting in an improved user interface for diagnosing alarms in semiconductor tools.

[0145] The recovery assistant 300 is an improved user interface for diagnosing alarms in semiconductor tools since the recovery assistant 300 eliminates many deficits of the conventional GUI relating to the efficient functioning of the system controller 150, requiring a user to jump around and switch views many times to find the right information. Since vast amounts of data about various components such as error logs and diagnostic information, though available, are not easy to locate and their relevance to the error is unknown, prior GUI interfaces required users to drill down through many pages of GUI to get to desired data or functionality. That process is slow, complex and difficult to learn, even to experienced FSEs.

[0146] Instead, the recovery assistant 300 improves the efficiency of using the GUI by bringing together automatic retrieval of relevant data and required analyses of the relevant data to pinpoint the root cause of the error and a limited list of steps to recover from the error, all of which is accomplished on a single GUI page - the recovery assistant page 302 - that is invoked directly with a single click on the recovery assistant button on the alarm page 200.

[0147] Displaying selected and highly relevant data (e.g., the diagnostics report 310) and only those recovery steps that are needed to fix the root cause of the error on the recovery assistant page 302 allows the user to resolve the error without actually navigating through a series of GUI pages and exploring multitude of often unrelated information. The recovery assistant 300 improves the speed of error resolution because the recovery assistant 300 saves the user from navigating to the required pages, and performing diagnostics and analyzing the relevance of diagnostics results within those pages. Rather than paging through multiple screens of options, the recovery assistant 300 comprises only two steps from start to finish (i.e., from clicking on the recovery assistant to resolving the alarm) - first, the diagnostics report 310 that determines the root cause for the FSE; and second, the recovery steps that can also be automatically performed for the FSE - to conclude which is the correct recovery option to select on the alarm page 200 to place the tool 100 in operating condition. The recovery assistant 300 also reduces the likelihood of human error during the diagnosis and recovery steps. When operators / FSEs do not know where to start diagnosing, do not know where to start for recovery, and have to explore a lot of Ul screens to find information to solve a problem,they tend to make mistakes, some of which can severely damage the tool and / or other components (e.g., break substrates). The recovery assistant 300 alleviates these issues. Thus, the recovery assistant 300 improves the functioning of semiconductor tools.

[0148] The foregoing description is merely illustrative in nature and is not intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims.

[0149] It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the examples is described above as having certain features, any one or more of those features described with respect to any one of the examples of the disclosure can be implemented in and / or combined with features of any of the other examples, even if that combination is not explicitly described. In other words, the described examples are not mutually exclusive, and permutations of one or more examples with one another remain within the scope of this disclosure.

[0150] Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

[0151] In some implementations, a controller is part of a system, which may be part of the above-described examples. Such systems can comprise semiconductor processing equipment, including a processing tool or tools, chamber or chambers, a platform or platforms for processing, and / or specific processing components (a wafer pedestal, a gasflow system, etc.). These systems may be integrated with electronics for controlling their operation before, during, and after processing of a semiconductor wafer or substrate.

[0152] The electronics may be referred to as the “controller,” which may control various components or subparts of the system or systems. The controller, depending on the processing requirements and / or the type of system, may be programmed to control any of the processes disclosed herein, including the delivery of processing gases, temperature settings (e.g., heating and / or cooling), pressure settings, vacuum settings, power settings, radio frequency (RF) generator settings, RF matching circuit settings, frequency settings, flow rate settings, fluid delivery settings, positional and operation settings, wafer transfers into and out of a tool and other transfer tools and / or load locks connected to or interfaced with a specific system.

[0153] Broadly speaking, the controller may be defined as electronics having various integrated circuits, logic, non-transitory memory, and / or software that receive instructions, issue instructions, control operation, enable cleaning operations, enable endpoint measurements, and the like. The integrated circuits may include chips in the form of firmware that store program instructions, digital signal processors (DSPs), chips defined as application specific integrated circuits (ASICs), and / or one or more microprocessors, or microcontrollers that execute program instructions (e.g., software).

[0154] Program instructions may be instructions communicated to the controller in the form of various individual settings (or program files), defining operational parameters for carrying out a particular process on or for a semiconductor wafer or to a system. The operational parameters may, in some examples, be part of a recipe defined by process engineers to accomplish one or more processing steps during the fabrication of one or more layers, materials, metals, oxides, silicon, silicon dioxide, surfaces, circuits, and / or dies of a wafer.

[0155] The controller, in some implementations, may be a part of or coupled to a computer that is integrated with the system, coupled to the system, otherwise networked to the system, or a combination thereof. For example, the controller may be in the “cloud” or all or a part of a fab host computer system, which can allow for remote access of the wafer processing. The computer may enable remote access to the system to monitor current progress of fabrication operations, examine a history of past fabrication operations, examine trends or performance metrics from a plurality of fabricationoperations, to change parameters of current processing, to set processing steps to follow a current processing, or to start a new process.

[0156] In some examples, a remote computer (e.g., a server) can provide process recipes to a system over a network, which may include a local network or the Internet. The remote computer may include a user interface that enables entry or programming of parameters and / or settings, which are then communicated to the system from the remote computer. In some examples, the controller receives instructions in the form of data, which specify parameters for each of the processing steps to be performed during one or more operations. It should be understood that the parameters may be specific to the type of process to be performed and the type of tool that the controller is configured to interface with or control.

[0157] Thus, as described above, the controller may be distributed, such as by comprising one or more discrete controllers that are networked together and working towards a common purpose, such as the processes and controls described herein. An example of a distributed controller for such purposes would be one or more integrated circuits on a chamber in communication with one or more integrated circuits located remotely (such as at the platform level or as part of a remote computer) that combine to control a process on the chamber.

[0158] Without limitation, example systems may include a plasma etch chamber or module, a deposition chamber or module, a spin-rinse chamber or module, a metal plating chamber or module, a clean chamber or module, a bevel edge etch chamber or module, a physical vapor deposition (PVD) chamber or module, a chemical vapor deposition (CVD) chamber or module, an atomic layer deposition (ALD) chamber or module, an atomic layer etch (ALE) chamber or module, an ion implantation chamber or module, a track chamber or module, and any other semiconductor processing systems that may be associated or used in the fabrication and / or manufacturing of semiconductor wafers.

[0159] As noted above, depending on the process step or steps to be performed by the tool, the controller might communicate with one or more of other tool circuits or modules, other tool components, cluster tools, other tool interfaces, adjacent tools, neighboring tools, tools located throughout a factory, a main computer, another controller, or tools used in material transport that bring containers of wafers to and from tool locations and / or load ports in a semiconductor manufacturing factory.

Claims

CLAIMSWhat is claimed is:1 . An error recovery system for a substrate processing tool, comprising: a processor; and non-transitory memory storing instructions configured to cause the processor to: display a recovery assistant button on a graphical user interface (GUI), the GUI being configured to display an error in the substrate processing tool; and in response to activation of the recovery assistant button: retrieve data relevant to the error from a database; generate a diagnostic report indicating a root cause of the error by analyzing the data; display the diagnostic report indicating the root cause of the error on the GUI; generate a set of steps to recover from the error based on the root cause; display the set of steps on the GUI; execute at least portions of the set of steps automatically, interactively, or using a combination thereof on the GUI; and execute a recovery option from a plurality of recovery options to place the substrate processing tool in normal operating condition.

2. The error recovery system of claim 1 wherein the GUI is configured to display the error on a first page of the GUI, and wherein the instructions are configured to cause the processor to: display the recovery assistant button on the first page of the GUI; display the diagnostic report on a second page of the GUI; display the set of steps on the second page of the GUI; and execute the set of steps on the second page of the GUI.

3. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to automatically execute the recovery option.

4. The error recovery system of claim 1 wherein: the GUI is configured to display the plurality of recovery options; and upon executing the set of steps on the GUI, the recovery option is interactively selectable for execution on the GUI.

5. The error recovery system of claim 2 wherein: the GUI is configured to display the plurality of recovery options on the first page of the GUI; and upon executing the set of steps on the second page of the GUI, the recovery option is interactively selectable for execution on the first page of the GUI.

6. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to determine at least one of (i) an amount of information to display in the diagnostic report and (ii) whether to execute the steps automatically or interactively based on a type of user using the GUI.

7. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to execute the set of steps by switching between executing the steps automatically and interactively.

8. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to, during interactive execution of a step in the set of the steps, display information on the GUI and receive input through the GUI.

9. The error recovery system of claim 2 wherein the instructions are configured to cause the processor to, during interactive execution of a step in the set of the steps, display information on the second page of the GUI and receive input through the second page of the GUI.

10. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, automatically perform (i) one or more operations of the step, (ii) access and process relevant data from the database, and (iii) display information about the step adjacent to the step on the GUI.1 1 . The error recovery system of claim 2 wherein the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, while on the second page of the GUI, automatically perform (i) one or more operations of the step, (ii) access and process relevant data from the database, and (iii) display information about the step adjacent to the step on the second page of the GUI.

12. The error recovery system of claim 7 wherein the one or more operations comprise performing a diagnostic procedure on a component of the substrate processing tool.

13. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, interactively display information on the GUI, receive input through the GUI, and resume the automatic execution of the step on the GUI in response to receiving the input.

14. The error recovery system of claim 2 wherein the instructions are configured to cause the processor to, during automatic execution of a step in the set of the steps, interactively display information on the second page of the GUI, receive input through the second page of the GUI, and resume the automatic execution of the step on the second page of the GUI in response to receiving the input.

15. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to interactively allow at least one of adding, deleting, replacing, and omitting one or more steps in the set of steps.

16. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to automatically at least one of add, delete, replace, and omit one or more steps in the set of steps during the execution of the set of the steps.

17. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to select the data relevant to the error from at least one of an error log and diagnostic data stored in the database based on a correlation of the data with the error.

18. The error recovery system of claim 17 wherein the instructions are configured to cause the processor to convert a data format of at least one of the error log and the diagnostic data retrieved from the database into a format adapted for ingesting and processing the data to generate the diagnostic report and the set of steps.

19. The error recovery system of claim 17 wherein the instructions are configured to cause the processor to generate a portion of the diagnostic data by performing a diagnostic procedure on a component of the substrate processing tool after the error.

20. The error recovery system of claim 1 wherein the instructions are configured to cause the processor to: formulate one or more hypotheses for determining the root cause of the error by correlating the error to the data retrieved from the database; and generate the diagnostic report comprising the one or more hypotheses.21 . The error recovery system of claim 20 wherein each of the hypotheses comprises information about a component related to the error and a corrective action to recover from the error.

22. The error recovery system of claim 20 wherein the hypotheses are independent of each other to eliminate redundant checking of inter-related or overlapping causes of the error.

23. The error recovery system of claim 20 wherein the instructions are configured to cause the processor to, using a set of rules designed for the error: analyze the error; collect and process relevant data based on the analysis; and formulate and filter the hypotheses to determine the root cause of the error based on the processing.

24. The error recovery system of claim 20 wherein the instructions are configured to cause the processor to: test each of the hypotheses based on relevant data; filter a first hypothesis from the hypotheses that indicates a less likely cause for the error; and select a second hypothesis from the hypotheses that indicates a most likely cause for the error as a hypothesis indicating the root cause of the error.

25. The error recovery system of claim 24 wherein the instructions are configured to cause the processor to filter the first hypothesis by performing a diagnostic procedure on a component indicated by the first hypothesis as being related to the error.

26. The error recovery system of claim 24 wherein the instructions are configured to cause the processor to filter the first hypothesis in response to determining that a component indicated by the first hypothesis as being related to the error has not caused another error similar to the error.

27. The error recovery system of claim 24 wherein the instructions are configured to cause the processor to confirm that the second hypothesis indicates the root cause of the error by identifying correlations in historical data of a component indicated by the second hypothesis as being related to the error.

Images