Methods and apparatus for monitoring a processing tool

Abstract

Methods and apparatus for monitoring a processing apparatus are provided. In some embodiments, techniques may include determining, in real-time, a first temperature associated with a first component of the processing apparatus based on one or more measurements obtained from at least one sensor disposed relative to the first component of the processing apparatus; and based at least on the first temperature associated with the first component reaching a first threshold value, performing at least a first preventive measure that mitigates degradation or failure associated with the processing apparatus.

Description

LAM1P015WO-11765-1WO METHODS AND APPARATUS FOR MONITORING A PROCESSING TOOL INCORPORATION BY REFERENCE

[0000] A PCT Request Form is filed concurrently with this specification as part of the present application. Each application that the present application claims benefit of or priority to as identified in the concurrently filed PCT Request Form is incorporated by reference herein in its entirety and for all purposes. BACKGROUND

[0001] Tools configured to deposit, etch and / or manipulate materials and films on semiconductor substrates using gaseous chemistries require delivery and receipt of process gases, reactant gases, carrier gases, and so on, to desired destinations such as processing chambers. Gas delivery can involve complex physical mechanisms, temperature and pressure control, and precise control of delivery parameters (e.g., flow rate, amount). Sufficient deviations in temperature or pressure from appropriate levels can introduce unfavorable conditions that can degrade the tool or result in suboptimal delivery parameters. For example, condensation of vapors may produce unacceptable levels of particle contamination, or corrosion in gas lines can occur, which can affect the processing or even result in failed products and waste.

[0002] To overcome these issues in certain configurations and tools, an array of heating elements may be installed to a gas line. Under normal conditions, this results in expected performance of the precursor gases, without issues with contamination, condensation, corrosion, etc. However, there are situations during servicing or maintenance in which a heating element may be deactivated or unplugged, or the heating element may experience failure. In such cases, the heated gas line or parts thereof may start cooling. In certain tools, these heating elements are controlled outside the tool’s software or controller, so the state of the heating element may not be tracked. Cooling may result in condensation, particle issues, and / or product failure.

[0003] Alternative to cooling scenarios, there are situations in which the heating element would be working normally, but the controlling thermocouple may be disconnected, resulting in more power being drawn by the heating element. This can result in the temperature rising higher than designed conditions and may have an effect of accelerated corrosion of typically stainless steel gas lines. This can also result in undesirable corrosion, particle issues, and / or product failure.

[0004] While thermocouples may be placed to monitor temperatures, this requires active and manual monitoring and can still result in failures in production environments. The outcomes of these failures can include costly replacements of entire gas manifolds, extended tool downtime,LAM1P015WO-11765-1WO and / or loss of wafers due to particle and yield issues. A different approach may mitigate the above issues and improve product yield.

[0005] The background description provided herein is for the purposes of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor implicitly admitted as prior art against the present disclosure. SUMMARY

[0006] In one aspect of the present disclosure, a method for monitoring a processing apparatus is disclosed.

[0007] In some embodiments, the method may include determining, in real-time, a first temperature associated with a first component of the processing apparatus based on one or more measurements obtained from at least one sensor disposed relative to the first component of the processing apparatus; and based at least on the first temperature associated with the first component reaching a first threshold value, performing at least a first preventive measure that mitigates degradation or failure associated with the processing apparatus.

[0008] In some embodiments, the method may include obtaining infrared data associated with a surface of a component of the processing apparatus based on one or more measurements by at least one infrared sensor disposed relative to the surface of the component; determining, in real-time, an infrared temperature associated with the surface of the component based on the infrared data; and performing a first preventive measure that mitigates degradation or failure of the processing apparatus based on (i) the infrared temperature associated with the surface of the component reaching a first threshold value, (ii) a rate of change of the infrared temperature, or (iii) a combination thereof.

[0009] In another aspect of the present disclosure, a system is disclosed. The system may include a processing apparatus comprising one or more gas delivery lines configured to carry one or more gases; one or more infrared sensors disposed relative to the one or more gas delivery lines; and one or more controllers communicatively coupled to the one or more infrared sensors, and configured to perform at least portions of the methods described herein.

[0010] In some embodiments, the one or more controllers may be configured to: determine, in real-time, a first temperature associated with a first component of the processing apparatus based on one or more measurements obtained from at least one sensor disposed relative to the first component of the processing apparatus; and based at least on the first temperature associated withLAM1P015WO-11765-1WO the first component reaching a first threshold value, perform at least a first preventive measure that mitigates degradation or failure associated with the processing apparatus.

[0011] In some embodiments, the one or more controllers may be configured to: obtain infrared data associated with a surface of a component of the processing apparatus based on one or more measurements by at least one infrared sensor disposed relative to the surface of the component; determine, in real-time, an infrared temperature associated with the surface of the component based on the infrared data; and perform a first preventive measure that mitigates degradation or failure of the processing apparatus based on (i) the infrared temperature associated with the surface of the component reaching a first threshold value, (ii) a rate of change of the infrared temperature, or (iii) a combination thereof.

[0012] In another aspect of the present disclosure, a controller apparatus configured to monitor a processing apparatus is disclosed. In some embodiments, the controller apparatus may include one or more gas delivery lines configured to carry one or more gases; one or more infrared sensors disposed relative to the one or more gas delivery lines; and one or more controllers communicatively coupled to the one or more infrared sensors, and configured to perform at least portions of the methods described herein.

[0013] In another aspect of the present disclosure, a processing apparatus configured to monitor a component is disclosed. In some embodiments, the processing apparatus may include be configured to perform at least portions of the methods described herein.

[0014] In another aspect of the present disclosure, an apparatus configured to monitor a processing apparatus is disclosed.

[0015] In some embodiments, the apparatus may be configured to: obtain infrared data associated with a surface of a component of the processing apparatus based on one or more measurements, wherein the apparatus is disposed relative to the surface of the component; enable determination, in real-time, of an infrared temperature associated with the surface of the component based on the infrared data; and enable performance of a first preventive measure that mitigates degradation or failure of the processing apparatus based on (i) the infrared temperature associated with the surface of the component reaching a first threshold value, (ii) a rate of change of the infrared temperature, or (iii) a combination thereof.

[0016] In some implementations, the apparatus may be further configured to send the infrared data to one or more controllers associated with the processing apparatus or to at least another apparatus (e.g., an infrared sensor) associated with the processing apparatus. In someLAM1P015WO-11765-1WO implementations, the apparatus may perform said determination and / or performance of the first preventive measure.

[0017] These and other features of the disclosed embodiments will be described in detail below with reference to the associated drawings. BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1A shows an example processing apparatus for depositing or etching a film on or over a substrate utilizing a plasma process, in which the example processing apparatus may include a camera sensor.

[0019] FIG. 1B presents a schematic view of an implementation of a multi-station processing tool; the tool includes four camera sensors.

[0020] FIG. 1C presents a top view of an example electronic device fabrication system having four multi-station fabrication tools, one of which includes camera sensors.

[0021] FIG. 1D presents an example of spectral sensitivity ranges for “standard” (visible- sensitive), IR-sensitive, and UV-sensitive sensor elements in a camera sensor or a combination of camera sensors.

[0022] FIG.2A is a simulated schematic illustrating an example portion of a chemical delivery system, according to some embodiments.

[0023] FIG. 2B depicts a portion of an example chemical delivery line and its associated components.

[0024] FIG. 2C depicts a cross-sectional view of the example chemical delivery line and its associated components.

[0025] FIG.2D depicts a cross-sectional view of the interior of the example chemical delivery line and its associated components.

[0026] FIG. 2E depicts a perspective view of the clamshell used with the example chemical delivery line.

[0027] FIG.3 is a diagram of an example configuration of an infrared (IR) sensor disposed with respect to a gas delivery line, according to some embodiments.

[0028] FIG. 4 is a graph of example temperature values of a gas delivery line and example temperature values of a heat jacket when power is applied, e.g., to a heating element of the gas delivery line.LAM1P015WO-11765-1WO

[0029] FIG. 5 is a graph of example temperature values of a gas delivery line and example temperature values of a heat jacket when power is reduced.

[0030] FIG. 6 is a graph of example temperature values of a gas delivery line and example temperature values of a heat jacket when power is reduced, e.g., when power is cut off from steady- state operation.

[0031] FIG.7 is another graph of example temperature values of a gas delivery line and example temperature values of a heat jacket when power is reduced, e.g., when power is cut off from steady- state operation.

[0032] FIG. 8A is a graph of temperature values of a gas delivery line in an example scenario when power is reduced, e.g., when power is cut off from steady-state operation.

[0033] FIG. 8B is a graph of temperature values of a heat jacket correlating to the temperature values of the gas delivery line in the example scenario of FIG.8A.

[0034] FIG. 9A shows an example thermal image representative of spatially varying infrared temperatures associated with points of interest of a portion of a processing apparatus, according to some embodiments.

[0035] FIG.9B is a graph showing example infrared temperature values obtained for the points of interest of FIG.9A, and an average infrared temperature value.

[0036] FIGS. 10A – 10C show example thermal images of a portion of a processing apparatus, obtained based on different global emissivity values.

[0037] FIG.11A shows an example thermal image with a first global emissivity value applied.

[0038] FIG. 11B is a graph of example infrared temperatures and thermocouple temperatures obtained over time with respect to a region indicated in the example thermal image of FIG. 11A over time.

[0039] FIG. 12A shows an example thermal image with a first global emissivity value and a second global emissivity value applied at different regions of the example thermal image.

[0040] FIG. 12B is a graph of example infrared temperatures and thermocouple temperatures obtained over time with respect to different regions indicated in the example thermal image of FIG.12A over time.

[0041] FIG.13 is an image of a perspective view of at least a portion of a processing apparatus, where a relevant component is visualized with indications representative of infrared temperatures.LAM1P015WO-11765-1WO

[0042] FIG. 14 is another image of a perspective view of at least a portion of a processing apparatus, where a relevant component is visualized with indications representative of infrared temperatures.

[0043] FIG.15 is a flow diagram of an example method of monitoring a processing apparatus, according to some embodiments.

[0044] FIG. 16 is a flow diagram of another example method of monitoring a processing apparatus, according to some embodiments.

[0045] FIG. 17 shows a schematic of an example process system that may be used to perform the methods described herein. DETAILED DESCRIPTION

[0046] The following terms are used throughout the instant specification:

[0047] The terms “semiconductor wafer,” “wafer,” “semiconductor substrate,” “substrate,” “wafer substrate” and “partially fabricated integrated circuit” may be used interchangeably. Those of ordinary skill in the art understand that the term “partially fabricated integrated circuit” can refer to a semiconductor wafer during any of many stages of integrated circuit fabrication thereon. A wafer or substrate used in the semiconductor device industry typically has a diameter of 200 mm, 300 mm, or 450 mm. Examples of wafer materials include silicon (Si), gallium arsenide (GaAs), and silicon germanium (SiGe). Besides semiconductor wafers, other workpieces that may take advantage of the disclosed embodiments include various articles such as printed circuit boards, magnetic recording media, magnetic recording sensors, mirrors, optical elements, display devices or components such as backplanes for pixelated display devices, flat panel displays, micro- mechanical devices and the like. The workpiece may be of various shapes, sizes, and materials.

[0048] “Manufacturing equipment” or “fabrication tool” refers to equipment in which a manufacturing process takes place. Manufacturing equipment may include a processing chamber in which the workpiece resides during processing. Typically, when in use, manufacturing equipment performs one or more electronic device fabrication operations. Examples of manufacturing equipment for semiconductor device fabrication include subtractive process reactors and additive process reactors. Examples of subtractive process reactors include dry etch reactors (e.g., chemical and / or physical etch reactors), wet etch reactors, and ashers. Examples of additive process reactors include chemical vapor deposition reactors, and atomic layer deposition reactors, physical vapor deposition reactors, wet chemical deposition reactors, electroless metal deposition cells, and electroplating cells.LAM1P015WO-11765-1WO

[0049] In various embodiments, a process reactor or other manufacturing equipment includes a tool for holding a substrate during processing. Such tool is often a pedestal or chuck, and these terms are sometimes used herein as a shorthand for referring to all types of substrate holding or supporting tools that are included in manufacturing equipment.

[0050] As used herein and unless otherwise qualified, the term “camera,” “camera sensor,” or “sensor” is not limited to sensors designed or configured to work with a camera. These terms include other multi-pixel radiation sensors, with or without color or multispectral filters, that can provide sensed information that can provide an image of a radiation distribution within a fabrication tool. According to different implementations, optical (including visual) sensors or cameras, or infrared (IR) sensors or cameras, or both may be part of a camera or a sensor.

[0051] The term “image” may refer to a spatial representation of a physical domain including one or more features. An image may be provided in the form of data or signals arranged to represent the physical domain. An image may be produced by a pixelated sensor such as a camera sensor. An image may contain the spatial representation of the physical domain in one dimension, two dimensions, or three dimensions. Multiple images obtained consecutively over time may form a video representation of the physical domain.

[0052] Information about conditions within a process chamber, as captured with a camera sensor or pixelated sensor, may include radiation intensity values as a function of position within the process chamber. In some embodiments, the radiation intensity values are provided as an image. In some embodiments, the radiation intensity values are provided as two-dimensional or three- dimensional pixelated values. In other embodiments, the radiation intensity values are provided in only one dimension such as along a slit or interface between components. In some implementations a one-dimensional sensor or array is configured to scan the interior of a process chamber to generate, e.g., a two-dimensional image of a portion of the process chamber interior.

[0053] Optionally, the intensity values are also provided as a function of wavelength. In some embodiments, a camera sensor or other pixelated sensor comprises separate detection elements, each configured to capture radiation values at a given location, but with different spectral sensitivity profiles; e.g., in the red, green, and blue regions. Some camera sensors are configured to capture radiation intensity values in discrete wavelength ranges that are sometimes referred to as bins. Such sensors include hyperspectral imagers that capture intensity values in narrow wavelength bins and multispectral imagers that capture intensity values over broader wavelength bins. Optionally, the intensity values are provided as function or time; for example, images may be captured as video frames. In some embodiments, multiple camera sensors provide information from different overlapping or contiguous regions within a fabrication tool.LAM1P015WO-11765-1WO

[0054] Figure 1D presents an example of spectral sensitivity ranges for “standard” (visible- sensitive), IR-sensitive, and UV-sensitive sensor elements in a camera sensor or a combination of camera sensors.

[0055] A “semiconductor device fabrication operation” or “fabrication operation” as used herein may refer to an operation performed during fabrication of semiconductor devices. Typically, the overall fabrication process includes multiple semiconductor device fabrication operations, each performed in its own semiconductor fabrication tool such as a plasma reactor, an electroplating cell, a chemical mechanical planarization tool, a wet etch tool, and the like. Categories of semiconductor device fabrication operations include subtractive processes, such as etch processes and planarization processes, and material additive processes, such as deposition processes (e.g., physical vapor deposition, chemical vapor deposition, atomic layer deposition, electrochemical deposition, electroless deposition). In the context of etch processes, a substrate etch process includes processes that etch a mask layer or, more generally, processes that etch any layer of material previously deposited on and / or otherwise residing on a substrate surface. Such an etch process may etch a stack of layers in the substrate.

[0056] In various approaches, materials (such as the above) in stacks of layers (e.g., thin films) may be deposited through chemical vapor deposition (CVD) techniques such as plasma enhanced chemical vapor deposition (PECVD), low pressure chemical vapor deposition (LPCVD), atmospheric pressure chemical vapor deposition (APCVD), metal organic chemical vapor deposition (MOCVD), atomic layer deposition (ALD), plasma enhanced atomic layer deposition (PEALD), or through direct metal deposition (DMD), etc. These examples are not intended to be limiting. Different deposition techniques may be used for different ends. For example, deposition may be useful whenever wafer stress and / or bowing are induced due to material present on the frontside of the wafer. Notably, different deposition techniques may be associated with or subject to different temperatures, pressures, precursors, or other deposition processes, conditions, or recipes. For instance, certain deposition techniques may overlap in temperatures ranges that can be used, which may advantageously be used to strategically deposit multiple layers at once or certain layers only.

[0057] Some conventional techniques for monitoring temperatures involve thermocouples. More specifically, numerous thermocouples (or other temperature-measuring probes) may be disposed about or installed on tools or portions thereof, such as gas delivery lines or pipes. Many can be communicatively connected to software for constant monitoring, which can provide point measurements. In some cases, there may be additional thermocouples that require manual monitoring.LAM1P015WO-11765-1WO

[0058] If a heating element (which may also be referred to herein as a heater) has failed or is in a failing state, the information tied to its failure may not be apparent easily to a user looking at numbers or plots of measurements obtained by the heating element. Also, even if a heating element is failing, the associated system, software, or architecture can compensate by providing higher power to match the proper temperature. Alternately, if a thermocouple or probe is not properly connected to the tool or gas line, the associated system may drive higher power to meet a setpoint. However, this may result in thermal runaway, increasing the heat at beyond the setpoint. Further, in some applications, a deviation in temperature from a designated temperature by a certain acceptable amount (e.g., 10 C) can result in particles due to condensation or accelerated corrosion. This can result in significant installation and warranty (I&W) cost to users of the tool, as well as frequent replacement of faulty parts and cleaning resulting in reduced machine availability. Handheld optical cameras are used at times, but this requires manual data collection and assessment.

[0059] As can be seen, these existing techniques are sensitive to temperature changes and can require constant monitoring and a rapid response time that may not be consistently achievable with manual monitoring methods. Hence, an approach that can preemptively determine or predict a significant temperature deviation at a chemical or gas delivery line (or other heated zones) of a processing apparatus within a reasonable time horizon would more effectively avoid particle issues caused by condensation or accelerated corrosion. Location and Integration of a Sensor in a Processing Apparatus

[0060] Figure 1A shows an example fabrication tool denoted as an example substrate processing apparatus 100. Processing apparatus 100 may be configured for depositing films on or over a semiconductor substrate utilizing any number of processes. For example, processing apparatus 100 may be adapted for performing, in particular, plasma-enhanced chemical vapor deposition (PECVD) or plasma-enhanced atomic layer deposition (PEALD). Processing apparatus 100 may include a camera or camera sensor 117 on a chamber wall. Camera sensor 117 may be configured to capture image data from the interior of processing apparatus 100. Note that while sensor 117 is shown as a single block, it is intended to depict implementations in which one, two, or more camera sensors are located proximate to one another, optionally sharing a single view port or other window into a process chamber or a process station 102. In some cases, the individual camera sensors within block 117 may be trained on different components or fields of vision within a chamber interior. In some cases, the individual camera sensors within block 117 may be configured to capture different spectral ranges (far infrared (IR), near IR, visible, ultraviolet (UV), etc.).LAM1P015WO-11765-1WO

[0061] In some example configurations, processing apparatus 100 of Figure 1A may employ a single process station 102 of a process chamber with a single substrate holder 108 (e.g., a pedestal) in an interior volume, which may be maintained under vacuum by a vacuum pump 118. A showerhead 106 and a gas delivery system 101, which are fluidically coupled to the process chamber, may permit the delivery of film precursors, for example, as well as carrier and / or purge and / or process gases, secondary reactants, etc.

[0062] In Figure 1A, gas delivery system 101 may include a mixing vessel 104 for blending and / or conditioning process gases for delivery to showerhead 106. One or more mixing vessel inlet valves 120 may control introduction of process gases to mixing vessel 104. Particular reactants may be stored in liquid or solid form (e.g., in an ampoule source) prior to vaporization and subsequent delivery to process station 102 of a process chamber. The implementation of Figure 1A may include a vaporization point 103 for vaporizing liquid or solid reactant to be supplied to mixing vessel 104. In some implementations, vaporization point 103 may include a heated liquid injection module. In some other implementations, vaporization point 103 may include a heated vaporizer. In yet other implementations, vaporization point 103 may be eliminated from the process station. In some implementations, a liquid flow controller upstream of vaporization point 103 may be provided for controlling a mass flow of liquid for vaporization and delivery to process station 102.

[0063] Showerhead 106 may operate to distribute process gases and / or reactants (e.g., film precursors) toward a substrate 112 at the process station, the flow of which may be controlled by one or more mechanical valves upstream from the showerhead (e.g., valves 120, 120A, 105). In the implementation depicted in Figure 1A, substrate 112 is depicted as located beneath showerhead 106, and is shown resting on a pedestal 108. Showerhead 106 may include any suitable shape and may include any suitable number and arrangement of ports for distributing process gases to substrate 112. In some implementations involving two or more process stations, gas delivery system 101 may include valves or other flow control structures upstream from the showerhead, which can independently control the flow of process gases and / or reactants to each station so as to, e.g., permit gas flow to one station while prohibiting gas flow to a second station. Furthermore, gas delivery system 101 may be configured to independently control process gases and / or reactants delivered to each station in a multi-station apparatus such that the gas composition provided to different stations is different; e.g., the partial pressure of a gas component may vary between stations at the same time.

[0064] In the implementation of Figure 1A, gas volume 107 is depicted as being located beneath showerhead 106. In some implementations, pedestal 108 may be raised or lowered to exposeLAM1P015WO-11765-1WO substrate 112 to gas volume 107 and / or to vary the size of gas volume 107. The separation between pedestal 108 and showerhead 106 is sometimes referred to as a “gap.” Optionally, pedestal 108 may be lowered and / or raised during portions of the deposition process to modulate process pressure, reactant concentration, etc., within gas volume 107. Showerhead 106 and pedestal 108 are depicted as being electrically coupled to a radio frequency (RF) signal generator 114 and matching network 116 for coupling power to a plasma generator. Thus, showerhead 106 may function as an electrode for coupling radio frequency power into process station 102. RF signal generator 114 and matching network 116 may be operated at any suitable RF power level, which may operate to form plasma having a desired composition of radical species, ions, and electrons. In addition, RF signal generator 114 may provide RF power having more than one frequency component, such as a low-frequency component (e.g., less than about 2 MHz) as well as a high frequency component (e.g., greater than about 2 MHz). In some implementations, plasma ignition and maintenance conditions are controlled with appropriate hardware and / or appropriate machine- or computer-readable instructions in a system controller which may provide control instructions via a sequence of input / output control instructions.

[0065] In general, any plasma-assisted fabrication tool may be used to implement the disclosed embodiments, including integration of a camera sensor configured to capture images of plasmas and / or plasma-related phenomena, or images of or relating to components of the fabrication tool. Example fabrication tools may include deposition apparatuses, which may include, but are not limited to, apparatus from the ALTUS®product family, the VECTOR® product family, and / or the SPEED® product family, the KIYO® product family, the STRIKER® product family, and the VERSYS® product family, each available from Lam Research Corp., of Fremont, California, or any of a variety of other fabrication tools employing plasma.

[0066] For simplicity, processing apparatus 100 is depicted in Figure 1A as a standalone station (102) of a process chamber for maintaining a low-pressure environment. However, some fabrication tools employ a plurality of process stations such as shown in Figure 1B, which schematically depicts an implementation of a multi-station fabrication tool 150, an example of a process apparatus. Fabrication tool 150 may employ a process chamber 165 that includes multiple fabrication process stations, each of which may be used to perform processing operations on a substrate held in a wafer holder, such as pedestal 108 of Figure 1A, at a particular process station. In the implementation of Figure 1B, the process chamber 165 is shown as having four process stations 151, 152, 153, and 154. However, in certain other implementations, multi-station processing apparatuses may have more or fewer process stations depending on the implementation and, for instance, the desired level of parallel wafer processing, size or space constraints, costLAM1P015WO-11765-1WO constraints, etc. Figure 1B additionally shows a substrate handler robot 175, which may operate under the control of a system controller 190, and configured to move substrates from a wafer cassette (not shown in Figure 1B) from loading port 180 and into multi-station process chamber 165, and onto one of process stations 151, 152, 153, and 154.

[0067] As depicted, process station 153 has an associated camera or camera sensor 121 located and configured to obtain images from within process station 153 and, in some embodiments, from within process chamber 154. Process station 151 has two associated cameras or camera sensors 123 and 124. Camera sensor 123 is located and configured to obtain images from within process station 151 and, in some embodiments, from within process chamber 152. Camera sensor 125 is located and configured to obtain images from within process station 151 and, in some embodiments, from within process chamber 153. Process station 152 has an associated camera or camera sensor 127 located and configured to obtain images from within process station 152 and, in some embodiments, from within process chamber 154. Any one or more of camera sensors 121, 123, 125, and 127 may be optically coupled to the interior of process chamber 165 via a view port or other window disposed in or on the chamber wall. Additionally, while not shown in Figure 1B, some embodiments may have one or more camera sensors adjacent to process station 154.

[0068] Fabrication tool 150 may include a system controller 190 configured to control process conditions and hardware states of the fabrication tool 150. System controller 190 may interact with one or more sensors, gas flow subsystems, temperature subsystems, and / or plasma subsystems—collectively represented as block 191—to control process gas flow, thermal conditions, and plasma conditions as appropriate for controlling a fabrication process. System controller 190 and subsystems 191 may act to implement a recipe or other process conditions in the stations of process chamber 165.

[0069] In multi-station fabrication tools, an RF signal generator may be coupled to an RF signal distribution unit, which may be configured to divide the power of the input signal into, for example, four output signals, which may correspond to four respective process stations. Output signals from an RF signal distribution unit may possess similar levels of RF voltage and RF current, which may be conveyed to individual stations of a multi-station fabrication tool.

[0070] Figure 1C provides a top view of an electronic device fabrication system 182 having four quad-station fabrication tools 188, 189, 193, and 195. Each quad-station tool contains four process stations, each configured to hold and process a substrate. At the front end of system 182 are three Front Opening Unified Pods (FOUPs) 183a, 183b, and 183c accessible by a front-end wafer handling robot 185, which may be configured to transfer wafers between the FOUPs and a load lock 187. A first wafer handler 190 may be located and configured to transfer wafers betweenLAM1P015WO-11765-1WO load lock 187 and quad-station fabrication tools 188 and 189. Wafer handler 190 may also be configured to transfer wafers to second load lock 191 that makes wafer available to quad-station fabrication tools 193 and 195 via a second wafer handler 192.

[0071] In an example, quad-station tool 195 may include three cameras 196, 197, and 198 disposed around its outer wall. The cameras are shown vertically affixed to three sides of the four- sided chamber of tool 195. The only side without a camera in this example is the side next to the wafer handler 192. While not shown in Figure 1C, similar camera arrangements can be provided on any one or more of each of the three other quad station chambers in the system. It should be understood that, in some cases, a system controller may be configured to modify a position or orientation of a given camera (e.g., move up, down, left, or right, or rotate with respect to an axis including a central axis).

[0072] Alternatives to the arrangements shown in Figures 1A, 1B, 1C, and 1E, may include (a) cameras on corners of chambers (not necessarily on flat walls) and / or (b) light pipes (rigid or flexible) or optical fibers disposed within the reactor (e.g., between one or more view ports or windows in a chamber wall and one or more locations within a chamber interior). Further, it should be understood that systems employing one, two, or more cameras are not limited to quad- station chambers or even multi-station chambers. As illustrated in the figures, in certain embodiments, a fabrication tool or a station in a fabrication tool may be outfitted with more than one camera. In some cases, a fabrication tool or station has 3 or more cameras, or 5 or more cameras, or 8 or more cameras, or 10 or more cameras. In some embodiments, a station of a multi- station tool may have 1 to 3 camera sensors. In some embodiments, 2, 3, or more camera sensors may share a single window or view port. In some chamber designs, there may not be a view port, or there may be insufficient view ports to accommodate all cameras. In such cases, chamber designs may include a wall region to accommodate one or more camera sensors. In general, there can be any of various combination of a single camera and / or multiple cameras and / or lighting systems that can be placed at different locations, including any of various view ports or other windows.

[0073] The individual camera sensors of a multi-sensor tool or station may be positioned and configured to capture different fields of view within the tool or station interior. In some implementations, different cameras may be located and oriented to capture images of the tool interior at different angles. In some implementations, different cameras are located and oriented to capture images of the tool interior at different translational offsets. In some cases, multiple cameras oriented in such manner may, for example, be arranged to share a single window or view port. In some embodiments, camera sensor analysis logic is configured to stitch or otherwiseLAM1P015WO-11765-1WO combine images from two or more individual camera sensors located and oriented to capture different regions and / or angles within a tool interior. However, as will be detailed below, cameras and sensors (including infrared cameras and sensors) may be positioned to capture images external to the tool.

[0074] As indicated, in embodiments employing multi-chamber fabrication tools, one or more camera sensors may be located and oriented to capture information about two or more chambers. This may be convenient when two or more stations are along a line of sight from a view port or other window outfitted with a camera sensor. In some implementations, camera sensor analysis logic is configured to use information about structural features of adjacent stations such as station walls, or showerheads, or pedestals, to provide context or frame of reference for plasma radiation data collected from a different station.

[0075] In some cases, a fabrication tool may include a lighting system configured to illuminate all or one or more portions of the too interior. In some implementations, a lighting system may be configured to allow a camera to take an illuminated image when the plasma is off (e.g., outside operation or in between pulses). Note that in some cases no lighting system may be employed and lighting may be used from the plasma itself. In some implementations, a lighting system may employ one or more LEDs or other light sources. The light sources may be monochromatic, polychromatic with discrete emission wavelengths, or broad spectrum. The light source may be active continuously, pulsed synchronously with one or more camera shutters, pulsed asynchronously with one or more camera shutters, or pulsed synchronously with other process parameters such as RF generators or gas delivery valves. In other implementations, multiple light sources may be employed in different locations within or outside the chamber. These multiple light sources can be energized continuously or sequentially with timing managed to enable structured lighting to be utilized to construct super-resolution images of features within the chamber. In some implementations, one or more notch or bandpass filters may be provided in front of a light source to produce effects that can support analysis (e.g., identification of particular chemical species by their emission spectra).

[0076] A camera sensor is typically disposed outside of a fabrication tool, although in some embodiments, it may be integrated with a chamber wall or other component or assembly within the chamber. In certain embodiments, a window specially constructed for one or more camera sensors may be integrated into a chamber wall. In some cases, a window may be constructed to permit a lighting system to shine light on the chamber interior to thereby allow the camera to capture images of illuminated chamber components. In certain embodiments, a camera sensorLAM1P015WO-11765-1WO may be optically coupled to an interior of a fabrication tool using an existing view port that is provided in a chamber wall to allow visual inspection of the tool interior.

[0077] In some embodiments, a camera sensor may be directly attached to a wall or window of a fabrication tool. A camera sensor may be affixed to a fabrication tool by various mechanisms such as an adhesive, a bolt or other mechanical fixture, a magnet, etc. In some embodiments, a camera sensor is disposed at a remote location from a fabrication tool. For example, a camera sensor may be optically coupled to a view port via a fiber or other light conduit. Some embodiments allow for a camera to be mounted in a protective enclosure within a fabrication tool interior. In some embodiments, a camera or camera sensor has an associated cooling system or thermal management device or component. Examples of thermal management elements include an insulating material (e.g., rubber gaskets), one or more heat dissipative structures, a flowing liquid heat exchanger, etc.

[0078] A view port or other window for allowing a camera sensor to “view” a chamber interior may be made from any of a variety of materials. Examples include UV fused silica, UV fused quartz, silicon glass, sapphire, borosilicate glass, and calcium fluoride. In some embodiments, a view port or other window may be placed at other components of a processing apparatus, such as at a heat jacket surrounding a clamshell that houses a gas delivery line, as will be detailed further below. In the context of the present disclosure, a “heat jacket” (or a “heater jacket”) may refer to a material or component made of material that may be used to insulate a component (e.g., of a semiconductor processing apparatus). For instance, heat generated from a gas delivery line or a clamshell or other housing may be substantially contained such that the outside surface of a heat jacket may not be as hot as the interior. Thus, the actual temperature of a gas delivery line may be different from what is detected on the exterior, e.g., using a temperature sensor such as a thermocouple(s). The aforementioned type of view port or window may be visually transparent and / or transparent to infrared waves and energy, and may be constructed of a material above. In other embodiments, laminates or composites of multiple materials may be utilized in fabricating the windows. In certain embodiments, the window may be substantially transmissive over a spectral range of about 100- 6000 nm or about 100-1000 nm. To make such a wide spectral range usefully available to a camera, a commercial sensor may need to be modified by removing one or more wavelength-specific or wavelength range-limiting filters on the sensor as manufactured or sold. In some embodiments, a window may include anti-reflective coating to avoid glare from accompanied illumination in the system described elsewhere herein.

[0079] A view port or other window for allowing a camera sensor to view a chamber interior may have any of various sizes and shapes. In certain embodiments, a window has a circular,LAM1P015WO-11765-1WO elliptical, rectangular, or polygonal shape. In some embodiments, a window in a chamber wall is constructed as (or including) an optical element such as a mirror, lens, filter, polarizer, or grating. In some embodiments, a window is integrated with a mirror. Some embodiments further may include optical mirrors or other optical components not integrated with the window but rather located within the chamber to, e.g., enable optical access to regions of the chamber that do not have line-of-sight to a view port or window. In certain embodiments, the window may be a cylindrical piece of sapphire. In certain embodiments, the window may be coated with one or more antireflective films. In certain embodiments, a window may have a maximum cross-sectional dimension (e.g., a diameter or diagonal) of about 5 cm or less, or about 5mm or less.

[0080] In some embodiments, optical elements permit indirect optical information to be captured by a camera sensor. “Indirect” optical information may include image information outside a line of sight of a camera sensor. Indirect optical information may be reflected, refracted, scattered or otherwise directed from its source, which is outside a field of view of the sensor, to a position inside the field of view of the sensor. To this end, a fabrication tool may include a mirror or other optical element configured to direct light or other optical information into a camera sensor’s field of view. In some cases, the optical element is part of a process chamber that has a non-optical function. For example, an aluminum component such as an aluminum chamber wall may reflect light on the IR region of the electromagnetic spectrum.

[0081] View ports have been observed to produce thermal and electrical anomalies. Therefore, eliminating view ports and replacing them with small dimension windows may provide benefits to the fabrication tool processing environment in certain embodiments.

[0082] One or more camera sensors may be arranged to provide a multiplexed processing of images. In some embodiments, a single remote sensor may process optical information (and generate images) from multiple locations (e.g., multiple view ports). For example, a single camera sensor may support multiple view ports. In some embodiments, a fabrication tool may employ one camera sensor to capture image data from two more stations of a multi-station chamber. For example, a tool may have a first window on a chamber wall adjacent a first station and a second window on the chamber wall adjacent a second station. The tool may additionally include a first optical fiber or light pipe optically coupling the first window to a camera sensor and a second optical fiber or light pipe optically coupling the second window to the camera sensor. The camera sensor may be configured to multiplex signals from the first and second optical fibers or light pipes. In some embodiments, the tool includes an array of light pipes and / or an array of optical fibers for conveying optical signals between a source in the tool and a camera sensor.LAM1P015WO-11765-1WO

[0083] Images or video clips from one or more camera sensors may be processed in a multiplexed fashion by image analysis logic running on hardware at any of various locations. In some embodiments, this approach is applied to study states of a fabrication tool or conduct other evaluation outside of real time.

[0084] In some embodiments, one or more fabrication tools and associated cameras may have local edge computers. An edge computer may be configured to execute programs for processing and / or managing camera sensor data. Examples of such programs include image analysis programs and image / video multiplexing programs. In certain embodiments, an edge computer may contain a program for multiplexing image / video data from one or more cameras at a fixed rate. In some cases, an edge computer may be configured to execute one thread for multiplexing video / images from multiple camera sensors and execute a different thread for analyzing images and / or video. An edge computer may run separate virtual machines for its various responsibilities associated with camera sensors.

[0085] In certain embodiments, an edge computer may be provided for a single station, but the computer is configured to multiplex and / or otherwise process data from multiple cameras, some of which may not be located at the single station. Characteristics of Cameras for Sensing Plasma Conditions

[0086] Camera sensors are characterized by various parameters including the number of pixels, range of wavelengths captured, and the like. In some embodiments, a camera sensor for capturing information about a plasma may be capable of sensing intensity values of electromagnetic radiation at wavelengths including at least portion of the UV spectrum, at least a portion of the visible spectrum, at least a portion of the IR spectrum, or any combination thereof. As an example, a camera sensor may be configured to sense intensity values over range including about 100 nm to about 1000 nm.

[0087] As examples for any embodiments herein, camera sensors may be constructed as charge- coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) arrays. In certain embodiments, a camera sensor as used herein may have at least about 5 megapixels or at least about 12 megapixels. In some embodiments, a camera sensor used herein may have as few as about 2 megapixels. In some embodiments, a camera sensor used herein may have more or less.

[0088] In some implementations, an image capture device may be a line or one-dimensional array of sensors or pixels. Such device may be configured to scan across a two-dimensional field of view. The scan direction may be substantially perpendicular to the axis of the line of sensors. In some embodiments, a one-dimensional image capture device may be oriented perpendicular toLAM1P015WO-11765-1WO a wafer or chamber component and optionally configured to scan from one side of the chamber to the other (or within other portion or field of view within the chamber).

[0089] In certain embodiments, a camera as used in any of the embodiments herein is configured with a shutter. In some implementations, a camera may be configured to capture video data of a plasma (or other phenomenon or event) in or at a fabrication tool. In certain embodiments, a camera may be configured to capture video information of a plasma (or other phenomenon or event) in or at a fabrication tool at a frame rate of about 30 to about 120 frames per second (FPS). In some embodiments, a camera may be configured to capture IR images or videos of a component of a fabrication tool, such as an exterior surface of a heat jacket, at a frame rate of, e.g., about 1 to about 30 FPS.

[0090] Some fabrication tools may include a still image or video display. Such display may be employed to allow process engineers or other staff to view the tool interior when a camera sensor or light conduit blocks access to a view port from outside the tool. Display may also allow a tool exterior, such as a surface of a heat jacket of a gas delivery line, or the gas delivery line itself, to be represented visually and / or based on temperature (e.g., infrared temperature or actual temperature) and monitored. In some embodiments, a view to a chamber interior or exterior to the chamber is provided via images or video streamed electronically (e.g., using the real-time streaming protocol (RTSP), the real-time messaging protocol (RTMP), low-latency HTTP live streaming (HLS), secure reliable transport (SRT), WebRTC, or the like), optionally to a remote location via a web application. Examples of remote sites include a fab monitoring room or facility, a smart phone, a tablet, and / or a desktop computer system. In some embodiments, communication of the image(s) or video may be made via network that includes, as a node, a camera on a process chamber. Such network may be wired or wireless, e.g., a mesh network. In certain embodiments, a network employs a protocol employing Wi-Fi, Bluetooth, cellular, etc. Other sensor types that may be used in conjunction with cameras

[0091] In some embodiments, a fabrication tool may include one or more sensors in addition to a camera sensor. Such additional sensor(s) may be configured to sense a plasma or other conditions in situ. Such sensors may include, but are not limited to, mechanical limit sensors, inertial sensors (e.g., accelerometers or gyroscopes), infrared (IR) sensors, acoustic sensors, mass flow sensors, pressure sensors such as the pressure manometers, and temperature sensors such as thermocouples, which may be located in a process gas delivery system, a pedestal, a chuck, etc. Specific examples of additional sensors include current sensors (e.g., VI probes), which may be affixed to one or more structural components such as a showerhead or pedestal, an in situLAM1P015WO-11765-1WO spectroscopic sensor configured to capture emitted radiation from a wafer or reactor component in the UV, visible, and / or IR spectrum (e.g., an optical emission spectroscopy sensor (OES)), an in situ sensor configured to detect optical absorption characteristics of gases in the process chamber, an in situ optical metrology tool such as a reflectometer.

[0092] Another example of an additional sensor may be a capacitive voltage sensor having a relatively high input impedance. Another example of an additional sensor may be an inductive current transformer having a relatively low input impedance that occasionally or periodically samples a current conducted from an RF signal generator without bringing about any significant voltage drop. In some embodiments, a current or voltage sensor may be coupled in series between RF signal generator and a multi-station fabrication tool. Camera Image Analysis

[0093] Image analysis logic may be configured to receive sensed values from one or more camera sensors on a fabrication tool. In certain embodiments, inputs to the image analysis logic may include pixel-by-pixel intensity values as a function an observable parameter such as wavelength, time, location, polarization, or any combination thereof. In certain embodiments, input data from a camera sensor may be provided in the form of image data, video data, spectral values, time series data, wafer metrology data, or the like. In some embodiments, the input data may be filtered by wavelength, polarization, etc. In some embodiments, analysis logic may be configured to receive and act on additional input information beyond camera sensor intensity data. Such additional input information may include metadata about the camera sensor and / or associated camera components, substrate metrology information, historical information about the fabrication tool, etc.

[0094] The analysis logic may be configured to output one or more properties of plasma in a fabrication tool and / or a classification of a state of the fabrication tool or a component thereof. Some examples of plasma properties were presented above. Some examples of a state of the fabrication tool or a component thereof may include temperature of a heat jacket, gas delivery line, or a heating element. In some embodiments, the analysis logic may be configured as a classifier for diagnostic purposes, or for predictive purposes, or for control purposes. Some examples of diagnostic classifications may include fault detection and anomalous conditions. Some examples of predictive classifications may include process or mechanical drift (e.g., a varying shape of a showerhead or other component) and associated predictive maintenance (generated by, e.g., regression analysis). Further examples are provided in International Application No. PCT / US2021 / 058550, entitled “PREDICTIVE MAINTENANCE FOR SEMICONDUCTORLAM1P015WO-11765-1WO MANUFACTURING EQUIPMENT” and filed November 9, 2021, which is assigned to the assignee hereof and incorporated herein by reference in its entirety. Examples of control classifications may include recommended modifications to apparatus or processes.

[0095] Camera sensor analysis logic may comprise any of various types of classifiers or models such as deep neural networks (e.g., convolutional neural networks, autoencoders, Unet, etc.), traditional or classical computer vision methods such as edge detection, image modification (such as blurring, changing contrast), intensity thresholding, color channel thresholding, etc.

[0096] Analysis logic may be configured to perform an image processing routine such as a segmentation or other edge finding routine. In certain embodiments, analysis logic may be configured to use segmentation or other edge detection method to determine plasma property information relative to a system component. Segmentation can isolate the component. For example, identification of showerhead and detection of plasma properties some distance away from it may employ segmentation. In the field, when mechanical mounting may not always be consistent, segmentation can help minimize errors (as opposed to fixing the (x, y) location for all images / video generated by a fleet of tools).

[0097] The logic may employ any of various techniques for edge detection or segmentation. For example, the logic may employ a threshold-based method, a deep learning model, etc. In some embodiments, the edge of a plasma or the boundary of a subregion within a plasma having defined plasma characteristic may be determined using a processing sequence such as the following: (a) data reduction, (b) denoising (e.g. gaussian blur), and (c) edge finding / thresholding (e.g., a Canny sequence of filter).

[0098] In some embodiments, image analysis logic may be configured to determine a location of a plasma based on, at least partly, a centroid of a region occupied by the plasma. A plasma’s centroid may be a geometric centroid determined from a region deemed by analysis logic to be within boundaries of the plasma. In some cases, the centroid may be calculated by considering the intensity of radiation within an image of the plasma. This may be implemented by weighting pixels or regions of an image based on, at least, the intensity values of the pixels or regions. A plasma’s centroid may also be determined by simply applying an intensity threshold to the pixel or region values, and only considering those pixels or regions having intensity values above the threshold when calculating the centroid.

[0099] Analysis and / or control logic may employ a plasma centroid to determine the alignment or tilt of the process chamber, the showerhead, the pedestal, and / or other component. For example, the logic may compare the centroid position of a process chamber under consideration to anLAM1P015WO-11765-1WO expected or baseline centroid position for a properly aligned process chamber or process chamber component. If the centroids do not agree to within a defined tolerance, the logic may flag the current system as being out of alignment.

[0100] In some embodiments, image analysis logic may be configured to determine a location of a plasma by integrating or summing optical intensity values over a bounded region of interest within a field of view of the first camera sensor. Examples of regions of interest include a gap between a pedestal and showerhead or a subregion of the gap such as shown in the figures.

[0101] In some cases, analysis logic may be configured to determine a plasma’s location based on, at least, a point or boundary of a plasma having a defined spectral characteristic. The defined spectral characteristic may be a region of the electromagnetic (EM) spectrum associated with a gas or component in the process chamber. The spectral region may be associated with an emission spectrum and / or one or more emission lines of a gas or component in the process chamber. A spectrally limited location of the plasma may be used to identify the composition and / or location of a gas within a process chamber or a station thereof. For example, if gas of a particular composition is to be located only above a showerhead and it is found to be located in the gap between a pedestal and showerhead, the analysis logic may flag the process chamber or the current process as needing inspection or modification.

[0102] In certain embodiments, analysis logic may be configured to perform streaming data analysis. As view ports are replaced with cameras, engineers will require a “window” to the fabrication tool interior. In certain embodiments, an image or video processing system is configured to provide access to live stream of data which may be accessed through a fabrication tool’s computer, local wireless streams to cellphone apps, and other intranet channels accessible via, e.g., a browser. In some embodiments, analysis logic is configured to perform analytics such as component segmentation and classification in real time. As an example, analysis logic may be configured to flag faults such as a stuck pin in a wafer support. The logic may ensure that such issues are addressed before, e.g., processing another wafer.

[0103] In certain embodiments, analysis logic may be configured to perform fixed frame (image) and / or video analysis. As explained elsewhere herein, the analysis logic may be configured to analyze and interpret where static image or multiple temporal frames depending on the use case.

[0104] In certain embodiments, analysis logic may be configured to with edge computing capabilities. To minimize network traffic, at least some computation may be performed on an edge node (or multiple edge nodes) and only limited data is transferred to remote computational orLAM1P015WO-11765-1WO memory resources (e.g., remote storage databases). Decisions on control, feedback, warnings, and the like may be based on computing capabilities on the edge nodes.

[0105] In certain embodiments, analysis logic may be configured to perform feed forward and / or feedback for multiple applications (process control, auto calibrations, hardware adjustments, etc.). In certain embodiments, computational resources may be configured to feed results of a camera analysis to a controller of process conditions at a subsequent (downstream) tool. For example, analysis may be conducted on plasma images captured while a wafer is being processed in a single station plasma tool. The results of analysis may be used to control conditions in an adjacent module for additional deposition. As another examples, information on non-uniformity on station 1 can be used to compensate process conditions at station 2, 3 or 4 (or another), or in the current or subsequent process steps.

[0106] In certain embodiments, analysis logic may be configured to perform multiplexing and / or stitching of images. Multiplexing (e.g., multi-threaded processing) may allow a single processor to process images from more than one camera. Multiplexing may allow a single processor to handle images from multiple wavelength ranges. In certain embodiments, multiple cameras may be used to generate not only a rich stream of temporal data but also spatial locations from multiple perspectives. While there may be some delays due to switching from camera to camera, for some applications, analysis logic can generate a combined image capturing more than just the field of view of one camera. In some embodiments, analysis logic may be configured to reconstruct three- dimensional information by combining additional sensors and cameras (and optionally modeling data).

[0107] Note that the foregoing imaging technologies can be used with thermal imaging with at least equal effectiveness, as will be described in greater detail below. Emissivity in Thermal Imaging

[0108] Emissivity, in some cases, can be defined as a dimensionless number that measures an object’s ability to emit energy as thermal radiation, such as infrared energy. More specifically, emissivity can be defined as a ratio of the energy radiated from a material’s surface to the energy radiated from a perfect emitter, known as a blackbody. Emissivity values can range from 0 (for a perfect reflector) and 1 (for a blackbody). Information regarding a material’s emissivity may be known. How well a material’s surface can emit infrared energy can depend on several factors. For example, the emissivity of a material’s surface depends at least in part on the type of material. That is, different materials have corresponding emissivity values. The emissivity of a surface may also depend on the nature of the surface, such as surface quality or oxidization. For example, a cleanLAM1P015WO-11765-1WO and polished metal surface may have a low emissivity, whereas a roughened and oxidized metal surface may have a high emissivity. A material with a high emissivity value absorbs more reflected or ambient infrared energy and emits only its own infrared radiation. Low emissivity refers to a surface that emits low levels of radiant thermal energy. Emissivity can also vary depending on the material’s temperature.

[0109] As examples of how emissivity can vary depending on the conditions, unoxidized aluminum has an emissivity of 0.02 at 77 °F (25 °C), 0.03 at 212 °F (100 °C), and 0.06 at 932 °F (500 °C). Oxidized aluminum has an emissivity of 0.11 at 390 °F (199 °C), 0.19 at 1110 °F (599 °C), 0.20 at 200 °F (93 °C), and 0.31 at 940 °F (504 °C). Highly polished aluminum has an emissivity of 0.09 at 212 °F (100 °C), while roughly polished aluminum has an emissivity of 0.18 at 212 °F (100 °C). On the other hand, an average emissivity of fiberglass insulation (which a heat jacket may be constructed of) may be about 0.75 to 0.85. Hence, generally speaking, the relatively low emissivity coefficients make aluminum suitable for limiting the radiated heat from an object, while relatively high emissivity coefficients make a component such as a heat jacket made of fiberglass suitable for monitoring infrared temperatures.

[0110] In some embodiments, infrared data may be obtained using one or more camera sensors, such as one or more infrared sensors. An infrared sensor or camera may be disposed in a view port of a process station, which may be on a side wall of the station, a top wall of the station, or any other suitable location (e.g., any vantage point that has a field of view to the processing apparatus or a portion thereof), as discussed above. A process station may have multiple (e.g., two, three, five, ten, or more) infrared sensors or cameras. In some embodiments, the process station may be part of a process chamber having multiple (e.g., two, three, four, eight, ten, or more) process stations, although it should be noted the techniques described herein may be practiced on a process chamber having multiple stations or a single station. In some embodiments, the infrared data may be captured as raw voltage signals and transformed into a two-dimensional representation of infrared temperature. As used herein, “infrared temperature” refers to an estimated temperature based on the infrared data that is based on the infrared energy indicated in the infrared data, which may or may not necessarily account for the emissivity of surfaces represented in the infrared data. Because the infrared data may be represented as pixels of an image, each pixel indicating an infrared temperature of a region corresponding to the pixel, infrared temperatures may be determined for thousands or tens of thousands of pixels or points. Inverse Planck’s Law and Radiated Power Density Planck’s Law may be used to relate wavelength of a radiation (which may be included in the infrared data) and estimate a temperature based thereon. The material being measured may also be a factor in the temperature estimation. Estimating infrared temperaturesLAM1P015WO-11765-1WO may thus be calibration-based estimations. In comparison, temperature information using physical thermocouples disposed in the station may yield temperature data for a very limited region of a station or a component of a fabrication tool (e.g., the region immediately around the thermocouple). Accordingly, the infrared data obtained and used herein may allow for substantially increased spatial resolution with which temperature information and / or film thickness information is determined, which may in turn enable high spatial resolution in detecting anomalies, in a fabrication process and / or a component of a process station.

[0111] In some implementations, the techniques disclosed herein may utilize a relationship between infrared temperature associated with a surface, emissivity of the surface, and an actual temperature of the surface. In some embodiments, the relationship between these three parameters may be learned, e.g., by a machine learning model, using experimental data. In some embodiments, temperature information associated with a component of a process station (e.g., gas delivery line or surrounding parts such as heating element or heat jacket) and / or a surface of a wafer may be determined by determining an infrared temperature based on infrared data, and estimating an emissivity of the surface imaged. In some approaches, the emissivity may be determined based on known material properties and / or based on experimental data. In some approaches, the difference between the infrared temperature and the actual temperature (or “ground truth” temperature) may be due to the emissivity of the surface, and the relationship among these data points can reveal the emissivity, e.g., using extrapolation, or inference of a machine learning model.

[0112] Accordingly, the emissivity may be determined based at least in part on the difference between the infrared temperature (and / or changes in the infrared temperature over time) and the ground truth actual temperature, and, in some embodiments, based on known material properties of the surface. Since the infrared temperature and the actual temperature of a component or material are typically not the same in many configurations (e.g., since a heat jacket typically insulates a gas line and heating element), a high statistical correlation between two may exist (with an R2 value close to 1 (e.g., above 0.99, above 0.95), particularly in temperature measurements in the 25–650 C range), which may be used to determine one knowing the other. In some cases, the difference between the infrared temperature and the actual temperature may stay substantially the same over time (e.g., after failure of a heating element associate with a gas delivery line), or stay within a certain range, which allows derivation of the actual temperature more accurate when determined relative to the infrared temperature. In some cases, over time (e.g., during the course of a fabrication process, over time spanning use of the process station, etc.), the difference between the infrared temperature and the actual temperature may increase as a result of the process, e.g.,LAM1P015WO-11765-1WO because of increasing film thickness. Using the infrared temperature and the emissivity, the actual temperature of the surface (e.g., a surface of a heat jacket, a surface of a pedestal, a surface of a wafer) may be determined using the relationship between the infrared temperature, emissivity, and actual temperature.

[0113] In some embodiments, determining the actual temperature (associated with, e.g., a component of a processing apparatus such as a gas delivery line) may involve providing the infrared temperature (e.g., obtained using one or more infrared sensors) and / or the emissivity data to a trained machine learning model configured to output an actual temperature based on the infrared temperature and / or the emissivity data. In some approaches, the machine learning model may have been trained on known experimental or derived data that includes ground truth actual temperature, measured infrared temperature, and emissivity of a material or component being measured. In some instances, the model may be trained using experimental training data in which actual temperature is measured using one or more thermocouples, thermistors, or the like, and infrared temperature is measured using one or more infrared sensors. Techniques such as gradient descent using a loss function, and backpropagation, using a set learning rate and / or regularization rate, for instance, may be used with the model. As another example, in some embodiments, determining the temperature information may involve using a lookup table or graph or pattern that associates infrared temperature, emissivity, and / or actual temperature for a component or type of material.

[0114] In some embodiments, for a given pixel or point (e.g., in an image representative of a component of a processing apparatus), temperature information may comprise actual temperature that is determined based on an infrared temperature and an emissivity for that pixel or point. For example, as mentioned above, the infrared temperature and the emissivity may be provided to a trained machine learning model to determine the corresponding actual temperature. As another example, the infrared temperature and the emissivity may be used as keys to a lookup table to determine the corresponding actual temperature. Note that, in some implementations, a lookup table may be generated using a trained machine learning model. In instances in which a trained machine learning model is utilized, the model may be trained using experimental training data in which actual temperature is measured using one or more thermocouples, thermistors, or the like, and infrared temperature is measured using one or more infrared sensors. In some embodiments, emissivity may be determined using known material properties and / or spectral or wavelength information. For example, emissivity of a bare silicon wafer may be determined based on known properties of silicon. As another example, known properties of silicon oxide, which may form during a fabrication process, may be used to determine emissivity information during a fabricationLAM1P015WO-11765-1WO process. Note that, in some embodiments, emissivity may be a global emissivity where the same emissivity value is applied to all pixels or regions. Conversely, in some embodiments, emissivity may be a local emissivity where emissivity is dependent on an angle, distance, and / or location of the infrared sensors or cameras. For example, emissivity may vary for shallow camera angles relative to increased camera angles (e.g., a viewpoint that is closer to top-down). In some embodiments, correction for infrared camera or sensor angle may be based on experimental data obtained using infrared sensors or cameras disposed at a particular angle and measurements of actual or ground truth temperatures. The actual temperatures may be used to determine the effect of the angle on emissivity at different regions or pixels.

[0115] In some cases, the emissivity data may be determined based on known material properties of components of the station (and / or of the wafer in some cases) at a particular point of a fabrication process. For example, during a fabrication process, emissivity associated with the surface of a heat jacket of a gas line may be determined based on known properties of the material being measured (e.g., heat jacket), which may vary as a result of a fabrication process of the material. As described above, the estimation of infrared temperatures may depend on the type of material being measured. So, emissivity may depend on the fabrication process of the material as well. In addition, in some configurations, local emissivity data (e.g., regions of the same material having different emissivity values) may correct for the angle of the infrared sensor(s) or cameras at different regions of the infrared temperature image, and in cases where change in emissivity occurs over time, different parts of a process.

[0116] In some embodiments, actual temperature may be determined on a pixel-by-pixel basis for a two-dimensional image representing infrared temperatures. For example, for a given pixel, the infrared temperature and the emissivity value corresponding to the pixel may be used to determine the actual temperature. In some implementations, the infrared temperatures may be indicated or visualized using different colors, shapes representing particular pixels, heatmap, topographic map with different types of line indicating boundaries or regions of temperatures, etc. Temperature Monitoring in Chemical Delivery Lines

[0117] As noted above, a component of interest in a process apparatus or fabrication tool may include chemical delivery lines such as gas delivery lines. In such tools, a complex system of multiple gas delivery lines connected between, e.g., a source and a process chamber, may be used with the process apparatus. Failure may occur at any point within that complex system even with traditional thermocouples, and thus, a way to monitor the temperature and / or predict failure toLAM1P015WO-11765-1WO enable a rapid response would be useful for preempting aforementioned problems such as precursor condensation, corrosion, particle issues, etc.

[0118] FIG.2A is a simulated schematic illustrating an example portion of a chemical delivery system 200, according to some embodiments. In some configurations, the example portion of the chemical delivery system 200 may include a chemical delivery line 201 (including a heat jacket along a portion 210), a gas inlet 203, an exhaust 205 (e.g., foreline of a process chamber), and one or more supporting structures 207 (e.g., stand and / or base structure) for the foregoing components. the chemical delivery line 201 may include a gas delivery line or gas pipe configured to carry chemicals, such as precursor gases, reactants, carrier gases, etc. to a destination such as a process chamber or process station. The gas inlet 203 may provide a flow of air and / or gas species toward the chemical delivery line 201 (at a temperature set by a controller, such as a mass flow controller (MFC)). In some cases, the flow may include carrier gases or reacting chemicals (e.g., precursor gases). The exhaust 205 may serve as an outlet for air or at least some of the flowing contents.

[0119] FIG. 2B depicts a portion of an example chemical delivery line 201 and its associated components. In some embodiments, the example chemical delivery line 201 may include a gas delivery pipe 202 at an inner portion of the example chemical delivery line 201. In some implementations, the gas delivery pipe 202 may be one of multiple conduits connecting one or more sources of chemicals to a process chamber or process station. At least one heating element 204 may be disposed proximate to the gas delivery pipe 202. As with the gas delivery pipe 202, the at least one heating element 204 may be an elongated metallic element configured to and positioned (e.g., parallel) to heat at least a portion of the gas delivery pipe 202. The clamshell 206 may provide a structure for housing and holding the at least one heating element 204 and the gas delivery pipe 202 in place relative to each other (e.g., at a fixed distance from each other). At least portions of gas delivery pipe 202 and clamshell 206 may be sheathed in or surrounded by a heat jacket 208, which may be a thermally insulated structure (e.g., fiberglass) that provides insulation of and retain much of the heat generated by the at least one heating element 204 and applied to the gas delivery pipe 202, thereby preventing rapid loss of heat to the surroundings. Further, the heat jacket 208 may sheathe or surround at least portions of the gas delivery pipe 202, the at least one heating element 204, and the clamshell 206, providing further insulation and physical integrity to the gas delivery pipe 202. In some embodiments, the clamshell 206 may be constructed of aluminum, and the surface of the heat jacket 208 may have an emissivity value associated with the thermally insulated material used for construction (e.g., fiberglass). In some cases, the emissivity value may be assumed to be a certain value, or identified based on the process or other condition(s).LAM1P015WO-11765-1WO

[0120] FIG.2C depicts a cross-sectional view of the example chemical delivery line 201 and its associated components. At least one heating element 204 may be disposed proximate to a gas delivery pipe 202. A clamshell 206 may provide a structure for the at least one heating element 204 and the gas delivery pipe 202. A heat jacket 208 may surround the foregoing components.

[0121] FIG.2D depicts a cross-sectional view of the interior of the example chemical delivery line 201 and its associated components. It can be seen that at least one cavity 212 may be disposed within the clamshell 206, which may provide a volume or space to allow heating of gas flowing through the gas delivery pipe 202. There may also be a location (such as a slot 221) for a thermocouple to obtain temperature measurements of the gas flowing through the gas delivery pipe 202.

[0122] FIG. 2E depicts a perspective view of the clamshell 206 used with a processing tool designed according to the example portion of chemical delivery line 201. In some embodiments, the clamshell 206 may be an aluminum clamshell. As seen in FIG. 2E, various locations labeled T1 through T4 indicate positions of thermocouples that may be used with the heat jacket 208 and the example chemical delivery line 201. For instance, T1 may be a location for a thermocouple that is in direct access with the gas delivery pipe 202; T2 may be a location for a thermocouple placed near the surface of the heat jacket 208, above the gas delivery pipe 202; T3 may be a location for a thermocouple placed near the surface of the heat jacket 208, below the gas delivery pipe 202; and T4 may be a location for a thermocouple placed near the surface of the heat jacket 208, above the cavity (e.g., 212). Each of these locations may be a slot (such as 221 described with respect to FIG. 2D) through which temperature measurements may be obtained, e.g., using a thermocouple.

[0123] It is a goal to determine and monitor that the temperature of the gas delivery pipe 202 is not deviated from the intended or designated temperature more than a threshold amount, for example, more than or less than 10 C, in order to prevent aforementioned issues. For example, if the temperature of the gas delivery pipe 202 is more than 10 C higher than the appropriate temperature, it may lead to higher risk of corrosion of the gas delivery pipe 202. Conversely, if the temperature of the gas delivery pipe 202 is more than 10 C lower than the appropriate temperature, it may lead to higher risk of precursor condensation and particles.

[0124] To monitor a chemical delivery system configuration such as that shown in FIGS. 2A– 2E, changes in temperature of the heat jacket 208 (ΔTHeat_Jacket) may be detected before changes in temperature of the gas delivery pipe 202 (ΔTGas_Line), by virtue of measuring devices and equipment such as IR sensors and thermocouples being positioned outside the gas delivery pipeLAM1P015WO-11765-1WO 202. For instance, one or more IR sensors may be placed at a predetermined angle and a distance with respect to the heat jacket 208.

[0125] In some embodiments, not all of the components of the chemical delivery line 201 may be used. In some configurations, gas delivery pipe 202 and clamshell 206 may be exposed directly to its surroundings and environment without a heat jacket.

[0126] FIG. 3 is a diagram of an example configuration 300 of an infrared (IR) sensor 301 disposed with respect to a portion 310 of a gas delivery line 302, according to some embodiments. The portion 310 may include at least section of a gas delivery line 302 along with one or more heating elements 304, a clamshell 306, and a heat jacket 308 disposed with respect to the gas delivery line 302. The gas delivery line 302 may be an example of the gas delivery pipe 202, heating element 304 may be an example of heating element 204, clamshell 306 may be an example of clamshell 206, heat jacket 308 may be an example of heat jacket 208, and portion 310 may be an example of portion 210. Portion 310 may generally refer to any portion of interest or section of the gas delivery line 302 or any gas line (or a component) in the chemical delivery system using the gas delivery line 302.

[0127] Since the heat jacket 308 and the clamshell 306 sheathes and surrounds parts of the gas delivery line 302 there is no direct line of sight to the part of the gas delivery line 302 that is heated by the one or more heating elements 304, it is an objective of the present disclosure to obtain infrared temperatures associated with the surface of heat jacket 308. Hence, in some implementations, IR sensor 301 may be configured and positioned to obtain IR signals and IR data, which may be used to determine infrared temperatures of at least the heat jacket 308 and thereby estimate actual temperature of the gas delivery line 302 based on the infrared temperatures. Such estimation may be made using a correlation or relationship between the infrared temperature (of the heat jacket 308) and the actual temperature (of the gas delivery line 302).

[0128] In some implementations, an infrared-transparent window may be embedded on the heat jacket 308. Such a window may be constructed of silicon glass. Unlike thermally insulated surfaces of the heat jacket 308, the infrared-transparent window may be transparent to infrared waves and energy, allowing them to travel through the window. As such, the infrared-transparent window may enable a more direct measurement from the clamshell 306, and prediction of a temperature of the gas delivery line 302.

[0129] Advantageously, this renders the use of thermocouples and manual operation of thermocouples optional (although they may be used for confirmation, validate, or corroborate the estimated temperatures), and can enable predictions of temperature changes and disparities, as willLAM1P015WO-11765-1WO be explained with respect to FIGS. 4 and 5 below. In addition, infrared and actual temperatures can be mapped over the entire chemical delivery system, which in some examples may be visualized as a heatmap or other image or data structure that is spatially or volumetrically (three- dimensionally) representative of measured or estimated temperatures (of the heat jacket, gas delivery line, or other components), temperature differentials (e.g., between heat jacket and gas delivery line), rates of temperature change, etc. Moreover, such information can be determined and / or represented on a pixel-by-pixel basis, e.g., intensity values representing temperature or associated data as a function of location and / or emissivity. In some implementations, values, ranges, or statistical values thereof (e.g., mean, median) for points, pixels, or regions of interest may be observed or monitored against a threshold (such as those discussed with respect to FIGS. 4 and 5 below).

[0130] In some configurations, the IR sensor 301 may be disposed relative to the heat jacket 308 at a prescribed angle (θ) and a prescribed distance (d). For example, d may be 10 inches, and θ may be 35 degrees. Θ and d may be any feasible value to obtain reliable IR data; e.g., θ may be anywhere from -180 to 180 degrees or -90 to 90 degrees relative to a horizontal axis 311 associated with the IR sensor 301 (where negative angles cause the IR sensor 301 to point upward). In some configurations, the angle and / or distance may be selected such that it is similar to or within a range as the viewing angle of another sensor, e.g., a camera that is set up in some orientation with respect to the process apparatus using the gas delivery line 302 (including at a view port or external to the process apparatus), or one or more other IR sensors disposed along the gas delivery line 302 or other gas lines. In some configurations, the angle and / or distance may be selected based on experimental findings that are determined to result in accurate, precise, or otherwise reliable or useful. In some configurations, the angle and / or distance may be selected based on hardware reliability; e.g., cables or wires may be prone to disconnection or malfunction in certain positions or angles and may not be used. In some configurations, the angle and / or distance may be selected based on other parameters such as emissivity of the heat jacket 308.

[0131] It will be understood that more than one IR sensor may be used to obtain IR signals and data about the gas delivery line 302 or portion 310 thereof. More than one IR sensor may be used throughout the chemical delivery system using the gas delivery line 302. In fact, in some chemical delivery systems, there may be 25, 30, 50 portions, heating elements, etc. Some or all of these portions may have at least one IR sensor measuring infrared temperatures (and / or thermocouples or other temperature-measuring devices measuring temperatures), such that multiple IR sensors are disposed throughout the chemical delivery system to monitor temperatures along one or more gas delivery lines. Some or all of these multiple IR sensors may be configured to be communicativeLAM1P015WO-11765-1WO (to perform, e.g., data communication via physical coupling or wirelessly) with one another and / or at least one controller or other hardware or software associated with the chemical delivery system.

[0132] Note that in some scenarios not illustrated herein, other components of a processing apparatus, tool, or system may be monitored. For example, the temperature downstream of a remote plasma clean (RPC) source may be monitored. The RPC source may generate fluorine (F) radicals, which are transported as atomic or molecular gas to help clean the processing apparatus, tool, or system. In some approaches, heat and an infrared temperature of an RPC source or changes thereto caused by an exothermic reaction at the RPC source may be measured at a surface of a gas line of the RPC source (which may not be protected with its own heat jacket or clamshell). The measured infrared temperature of the RPC source may be correlated to the actual temperature of the RPC source, and an estimated temperature of the RPC source can be determined. If the RPC source were deteriorating, for example, the reaction rate of the exothermic reaction creating heat may decrease, creating less heat in steady state, and the measured infrared temperature (and the actual temperature) of the RPC source may decrease accordingly. In some approaches, infrared temperatures of the surface of the heat jacket (e.g., 208 or 308) of a gas delivery line can be measured and correlated to the actual temperature of the RPC source. The temperature of the gas delivery line (and / or clamshell and heat jacket) may be influenced by and thus correlate to the temperature measurable with respect to the infrared temperature of the RPC source. In some cases, the correlation may be determined through analysis of prior measurements of actual temperatures of the RPC source and infrared temperatures of another component (e.g., gas delivery line or its heat jacket or clamshell), including using linear extrapolation or a supervised learning algorithm (e.g., linear regression to train a machine learning model). This correlation determination is similar with correlating the infrared temperatures and actual temperatures of the gas delivery line and its components, as will be further discussed in detail below (for example, with respect to FIGS. 8A and 8B). Monitoring the health of the RPC source (or other components) can thus be done indirectly based on infrared temperatures of the RPC source (or, e.g., of the surface of the heat jacket of the gas delivery line) that correlate to the heat generation at the RPC source. Thus, the present disclosure is not limited to measuring the temperature of a chemical delivery line and can be used to monitor any physical component of a system based on thermal-based measurements, e.g., using at least one IR sensor 301. Further, the techniques described below with respect to correlation of infrared temperature, actual temperature, and / or emissivity, and / or variations or changes thereof may be used with similar effectiveness when applied to various components of the system. While some components (e.g., non-insulated components) may not have an internal temperature that is significantly different from an externally measured infrared temperature, in some cases, infrared temperatures can still be correlated to temperatures at other locations. ForLAM1P015WO-11765-1WO instance, the infrared temperature of an RPC source may be monitored directly and also used to estimate a temperature of a component that does not have a direct line of sight, either because there are other components such as chemical delivery lines, wires, or walls, or because the interior of the component is covered by insulation or a housing.

[0133] Another example of a component of the processing apparatus, tool, or system which may be monitored may include a foreline where gases (e.g., precursors) leave the system. The foreline may have issues similar to those of precursor or chemical delivery lines, such as condensation. There may also be an issue of backflow upstream into the chamber as a result of a process imbalance, e.g., if temperatures are too high or low or a pressure imbalance (different pressures in chamber, foreline, etc.). Other examples may include hot and / or cold water lines. A clogged water line may not flow heat effectively and could increase the temperature locally (and thereby detectable using the concepts described in the present disclosure.

[0134] FIG.4 is a graph of example temperature values of a gas delivery line (TGas_Line) 402 and example temperature values of a heat jacket (THeat_Jacket) 404 when power is applied, e.g., to a heating element of the gas delivery line. As can be noticed, there is a significant temperature disparity 406 between the two sets of example temperatures, particularly over time, the temperatures may saturate to a steady state (or substantially so), where the temperature may reach a setpoint (or a range) from a starting point (e.g., room temperature). In some implementations, a controller may maintain the temperature. The example temperature values of the gas delivery line 402 and the example temperature values of a heat jacket 404 may be actual temperatures, e.g., obtained experimentally using thermocouples. However, similar temperature changes may be seen where infrared temperature values are obtained for the heat jacket, e.g., using one or more IR sensors such as IR sensor 301.

[0135] FIG. 5 is, conversely, a graph of example temperature values of a gas delivery line (TGas_Line) 502 and example temperature values of a heat jacket (THeat_Jacket) 504 when power is reduced. In this example, there is a 50% power reduction.

[0136] In either scenario, a correlation or relationship between TGas_Lineand THeat_Jacketmay exist. Experimental observation or simulation and approximation may be one approach to determine such correlation or relationship.

[0137] For example, referring to FIG. 5, the change (reduction) of heat jacket temperature (ΔTHeat_Jacket) and the rate of change of heat jacket temperature (ΔΔTHeat_Jacket) over time may be determined via measurements at different conditions. In certain cases, ΔTHeat_Jacket(505) may be a reduction of about 1.70 – 2.00 C over respective time periods spanning about 100 – 410 secondsLAM1P015WO-11765-1WO in order to reach a ΔTGas_Line (506) reduction of 10 C of gas delivery line temperature 502. In addition, ΔΔTHeat_Jacketmay be about 0.01 C / s. Under specific conditions, such as how much power is reduced (simulating a power reduction or heater inefficiency) or added (simulating a power surge or an unresponsive or disconnected thermocouple) to a heating element, ΔTHeat_Jacketand ΔΔTHeat_Jacket may vary for a given ΔTGas_Line. For example, in order to reach a ΔTGas_Line (506) reduction of 5 C, ΔTHeat_Jacketmay be between about 0.80 – 1.10 C over respective time periods spanning about 60 – 200 seconds, and ΔΔTHeat_Jacket may be greater than 0.01 C / s (e.g., 0.017 C / s) or less than 0.01 C / s (e.g., 0.004 C / s). In order to reach a ΔTGas_Line(506) reduction of 5 C at 50% power reduction, for instance, it may take about 70 seconds for ΔTHeat_Jacket to change by about 0.89 C, which corresponds to a ΔΔTHeat_Jacketrate of about 0.0127 C / s. Similar information relating to other values of ΔTGas_Line (506) (e.g., 3, 7, 13, 15 C) may be determined in similar fashion.

[0138] In some embodiments, temperature (e.g., infrared temperature) of the heat jacket (THeat_Jacket) and / or changes thereof (ΔTHeat_Jacketor ΔΔTHeat_Jacket) may be monitored (e.g., using one or more IR sensors such as IR sensor 301) to estimate a temperature of the gas delivery line (TGas_Line) and / or changes thereof (ΔTGas_Lineor ΔΔTGas_Line). In some implementations, estimation of TGas_Line based on THeat_Jacket may be performed by correlation (using, e.g., information obtained using above approaches, which may be obtained and stored in a data structure such as a lookup table, comma-separated value (CSV) file, etc.) or curve fitting between the information obtained using above approaches (linear, exponential, etc.). In some implementations, estimation of TGas_Linebased on THeat_Jacket may be performed using a trained machine learning model (trained using techniques described above at least on, e.g., the information obtained using above approaches).

[0139] Monitoring the heat jacket temperatures can allow determination of whether the temperature of the gas delivery line or changes thereof (ΔTGas_Line or ΔΔTGas_Line) breaches a threshold, so as to keep ΔTGas_Line under the threshold or cause an alert or shutdown when a certain threshold is breached. Multiple thresholds or sub-thresholds may be set (e.g., 3 C, 5 C, 10 C). If a threshold is breached, one or more warnings may be issued, and in some cases, a shutdown of the process apparatus or a component thereof (e.g., the gas delivery line, heating element). For instance, a first warning may be issued at a ΔTGas_Line of 3 C, a second warning may be issued at a ΔTGas_Line of 5 C, and a shutdown may occur at a ΔTGas_Line of 10 C. In some examples, as discussed above, a final threshold of 10 C may be set, while in other approaches, a final threshold of 5 C or 15 C (or other values) may be set, along with any desired sub-thresholds under said final threshold. It will be appreciated that any threshold(s) may be set depending on the desired sensitivity and application of the process apparatus using the gas delivery line. The threshold(s) may alternativelyLAM1P015WO-11765-1WO or additionally be dependent on gas species and material of the gas line; e.g., the threshold may be 20 C in some cases.

[0140] FIG. 6 is a graph 600 of example temperature values of a gas delivery line 602 and example temperature values of a heat jacket 604 when power is reduced, e.g., when power 601 is cut off from steady-state operation. In some scenarios, at least a portion of the chemical delivery system such as the gas delivery line may experience a heat-up process, e.g., when power 601 is driven to one or more heating elements and causes the gas delivery line and any chemicals (e.g., gases) to increase in temperature as indicated in the graph 600. Consequently, the heat jacket may also be heated up and increase in temperature as indicated in the graph 600. In some implementations, the temperature of the heat jacket may be determined by use of one or more IR sensors and / or one or more thermocouples positioned with respect to the heat jacket (e.g., locations T1 through T4 shown in FIG.2D, or other locations). This may occur at time t0. Other gas delivery lines in the chemical delivery system may experience similar heating up at or around t0.

[0141] Subsequently, at t1, the temperature of the gas delivery line 602 and the temperature of the heat jacket 604 may reach steady state. In some cases, there may be oscillations during steady state, while in other cases, there may be smaller or infrequent oscillations (such as those shown in FIG.7), which may depend on how an associated controller is configured or how the controller is tuned or calibrated to provide power 601 to the heating element(s). Regardless, the steady-state temperatures of the gas delivery line may stay within a small temperature range without significant deviation from a setpoint or target temperature 610.

[0142] At some subsequent time t2, power may be cut off (e.g., as a result of power failure), which results in decreases in temperatures of the gas delivery line and the heat jacket 602, 604. Temperatures of the gas delivery line and the heat jacket 602, 604 may continue to decrease at time t3. Incidentally, temperatures of a clamshell 603 surrounding the gas delivery line may also follow similar patterns in temperature as the temperatures of the gas delivery line 602. By or at t4, temperatures may return to approximately the temperatures at t0 and reach steady state.

[0143] Between t2 and t4, a correlation or relationship may exist between the decrease in the temperature of the heat jacket 604 and the decrease in the temperature of the gas delivery line 602. Thus, the temperature of the gas delivery line 602 may be capable of being determined or estimated based on the temperature of the heat jacket 604 or a change thereof (and vice versa) using one or more approaches, e.g., linear correlation, which will be explored further with respect to FIGS.8A and 8B. In some implementations, a trained machine learning model may be used to predict the temperature of the gas delivery line 602 at a future time, where the machine learning model may have been trained using training data that includes actual temperature of the gas delivery lineLAM1P015WO-11765-1WO (ground truth obtained using one or more thermocouples, thermistors, or the like) and infrared temperature of a measured surface (as well as emissivity in some approaches), as discussed elsewhere herein. In some embodiments, the temperature of the heat jacket 604 can be determined or estimated using IR data obtained using one or more IR sensors, e.g., IR sensor 301. Put another way, by measuring the temperature of the heat jacket 604, the actual temperature of the gas delivery line 602 can be monitored. Predictions of future temperatures of the gas delivery line 602 at a time horizon can also be made. In some examples, time-series modeling or machine learning techniques can be used. In machine learning implementations, techniques may be used such as supervised or unsupervised training, which may involve selection of a training dataset (train set) that includes various training data including infrared temperatures and actual temperatures (per pixel or otherwise determined, e.g., statistically such as by average, or by window or region of pixels) as well as ground truth infrared and actual temperatures (ideal expected results), minimization of a loss function, selection of a learning rate and a regularization rate, selection of a portion (e.g., 20%) of the training data as a validation set, and / or adjustments of weights, biases, and other parameters. Such a training dataset may be stored on a non-transitory computer-readable storage medium. The training dataset may then be used by a computerized apparatus or system (e.g., a controller) to train a deep learning or machine learning model. Pre-training and fine tuning may also be performed in some cases. Monitoring the temperature of the gas delivery line 602 may therefore allow preemptive warnings or shutdown and prevent costly damages and replacements of equipment, e.g., before a deviation between the target temperature 610 and the temperature of the gas delivery line 602 reaches a threshold amount (e.g., 10 C), where such a drop in temperature of the gas delivery line 602 might have been caused by power failure.

[0144] FIG. 7 is a graph 700 of example temperature values of a gas delivery line 702 and example temperature values of a heat jacket 704 when power is reduced, e.g., when power is cut off from steady-state operation. At t0, power may be applied to raise the temperature of the gas delivery line (in this scenario, from previously declined temperatures rather than from an offline, completely cooled state) and thereby cause the temperature of the heat jacket to increase as well. Temperatures may reach steady state at or after t1. Here, there may be some infrequent oscillations in the temperature of the gas delivery line 702, which may stay within a small temperature range without significant deviation from a setpoint or target temperature 710.

[0145] At t2, power may be removed, which results in decreases in temperatures of the gas delivery line and the heat jacket 702, 704. After t2, a correlation between the temperatures 702, 704 may enable determination or estimation of the temperature of the gas delivery line 702 based on the temperature of the heat jacket 704 or a change thereof. In some embodiments, theLAM1P015WO-11765-1WO temperature of the heat jacket 704 can be determined or estimated using IR data obtained using one or more IR sensors, e.g., IR sensor 301. Temperatures of the gas delivery line and the heat jacket 702, 704 may continue to decrease at time t3. However, preemptive warnings or shutdown may occur if the deviation between the target temperature 710 and the temperature of the gas delivery line 702 reaches a threshold amount (e.g., 10 C), where such a drop in temperature of the gas delivery line 702 might have been caused by power failure.

[0146] FIG. 8A is a graph 800 of temperature values of a gas delivery line 802 over time in an example scenario when power is reduced, e.g., when power is cut off from steady-state operation. In this example scenario, at time t1, the temperature of the gas delivery line 802 may be at steady state or substantially at steady state relative to and approximately at or within a range of the setpoint or target temperature 810.

[0147] At time t2, power may be cut off or reduced, e.g., as a result of power failure or reduction. Temperature of the gas delivery line 802 may begin to decline as a result. Note the relative consistent decline in temperature of the gas delivery line 802 during time period Δt, which may be the length of time between time t2 and time t3. A total change in temperature ΔT may occur during time period Δt. In some cases, the decline in temperature of the gas delivery line 802 may be linear or substantially linear (with an R2value close to 1 (e.g., above 0.99, above 0.95) indicating a high linear fit or correlation. Such linear or substantially linear relationship can be assumed especially within small temperature ranges such as 10 C or 20 C. In some examples, a linear curve (e.g., a line) 805 may be fitted with the temperature values of the gas delivery line 802. In some implementations, this line and its slope may be used to determine a rate of change in temperature. In some implementations, the rate of change in temperature may be determined by ΔT / Δt. For example, ΔT may be about 10 C, and Δt may be about 500 seconds. Hence, the rate of change in temperature of the gas delivery line 802 may be about 0.02 C / s or about 1.2 C / minute.

[0148] Incidentally, the temperature of a clamshell 803 surrounding the gas delivery line may decline in a similar fashion. A rate of change of the temperature of the clamshell 803 may also be determined in a similar approach as above.

[0149] Referring now to FIG. 8B, a graph 820 is shown of temperature values of a heat jacket 804 over time correlating to the temperature values of the gas delivery line in the example scenario of FIG. 8A. Various points in time are marked t1, t2 and t3, which correspond to t1, t2 and t3 in graph 800 of FIG. 8A. It can be seen that, at time t2, power may be cut off or reduced, e.g., as a result of the aforementioned power failure or reduction occurring at time t2 in graph 800.LAM1P015WO-11765-1WO

[0150] Here, a first time period 822 is a portion of time period Δt, where Δt corresponds to the length of time between time t2 and time t3 as above. In some scenarios, during first time period 822, there may be no significant response to the temperature of the heat jacket 804 that would indicate a power failure or power issue. It can be seen that there is little to no variation in temperature; data points representing the temperature of the heat jacket 804 are moving substantially sideways.

[0151] However, the temperature of the heat jacket 804 may begin to decline after the first time period 822. In this case, the temperature of the heat jacket 804 may continue to decline during a second time period 824, which is also a portion of time period Δt. A change in temperature ΔT2 may occur during the second time period 824 (Δt2). In some cases, the decline in temperature of the heat jacket 804 may be linear or substantially linear (with an R2value close to 1 (e.g., above 0.99, above 0.95) indicating a high linear fit or correlation). In some examples, a linear curve (e.g., a line) 807 may be fitted with the temperature values of the heat jacket 804. In some implementations, this line and its slope may be used to determine a rate of change in temperature. In some implementations, the rate of change in temperature may be determined by ΔT2 / Δt2. For example, ΔT may be about 2 C, and Δt may be about 200 seconds. Hence, the rate of change in temperature of the heat jacket 804 may be about 0.01 C / s or about 0.6 C / minute.

[0152] In some implementations, a correlation between the rate of change in temperature of the gas delivery line 802 and the rate of change in temperature of the heat jacket 804 may be determined. In the above examples, the rate of change in temperature of the gas delivery line 802 may be about 0.02 C / s, while the rate of change in temperature of the heat jacket 804 may be about 0.01 C / s. That is to say, there may be a 2:1 relationship between the two rates of change in temperature, in which for every 0.01 C / s change in the heat jacket (which in some implementations may be measured using, e.g., an IR sensor such as IR sensor 301), it can be estimated that there is a 0.02 C / s change occurring in the gas delivery line.

[0153] In some implementations, the actual temperature of the gas delivery line may be estimated based on the infrared temperature of the heat jacket. In some cases, a correlation between temperature values may be known. As but one example, an infrared temperature indicating that the heat jacket is at temperature T_heat1 as shown in FIG. 8B may have a correlation to a gas delivery line temperature of T_gas1 as shown in FIG. 8A. Then, it can be determined what the deviation from the setpoint or target temperature 810 is. This deviation may be compared to a threshold. If the deviation meets or exceeds a threshold, a remedial action may be triggered.

[0154] In some examples, a threshold such as 10 C (or other(s) as mentioned elsewhere herein) may be preselected or determined, where the threshold indicates that one or more remedial actionsLAM1P015WO-11765-1WO should be taken and / or which actions should be taken, such as warnings or a shutdown. In some cases, the initial lack of response may be accounted for, such as the 200 seconds during the first time period 822. For instance, once the temperature of the heat jacket 804 begins to drop, it can be determined that 200 seconds have passed, though this additional time may be different based on scenario, application, apparatus, component, process, etc.

[0155] In some implementations, during the second time period 824, one or more salient actions may occur or be caused. More specifically, based on the temperature of the heat jacket 804 and / or the rate of change in temperature of the heat jacket 804, one or more preventive measures, such as remedial actions, may be performed to, e.g., mitigate or prevent degradation or failure of a component or apparatus. For example, a first warning 828a may be issued if the temperature of the heat jacket 804 (whether based on an infrared temperature and / or a trained machine learning model, which may corroborate each other) has reached a certain threshold level, or if the rate of change in the temperature of the heat jacket 804 has continued for a certain length of time (time threshold). In some cases, another, second warning 828b may be issued if the temperature of the heat jacket 804 has reached a certain threshold level (different from that associated with the first warning 828a), or if the rate of change in the temperature of the heat jacket 804 has continued for a certain length of time (different from that associated with the first warning 828a). Depending on the implementation, these warnings may be in the form of a visual indication, such as words on a display, and / or an audio indication, such as a sound effect. Other forms of alerts may be used, e.g., haptic, light flashes. In some cases, a shutdown 829 may occur if the threshold(s) associated with the first and / or second warnings 828a, 828b have been met or exceeded, and the temperature of the heat jacket 804 continues to decline by a certain threshold amount or length of time. The shutdown 829 may involve deactivation of a component, activation of an emergency power supply to restore or maintain temperature, stopping of a process, shutting off of a precursor flow, flowing of purge gas, and / or other appropriate measures to prevent further tool damage or failure. An example of a purge gas may include argon (Ar). Other types of purge gases can also be used. At least some of these actions may occur at a third time period 826 (or before the shutdown 829 occurs, sometime during the second time period 824). In some cases, a hard shutdown may occur in which the entire processing apparatus may turn off, e.g., in an emergency situation to prevent catastrophic failure.

[0156] As can be seen, measuring the temperature of the heat jacket 804 can be a basis for performing actions with respect to the processing apparatus or fabrication tool. Further, such temperature information can be tracked across the system as a whole without being limited to locations of thermocouples that make physical contact with components of the processingLAM1P015WO-11765-1WO apparatus, and without having direct line of sight to the gas delivery line. In fact, such temperature information can be visualized and identified on a granular level (e.g., pixel by pixel) in ways mentioned above and discussed in further detail below. Pixel-based Thermal Imaging

[0157] In some embodiments, one or more IR sensors may be deployed and positioned with respect to one or more portions or components of a processing apparatus or fabrication tool. Such IR sensors may be of the type described above, such as IR sensor 301. In larger processing apparatus or systems, multiple IR sensors can provide information (e.g., infrared temperatures) about the entire system at once (including temperatures at different locations over time), allowing monitoring of the entire system for failures and issuance of warnings or other remedial actions according to infrared temperatures and changes thereto, as discussed above.

[0158] Infrared temperatures can be obtained and calibrated for accuracy according to different approaches in which emissivity values are locally determined or globally applied in one or more regions of images representative of IR data. Local Emissivity Example

[0159] In some implementations, a thermal image representative of temperatures may be generated based on IR data obtained using the IR sensor(s). A local emissivity value may be obtained (or known) for a given pixel or region of interest, which may be determined based on the angle and distance of a given IR sensor and the material of the object being imaged (e.g., aluminum). To illustrate, FIG.9A shows an example thermal image 900 representative of spatially varying infrared temperatures associated with a portion of a processing apparatus, according to some embodiments. Temperatures may be visually indicated, e.g., in conjunction with a color scale 901. In some cases, the image may include temperatures associated with a portion of a gas delivery line which is sheathed by a heat jacket 902. The example thermal image 900 may be representative of the heat jacket 902 discussed with respect to FIGS.2A–2E.

[0160] In the example thermal image 900, various points of interest P1 through P9 are identified and indicated. In some implementations, one IR sensor may capture infrared temperatures associated with all 9 points. In some implementations, multiple IR sensors may capture infrared temperatures associated with respective ones of the 9 points. In some cases, IR data may be obtained for a given point using more than one IR sensor. In some implementations, infrared temperatures may be obtained per pixel (making each pixel its own temperature detector and limited only by resolution and capabilities of IR sensors used), and in some cases, statistically determined (e.g., averaged) within a larger spatial region. For example, a kernel having dimensionsLAM1P015WO-11765-1WO of 3x3, 5x5, 12x12, etc. may be used as the spatial region. Such pixel averaging may reduce spatial noise. Although referred to as a “point” of interest, P1 in the example thermal image 900 may be a spatial region (e.g., a 12x12 square) that has a single infrared temperature value that is a result of the infrared temperatures associated with the pixels within the spatial region. For a 12x12 square, for instance, 144 values may be averaged. In some approaches, some pixels may be weighted more or less depending on relative location of a pixel to provide a weighted average; e.g., infrared temperature values associated with pixels closer to the center of the spatial region may be weighted more, and / or infrared temperature values associated with pixels on the edges of the spatial region may be given a lower weight. Other statistical methods may be used, e.g., median, moving average over time. Moreover, in some embodiments, the infrared temperatures may be obtained over time at an interval, e.g., per second, per multiple seconds, per minute, etc. The size of the spatial region and / or the length of the interval may depend on desired accuracy, resolution, response time for alerts, sensitivity, etc., which in turn may depend on the process being performed by the apparatus or system. In other implementations, at least some of P1 through P9 may represent one pixel rather than a region.

[0161] In some implementations, IR data may be obtained over the entire field of view of the IR sensor(s), and isolated and calibrated only for regions of interest which may include points of interest P1 through P9. Infrared temperatures can be obtained based on the IR data captured over time using the IR sensor(s). Infrared temperatures may be calibrated with the corresponding local emissivity value, which may be locally determined at each point or at a spatial region (which may be statistically determined, e.g., averaged, weighted average, median). Accounting for local emissivity may result in higher accuracy for the obtained temperature values. Hence, infrared temperatures may be a function of angle and / or distance of the IR sensor(s) and / or material of the surface of the material, e.g., the heat jacket 902.

[0162] It is noted that the example thermal image 900 is one way to spatially represent the infrared temperatures as a heatmap. Using the approaches described above, other types of images or representations may be obtained for visualization, e.g., a topographic map identifying infrared temperatures above or below a certain level or infrared temperatures in different types of lines (solid, dotted, etc.), or a three-dimensional image, heatmap, or topographic map. Raw IR data and / or infrared temperatures may be stored in other well-known ways, e.g., in a data structure such as a lookup table or CSV file.

[0163] In some embodiments, further evaluation of the infrared temperature values may be performed to obtain broader insights into the gas delivery line (or other component) being monitored. For example, infrared temperatures determined at points of interest P1 through P9 mayLAM1P015WO-11765-1WO be averaged to obtain a single infrared temperature value that is monitored over time. FIG.9B is a graph showing example infrared temperature values obtained for points of interest (e.g., P1 through P9 in FIG. 9A), and an average infrared temperature value 922 may be determined. In some cases, the average infrared temperature value 922 may be determined at the same or different interval as that at which the infrared temperatures are obtained. In some implementations, this average infrared temperature value 922 may be determined and tracked over time. In some implementations, a rate of change of the average infrared temperature value 922 may be determined and tracked over time. As discussed with respect to FIGS. 8A and 8B, the average infrared temperature value 922 or its rate of change may correlate with actual temperatures of the gas delivery line, and may be compared with a threshold to determine whether to issue any warnings or perform at least a partial shutdown of the processing apparatus. In some implementations, other statistical methods for determining a single infrared temperature value may be used, e.g., weighted average, moving average, median. In some implementations, a range of infrared temperatures may be considered. For example, the infrared temperatures determined for points of interest P1 through P9 may range from 74.7 to 78.2 C (accounting for respective local emissivity values), meaning that there is a range of 3.5 C for the points of interest. This range may provide used to derive further insight into the deviation of the actual temperature of the gas delivery line from a setpoint or desired or target temperature.

[0164] As alluded to above, emissivity may be locally determined or globally applied. Global emissivity refers to the same emissivity value applied to all pixels or regions, e.g., within an image. However, different global or local emissivity values may be used when available or when regions are divided. For instance, a top surface of a heat jacket may be at a different distance or angle from a side surface of the heat jacket such that the two surface are given different local emissivity values, e.g., as discussed with respect to FIGS. 12A and 12B below. Applying and assuming the same emissivity for the entire region may simplify the determination of infrared temperatures of points of interest. In some approaches, the emissivity of the material of the object being monitored (e.g., surface of heat jacket) may be used, although calibration may still improve accuracy regardless. That said, the change in temperature alone, even if true emissivity values are not known, can still provide insights into the health of a component based on future predicted temperatures. Global Emissivity Example

[0165] FIGS. 10A and 10B show example thermal images 1000, 1020 of a portion of a processing apparatus, obtained based on different global emissivity values. In thermal image 1000, a first global emissivity value may have been applied, while in thermal image 1020, a second global emissivity value different from the first global emissivity value may have been applied.LAM1P015WO-11765-1WO Emissivity values may range from 0 to 1, inclusive. Although the example thermal images 1000, 1020 capture the same component, infrared temperatures at corresponding points of interest associated with a heat jacket 1002 between the two thermal images may be determined to be different. Hence, calibration may involve determining the optimal emissivity value at which to calculate the infrared temperatures.

[0166] However, it is noted that incorrect emissivity settings can still result in correct estimations of changes in actual temperature of a chemical delivery line. Even if true emissivity is not known, infrared temperatures can still be detected and used to track changes and predict excessive deviations. By way of an illustrative example, a drop of 10 C in the gas delivery line (actual temperature) may correspond to a drop of 3 C (infrared temperature) of the heat jacket surface from the steady state expected temperature. While the estimation of temperatures based on infrared temperatures may be based on correlations between changes between infrared temperature and actual temperature, emissivity can improve the correlations.

[0167] In determining the optimal emissivity value, in some approaches, different global emissivity values may be assigned to different regions of a same image. FIG. 10C shows an example thermal image 1040 of a portion of a processing apparatus. Here, two regions are defined within the example thermal image 1040, which divide the heat jacket 1002 into an upper region 1042 and a lower region 1044. Each of the two regions 1042, 1044 may be associated with respective different global emissivity values. Although the term “global” is used to describe parts of an image, the term may refer to broadly applied emissivity values that are, e.g., applied to a region, not necessarily to the entire image. This more granular approach may further improve accuracies of infrared temperatures determined at points of interest, where different parts of a component, particularly one having a complex shape, may be at different angles or distances from an IR sensor. For example, the part of the component associated with the upper region 1042 may be closer to the IR sensor than the part associated with the lower region 1044, such that assigning different global emissivity values to each region may allow determination of infrared temperatures with higher accuracy. To these ends, any various two or more regions of any respective shape and size may be defined to achieve the desired accuracy, granularity, etc.

[0168] To illustrate the different infrared temperatures that can be determined at global different emissivity values, FIGS. 11A – 13E show sets of example infrared temperatures obtained with respect to different regions of thermal images having different global emissivity values.LAM1P015WO-11765-1WO Global Emissivity Data Example 1

[0169] More specifically, FIG. 11A shows an example thermal image 1100 with a first global emissivity value applied. In this example, four different regions 1110, 1120, 1130, 1140 are defined. The regions may represent different parts of a component such as a portion of a processing apparatus. Although different regions are defined, the same first global emissivity is applied. As will become clearer below, corresponding regions will be compared at different global emissivity values.

[0170] FIG. 11B is a graph 1112 of example infrared temperatures 1114 and thermocouple temperatures 1116 obtained over time with respect to region 1110 indicated in FIG. 11A at the first global emissivity value. In some implementations, example infrared temperatures 1114 may be obtained with respect to points of interest using approaches described above, e.g., using one or more IR sensor(s) disposed with respect to the component. Each dotted curve representative of example infrared temperatures 1114 may correspond to respective points of interest, which may each be a region with a statistical method applied to pixels of the region (e.g., averaging a 12x12 region of pixels) to reduce spatial noise. In addition, each data point of the dotted curve may be a moving average (e.g., a 10-second moving average or based on another time period) to reduce temporal noise. Moreover, the temperatures may be normalized to the highest temperature. That is, the highest temperature shown in the graph 1112 may have a value of 1 and correlate to the highest temperature. The graph 1112 further includes example thermocouple temperatures 1116 as a comparative reference, which may be normalized over time (but may not be averaged temporally or spatially). It can be seen in graph 1112 that the example infrared temperatures 1114 generally correlate with the thermocouple temperatures 1116 but does not match exactly. This disparity will be evaluated later with respect to other examples at other global emissivity values. Global Emissivity Data Example 2

[0171] As mentioned above with respect to FIG. 10C, accuracies of infrared temperatures can be further improved by defining regions with different global emissivity values applied to each region.

[0172] FIG.12A shows an example thermal image 1200 with a first global emissivity value and a second global emissivity value applied at different regions 1210, 1220, 1230, 1240 of the example thermal image 1200. These regions may correspond to regions 1110, 1120, 1130, 1140, respectively, and represent corresponding parts of the component represented in example thermal image 1100. Regions 1210 and 1230 may have a first global emissivity value (labeled “Emissivity A”) applied thereto, and regions 1230 and 1240 may have a second global emissivity value (labeledLAM1P015WO-11765-1WO “Emissivity B”) applied thereto. The second global emissivity value may be different from the first global emissivity value. Angle of incidence relative to an IR sensor may affect the emissivity values in different regions. For example, regions 1210 and 1230 may correspond to a top surface of a heat jacket, while regions 1220 and 1240 may correspond to a side surface of the heat jacket. The material may be the same in both regions, but the angle at which IR data is obtained may affect result in different emissivity values being applied to the regions. Note that these first and second global emissivity values may not necessarily be those used in FIGS.11A and 11B.

[0173] FIG. 12B is a graph 1212 of example infrared temperatures 1214 and 1215 and thermocouple temperatures 1216 obtained over time with respect to different regions indicated in FIG. 12A at the first and second global emissivity values. These temperature values may be obtained in the same manner as example infrared temperatures 1114 and thermocouple temperatures 1116 of FIG. 11B. It can be observed that example infrared temperatures 1214 and 1215 generally correlate with the thermocouple temperatures 1216, and the disparity between the two sets of temperatures may be minimal, similar to graph 1112 of FIG.11B.

[0174] The degree of similarity or difference between example infrared temperatures and thermocouple temperatures may be indicative of a higher or lower correlation to one another. This may be helpful for calibrating emissivity values as being an appropriate choice for a particular region(s), if data shows a smaller disparity with a given emissivity.

[0175] By dividing a thermal image into multiple regions, a more customized or selective global emissivity value may be applied to appropriate regions of the thermal image, which may result in improved correlations and accuracies of infrared temperatures measured with respect to the component being monitored and measured. That is to say, a given emissivity value may provide a better correlation between infrared temperatures and temperatures obtained via thermocouple, and configurations utilizing that emissivity value may yield better accuracy of infrared temperatures, which in turn may result better correlation with temperatures of a gas delivery line and also result in quicker, more useful and accurate warnings with fewer false positives and fewer missed warnings.

[0176] In different implementations, how the regions and emissivity values are defined may be known ahead of time and determinable via experimentation or observation, and calibration.

[0177] As such, emissivity values, whether locally or globally applied, can be a basis for obtaining infrared temperatures.LAM1P015WO-11765-1WO System-wide Imaging and Visualization

[0178] FIG. 13 is an example image 1300 of a perspective view of at least a portion of a processing apparatus, where a relevant component is visualized with indications representative of infrared temperatures. In some examples, the relevant component may be a chemical delivery line, such as an example gas delivery line 1302. In some embodiments, using the techniques described herein, a heatmap can be obtained, which provides pixel-by-pixel indication of infrared temperatures obtained using one or more IR sensors, such as IR sensor 301. In example image 1300, the heatmap may be a three-dimensional heatmap.

[0179] In some implementations, the heatmap may be interacted with, e.g., rotated, zoomed in, etc., which can provide information (e.g., infrared temperatures) about the gas delivery line 1302 from different perspectives and at different granularities to a user visually. In some approaches, such three-dimensional imaging may be performed based on multiple IR sensors disposed at different angles and locations to obtain IR data from different perspectives.

[0180] In some implementations, as noted above, images may be stitched or otherwise combined from two or more individual camera sensors (e.g., IR sensors) located and oriented to capture different regions and / or angles. Various image processing techniques can be used to perform such stitching or combination of images. In some approaches, input images may be projected onto a cylinder, unrolled, and then combined (e.g., stitched). More specifically, keypoints may be identified and matched between the images, e.g., using image processing algorithms such as scale- invariant feature transform (SIFT) feature detectors, and / or feature matching algorithms such as Fast Library for Approximate Nearest Neighbors (FLANN)-based methods to choose the best algorithm and optimum parameters (or using similar methods optimized for fast nearest neighbor search in large datasets) and find matches. Further, in some variants, homography matrix (a mapping between two images that observe the same plane) can be estimated between the images, which may also involve further iterative methods, e.g., a random sample consensus (RANSAC) algorithm that minimizes the error between the matched keypoints. The images may then be aligned according to their overlapping regions. In some approaches, barrel distortion from the input images may be corrected before stitching. Barrel distortion may occur if wide-angle lenses are used. In some approaches, panorama straightening functions, methods, or software can additionally be used to modify distortions or improve the image combination, e.g., PTGui, Puppet Warp.

[0181] Performing such imaging, measurements, and / or data collection and in real-time (e.g., infrared temperatures are obtained based on IR data and / or temperatures of a relevant component are estimated as IR data is collected using IR sensor(s)) can advantageously enable powerfulLAM1P015WO-11765-1WO monitoring capabilities that allow changes in temperature to be studied in detail and / or predicted ahead of failure. In particular, large-scale imaging and visualization of entire systems (e.g., processing apparatus and fabrication tools) or portions thereof can advantageously be enabled as well. Based on such real-time monitoring and determination of temperatures (infrared temperatures or otherwise), remedial measures can be proactively taken with respect to gas lines or other components, e.g., issue warnings, perform a shutdown, purge precursors, etc.

[0182] Consequently, information (e.g., infrared temperatures) about at least portions of the gas delivery line 1302 can be monitored algorithmically (e.g., determining and storing infrared temperatures obtained via IR sensor(s)) and visually, at various points of interest 1304a – 1304s.

[0183] FIG. 14 is another example image 1400 of a perspective view of at least a portion of a processing apparatus, where a relevant component is visualized with indications representative of infrared temperatures. In some examples, the relevant component may be a chemical delivery line, such as an example foreline 1402 from which chemistries may exit from a processing apparatus or process chamber. Here, the example image 1400 may be another three-dimensional heatmap, and it may indicate pixel-by-pixel indication of infrared temperatures obtained using one or more IR sensors at a higher resolution and granularity, which can represent information (e.g., infrared temperatures) about the example foreline 1402 similar to that discussed with respect to example image 1300. Consequently, information (e.g., infrared temperatures) about at least portions of the example foreline 1402 can be monitored algorithmically (e.g., determining and storing infrared temperatures obtained via IR sensor(s)) and visually, at various points of interest 1404a – 1404l. Methods

[0184] FIG. 15 is a flow diagram of an example of a method 1500 of monitoring a processing apparatus, according to some embodiments. One or more of the functions of the method 1500 may be performed by or caused by a computerized apparatus or system, such as a deposition apparatus or deposition tool configured to perform deposition of material (e.g., film) on a substrate. Structure for performing the functionality illustrated in one or more of the blocks shown in FIG.15 may be performed by hardware and / or software components of such computerized apparatus or system, such as, for example, a controller apparatus, a computerized system, or a computer-readable apparatus including a storage medium storing computer-readable and / or computer-executable instructions that are configured to, when executed by a processor apparatus, cause the at least one processor apparatus or a computerized apparatus to perform the operations. A controller may be one example of the computerized apparatus or system. A process chamber may be another exampleLAM1P015WO-11765-1WO or component of the computerized apparatus or system. Example components of above apparatus are illustrated in, e.g., FIGS.1A – 1C and 17, and described in more detail elsewhere herein.

[0185] It should also be noted that the operations of FIG. 15 may be performed in any suitable order, not necessarily the order depicted in FIG. 15. Further, the process shown in FIG. 15 may include additional or fewer operations than those depicted in FIG.15.

[0186] At block 1510, the method 1500 may include determining, in real-time, a first temperature associated with a first component of the processing apparatus based on one or more measurements obtained from at least one sensor disposed relative to the first component of the processing apparatus. For example, the at least one sensor may be obtaining signals from at least the first component at a defined or prescribed angle (θ) and a defined or prescribed distance (d).

[0187] As used herein, the term “real-time” or “real time” may refer to an action that occurs simultaneously with another action, or substantially simultaneously with another action (e.g., immediately as a result of the other action). The term may alternatively or additionally describe a continuous action. In some scenarios described herein, an IR sensor during operation may continuously detect and measure infrared signals, which may be processed or sent to, e.g., a controller that makes determinations or estimation of actual temperature based on the infrared signals. Hence, a “real-time” determination of a temperature or other information, while it may be a prediction of what it will be at a future time, may not itself occur at a future time, whether at a predetermined time or not (e.g., based on stored data or in a scheduled or an ad hoc manner). Rather, in some configurations, the determination (e.g., by a controller) may occur as the monitoring occurs (e.g., by a sensor).

[0188] In some embodiments, the first component may be configured to insulate a clamshell that houses one or more heating elements configured to adjust a temperature of a gas delivery line of the processing apparatus, and the gas delivery line may be configured to carry a precursor, a reactant, a purge gas, or a combination thereof. In some implementations, the first temperature associated with the first component may include an infrared temperature associated with a surface of the first component; the method may further include estimating a temperature associated with the gas delivery line based on a correlation with the infrared temperature associated with the surface of the first component, the temperature associated with the gas delivery line being different from the infrared temperature associated with the surface of the first component; and the performing of the first preventive measure may be further based on the estimated temperature associated with the gas delivery line reaching a threshold value for the gas delivery line.LAM1P015WO-11765-1WO

[0189] In some embodiments, the at least one sensor may include at least one infrared sensor disposed at a prescribed distance and a prescribed angle relative to the first component and configured to detect infrared signals, the one or more measurements including the at least infrared signals. In some configurations, the at least one sensor may be further configured to detect other types of signals as well, e.g., optical signals. In some cases, the first temperature associated with the first component is determined further based on (i) an estimated temperature determined based on the infrared signals, (ii) emissivity associated with a surface of the first component, the emissivity based on the prescribed distance and the prescribed angle of the at least one infrared sensor, a material of the surface of the first component, or a combination thereof, or (iii) a combination thereof.

[0190] In some implementations, portions of the at least infrared signals correspond to a respective plurality of pixels of an image representing at least a portion of the first component. For example, the image may be a heatmap (two-dimensional or three-dimensional) of the types described above, a topographic map, or others described above, each of which may be interacted with (e.g., rotated, zoomed in, etc.). In some implementations, the method may further include obtaining, based on the one or more measurements obtained from the at least one sensor, a plurality of temperatures associated with the respective plurality of pixels representing at least the portion of the first component, the plurality of temperatures including the first temperature associated with the first component; and the first temperature may be associated with at least a first pixel of the plurality of pixels.

[0191] In some implementations, the respective plurality of pixels may represent the first component and one or more additional components of the processing apparatus. In some implementations, the method may further include determining, in real-time, one or more additional temperatures associated with the one or more additional components of the processing apparatus based on one or more measurements obtained from one or more sensors disposed relative to the one or more additional components of the processing apparatus; and the performing of the first preventive measure may be based at least on the first temperature or the one or more additional temperatures reaching the first threshold value.

[0192] In some implementations, the first temperature may be determined further based on at least a portion of the plurality of temperatures, the at least portion of the plurality of temperatures including one or more temperatures associated with one or more pixels adjacent to the first pixel. In some examples, as discussed above, kernels or sliding windows that are of a defined size (e.g., 3x3, 5x5, 12x12) may be used to assess the temperatures, or more specifically, a statisticallyLAM1P015WO-11765-1WO determined value (e.g., average, weighted average, median) of the first pixel and the one or more pixels adjacent thereto, which may reduce spatial noise.

[0193] At block 1520, the method 1500 may include, based at least on the first temperature associated with the first component reaching a first threshold value, performing at least a first preventive measure that mitigates degradation or failure associated with the processing apparatus.

[0194] In some embodiments, the first preventive measure may include: issuance of a warning relating to degradation or failure of at least the first component of the processing apparatus; deactivation of at least the first component of the processing apparatus; shutting off a flow of a chemical through a chemical delivery line; flow of a purge gas; or a combination thereof.

[0195] In some embodiments, the method 1500 may further include determining a rate of change of temperature associated with the first component of the processing apparatus over a period of time; and predicting, based at least on the rate of change of temperature, a future temperature associated with the first component. In some implementations, the performing of the first preventive measure may be further based on the future temperature reaching the first threshold value.

[0196] In some embodiments, the method 1500 may further include, based on the first temperature associated with the first component reaching a second threshold value beyond the first threshold value, performing a second preventive measure that mitigates the degradation or failure of the processing apparatus. In some implementations, the first preventive measure may include issuance of a warning relating to the degradation or failure associated with the processing apparatus, and the second preventive measure may include deactivation of at least a portion of the processing apparatus. In some examples, a first warning may be issued at the first threshold value, and a partial shutdown or a soft shutdown may be performed at the second threshold value. In some examples, there may be more than two threshold values, and two warnings may be issued at first and second thresholds and a shutdown may occur at a third threshold, as shown in FIG.8B.

[0197] In some embodiments, the method 1500 may further include estimating a health of a second component based on the first temperature associated with the first component. In some scenarios, the second component may include a remote plasma clean (RPC) source.

[0198] In some implementations, the one or more measurements are received through an infrared-transparent window disposed on the first component. For example, the infrared- transparent window may be disposed on a heat jacket to allow direct measurement of a clamshell, which may provide a more direct prediction of a temperature of a gas delivery line.LAM1P015WO-11765-1WO

[0199] FIG. 16 is a flow diagram of another example of a method 1600 of monitoring a processing apparatus, according to some embodiments. One or more of the functions of the method 1600 may be performed by or caused by a computerized apparatus or system, such as a deposition apparatus or deposition tool configured to perform deposition of material (e.g., film) on a substrate. Structure for performing the functionality illustrated in one or more of the blocks shown in FIG. 16 may be performed by hardware and / or software components of such computerized apparatus or system, such as, for example, a controller apparatus, a computerized system, or a computer- readable apparatus including a storage medium storing computer-readable and / or computer- executable instructions that are configured to, when executed by a processor apparatus, cause the at least one processor apparatus or a computerized apparatus to perform the operations. A controller may be one example of the computerized apparatus or system. A process chamber may be another example or component of the computerized apparatus or system. Example components of above apparatus are illustrated in, e.g., FIGS. 1A – 1C and 17, and described in more detail elsewhere herein.

[0200] It should also be noted that the operations of FIG. 16 may be performed in any suitable order, not necessarily the order depicted in FIG. 16. Further, the process shown in FIG. 16 may include additional or fewer operations than those depicted in FIG.16.

[0201] At block 1610, the method 1600 may include obtaining infrared data associated with a surface of a component of the processing apparatus based on one or more measurements by at least one infrared sensor disposed relative to the surface of the component. In some embodiments, the at least one infrared sensor may be an example of IR sensor 301. In some embodiments, the at least one infrared sensor may be disposed at a prescribed angle and / or a prescribed distance relative to the surface of the component. In some configurations, multiple infrared sensors may be disposed relative to the surface of the component.

[0202] At block 1620, the method 1600 may include determining, in real-time, an infrared temperature associated with the surface of the component based on the infrared data.

[0203] In some embodiments, the infrared temperature may be indicated by one or more pixels of an image representing at least a portion of the component of the processing apparatus and associated with an emissivity value. In some implementations, the one or more pixels may include a plurality of pixels representative of a region of the at least portion of the component, and the infrared temperature may include a value that is averaged spatially with respect to the plurality of pixels, averaged temporally over a period of time, or a combination thereof.LAM1P015WO-11765-1WO

[0204] In some embodiments, a plurality of infrared sensors may be disposed relative to a plurality of components of the processing apparatus. The plurality of infrared sensors may be configured to obtain infrared data associated with the plurality of components of the processing apparatus. In some implementations, the image representing at least a portion of the component of the processing apparatus may include a heatmap representative of at least the at least portion of the component of the processing apparatus. In some cases, the entire processing apparatus may be imaged, e.g., in a two-dimensional or a three-dimensional image such as a heatmap (each of which may be interacted with).

[0205] At block 1630, the method 1600 may include performing a first preventive measure that mitigates degradation or failure of the processing apparatus based on (i) the infrared temperature associated with the surface of the component reaching a first threshold value, (ii) a rate of change of the infrared temperature, or (iii) a combination thereof.

[0206] In some embodiments, the infrared temperature associated with the surface of the component may correlate with a temperature associated with an interior of the component; and the first threshold value may be selected based on an acceptable deviation of the temperature associated with the interior of the component from a setpoint. An example of an acceptable deviation may be a temperature deviation of 10 C. Thus, the first threshold value may be 10 C in this example. As noted elsewhere herein, other thresholds may be selected as desired.

[0207] In some embodiments, the method 1600 may further include estimating the temperature associated with the interior of the component using a trained machine learning model. In some implementations, the trained machine learning model may be configured to output the temperature associated with the interior of the component based on the infrared temperature associated with the surface of the component, an emissivity value, or a combination thereof.

[0208] In some embodiments, the first threshold value may include a deviation from the setpoint which is less than the acceptable deviation.

[0209] In some embodiments, the interior of the component may include a gas delivery line of the processing apparatus, and the surface of the component of the processing apparatus may include a surface of a heat jacket that surrounds the gas delivery line.

[0210] In some embodiments, the method 1600 may further include performing a second preventive measure based on (i) the infrared temperature associated with the surface of the component reaching a second threshold value, (ii) the rate of change of the infrared temperature, or (iii) a combination of thereof.LAM1P015WO-11765-1WO

[0211] In some embodiments, the first preventive measure may include: issuance of a warning relating to the degradation or failure of at least the component of the processing apparatus; deactivation of at least the component of the processing apparatus; shutting off a flow of a chemical through a chemical delivery line; flow of a purge gas; or a combination thereof. Apparatus

[0212] FIG.17 is a schematic of a process system suitable for conducting deposition processes, such as frontside deposition processes, in accordance with embodiments. The system 1700 includes a transfer module 1703. The transfer module 1703 provides a clean, pressurized environment to minimize risk of contamination of substrates being processed as they are moved between various reactor modules. Mounted on the transfer module 1703 is a multi-station reactor 1709 capable of performing ALD, treatment, and CVD according to various embodiments. Multi- station reactor 1709 may include multiple stations 1711, 1713, 1715, and 1717 that may sequentially perform operations in accordance with disclosed embodiments. Stations may include a heated pedestal or substrate support, one or more gas inlets or showerhead or dispersion plate.

[0213] Mounted on the transfer module 1703 may be one or more single or multi-station modules 1707 capable of performing plasma or chemical (non-plasma) pre-cleans, other deposition operations, or etch operations. The module may also be used for various treatments to, for example, prepare a substrate for a deposition process. The system 1700 also includes one or more wafer source modules 1701, where wafers are stored before and after processing. An atmospheric robot (not shown) in the atmospheric transfer chamber 1719 may first remove wafers from the wafer source modules 1701 to loadlocks 1721. A wafer transfer device (generally a robot arm unit) in the transfer module 1703 moves the wafers from loadlocks 1721 to and among the modules mounted on the transfer module 1703.

[0214] In various embodiments, a system controller 1742 is employed to control process conditions during deposition. The system controller 1742 will typically include one or more memory devices and one or more processors. A processor may include a central processing unit (CPU), graphics processing unit (GPU), logic, or other computing unit (computer) or processing unit, analog and / or digital input / output connections, stepper motor controller boards, etc.

[0215] The system controller 1742 may control all the activities of the deposition apparatus. The system controller 1742 executes system control software, including sets of instructions for controlling the timing, mixture of gases, chamber pressure, chamber temperature, wafer temperature, radio frequency (RF) power levels, wafer chuck or pedestal position, and otherLAM1P015WO-11765-1WO parameters of a particular process. Other computer programs stored on memory devices associated with the system controller 1742 may be employed in some embodiments.

[0216] The depicted embodiment includes a user interface associated with the system controller 1742. The user interface may include a display screen, graphical software displays of the apparatus and / or process conditions, and user input devices such as pointing devices, keyboards, touch screens, microphones, etc.

[0217] System control logic may be configured in any suitable way. In general, the logic can be designed or configured in hardware and / or software. The instructions for controlling the drive circuitry may be hard coded or provided as software. The instructions may be provided by “programming.” Such programming is understood to include logic of any form, including hard coded logic in digital signal processors, application-specific integrated circuits, and other devices which have specific algorithms implemented as hardware. Programming is also understood to include software or firmware instructions that may be executed on a general-purpose processor. System control software may be coded in any suitable computer readable programming language.

[0218] The computer program code for controlling the processes in a process sequence can be written in any conventional computer readable programming language: for example, assembly language, C, C++, Pascal, Fortran, or others. Compiled object code or script is executed by the processor to perform the tasks identified in the program. Also as indicated, the program code may be hard coded.

[0219] The controller parameters relate to process conditions, such as, for example, process gas composition and flow rates, temperature, pressure, cooling gas pressure, substrate temperature, and chamber wall temperature. These parameters are provided to the user in the form of a recipe and may be entered utilizing the user interface.

[0220] Signals for monitoring the process may be provided by analog and / or digital input connections of the system controller 1742. The signals for controlling the process are output on the analog and digital output connections of the system 1700.

[0221] The system software may be designed or configured in different ways. For example, various chamber component subroutines or control objects may be written to control operation of the chamber components necessary to carry out the deposition processes in accordance with the disclosed embodiments. Examples of programs or sections of programs for this purpose include substrate positioning code, process gas control code, pressure control code, and heater control code.LAM1P015WO-11765-1WO

[0222] In some implementations, a system controller 1742 is part of a system, which may be part of the above-described examples. Such systems can include semiconductor processing equipment, including a processing tool or tools, chamber or chambers, a platform or platforms for processing, and / or specific processing components (a wafer pedestal, a gas flow system, etc.). These systems may be integrated with electronics for controlling their operation before, during, and after processing of a semiconductor wafer or substrate. The electronics may be referred to as the “controller,” which may control various components or subparts of the system or systems. The system controller 1742, depending on the processing requirements and / or the type of system, may be programmed to control any of the processes disclosed herein, including the delivery of processing gases, temperature settings (e.g., heating and / or cooling), pressure settings, vacuum settings, power settings, radio frequency (RF) generator settings in some systems, RF matching circuit settings, frequency settings, flow rate settings, fluid delivery settings, positional and operation settings, wafer transfers into and out of a tool and other transfer tools and / or load locks connected to or interfaced with a specific system. System Controllers

[0223] Broadly speaking, a controller may be defined as electronics having various integrated circuits, logic, memory, and / or software that receive instructions, issue instructions, control operation, enable cleaning operations, enable endpoint measurements, and the like. Such controller may be used in or with any of the apparatus described herein. The integrated circuits may include chips in the form of firmware that store program instructions, digital signal processors (DSPs), chips defined as application specific integrated circuits (ASICs), and / or one or more microprocessors, or microcontrollers that execute program instructions (e.g., software). Program instructions may be instructions communicated to the controller in the form of various individual settings (or program files), defining operational parameters for carrying out a particular process on or for a semiconductor wafer or to a system. The operational parameters may, in some embodiments, be part of a recipe defined by process engineers to accomplish one or more processing steps during the fabrication of one or more layers, materials, metals, oxides, silicon, silicon dioxide, surfaces, circuits, and / or dies of a wafer.

[0224] A system controller may be a part of or coupled to a computer that is integrated with, coupled to the system, otherwise networked to the system, or a combination thereof. For example, a system controller may be in the “cloud” or all or a part of a fab host computer system, which can allow for remote access of the wafer processing. The computer may enable remote access to the system to monitor current progress of fabrication operations, examine a history of past fabrication operations, examine trends or performance metrics from a plurality of fabrication operations, toLAM1P015WO-11765-1WO change parameters of current processing, to set processing steps to follow a current processing, or to start a new process. In some examples, a remote computer (e.g. a server) can provide process recipes to a system over a network, which may include a local network or the Internet. The remote computer may include a user interface that enables entry or programming of parameters and / or settings, which are then communicated to the system from the remote computer. In some examples, the controller receives instructions in the form of data, which specify parameters for each of the processing steps to be performed during one or more operations. It should be understood that the parameters may be specific to the type of process to be performed and the type of tool that the controller is configured to interface with or control. Thus, as described above, the controller may be distributed, such as by including one or more discrete controllers that are networked together and working towards a common purpose, such as the processes and controls described herein. An example of a distributed controller for such purposes would be one or more integrated circuits on a chamber in communication with one or more integrated circuits located remotely (such as at the platform level or as part of a remote computer) that combine to control a process on the chamber.

[0225] Without limitation, example systems may include a plasma etch chamber or module, a deposition chamber or module, a spin-rinse chamber or module, a metal plating chamber or module, a clean chamber or module, a bevel edge etch chamber or module, a physical vapor deposition (PVD) chamber or module, a CVD chamber or module, an ALD chamber or module, an atomic layer etch (ALE) chamber or module, an ion implantation chamber or module, a track chamber or module, and any other semiconductor processing systems that may be associated or used in the fabrication and / or manufacturing of semiconductor wafers.

[0226] As noted above, depending on the process step or steps to be performed by the tool, the controller might communicate with one or more of other tool circuits or modules, other tool components, cluster tools, other tool interfaces, adjacent tools, neighboring tools, tools located throughout a factory, a main computer, another controller, or tools used in material transport that bring containers of wafers to and from tool locations and / or load ports in a semiconductor manufacturing factory.

[0227] A system controller may include various programs. A substrate positioning program may include program code for controlling chamber components that are used to load the substrate onto a pedestal or chuck and to control the spacing between the substrate and other parts of the chamber such as a gas inlet and / or target. A process gas control program may include code for controlling gas composition, flow rates, pulse times, and optionally for flowing gas into the chamber prior to deposition in order to stabilize the pressure in the chamber. A pressure control program mayLAM1P015WO-11765-1WO include code for controlling the pressure in the chamber by regulating, e.g., a throttle valve in the exhaust system of the chamber. A heater control program may include code for controlling the current to a heating unit that is used to heat the substrate. Alternatively, the heater control program may control delivery of a heat transfer gas such as helium to the wafer chuck.

[0228] Examples of chamber sensors that may be monitored during deposition include mass flow controllers, pressure sensors such as manometers, and thermocouples located in the pedestal or chuck. Appropriately programmed feedback and control algorithms may be used with data from these sensors to maintain desired process conditions.

[0229] The foregoing describes implementation of disclosed embodiments in a single or multi- chamber semiconductor processing tool. The apparatus and process described herein may be used in conjunction with lithographic patterning tools or processes, for example, for the fabrication or manufacture of semiconductor devices, displays, LEDs, photovoltaic panels, and the like. Typically, though not necessarily, such tools / processes will be used or conducted together in a common fabrication facility. Lithographic patterning of a film typically includes some or all of the following steps, each step provided with a number of possible tools: (1) application of photoresist on a workpiece, i.e., substrate, using a spin-on or spray-on tool; (2) curing of photoresist using a hot plate or furnace or UV curing tool; (3) exposing the photoresist to visible or UV or x-ray light with a tool such as a wafer stepper; (4) developing the resist so as to selectively remove resist and thereby pattern it using a tool such as a wet bench; (5) transferring the resist pattern into an underlying film or workpiece by using a dry or plasma-assisted etching tool; and (6) removing the resist using a tool such as an RF or microwave plasma resist stripper.

[0230] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Embodiments disclosed herein may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail to not unnecessarily obscure the disclosed embodiments. Further, while the disclosed embodiments will be described in conjunction with specific embodiments, it will be understood that the specific embodiments are not intended to limit the disclosed embodiments. It should be noted that there are many alternative ways of implementing the processes, systems, and apparatus of the present embodiments. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein.

Claims

LAM1P015WO-11765-1WO CLAIMS What is claimed is:

1. A method for monitoring a processing apparatus, the method comprising: determining, in real-time, a first temperature associated with a first component of the processing apparatus based on one or more measurements obtained from at least one sensor disposed relative to the first component of the processing apparatus; and based at least on the first temperature associated with the first component reaching a first threshold value, performing at least a first preventive measure that mitigates degradation or failure associated with the processing apparatus.

2. The method of claim 1, wherein the first component is configured to insulate a clamshell that houses one or more heating elements configured to adjust a temperature of a gas delivery line of the processing apparatus, and the gas delivery line is configured to carry a precursor, a reactant, a purge gas, or a combination thereof.

3. The method of claim 2, wherein: the first temperature associated with the first component comprises an infrared temperature associated with a surface of the first component; the method further comprises estimating a temperature associated with the gas delivery line based on a correlation with the infrared temperature associated with the surface of the first component, the temperature associated with the gas delivery line being different from the infrared temperature associated with the surface of the first component; and the performing of the first preventive measure is further based on the estimated temperature associated with the gas delivery line reaching a threshold value.

4. The method of claim 1, wherein: the at least one sensor comprises at least one infrared sensor disposed at a prescribed distance and a prescribed angle relative to the first component and configured to detect infrared signals, the one or more measurements comprising the at least infrared signals; and the first temperature associated with the first component is determined further based on (i) an estimated temperature determined based on the infrared signals, (ii) emissivity associated with a surface of the first component, the emissivity based on the prescribed distance and the prescribed angle of the at least one infrared sensor, a material of the surface of the first component, or a combination thereof, or (iii) a combination thereof.LAM1P015WO-11765-1WO 5. The method of claim 4, wherein: portions of the at least infrared signals correspond to a respective plurality of pixels of an image representing at least a portion of the first component; the method further comprises obtaining, based on the one or more measurements obtained from the at least one sensor, a plurality of temperatures associated with the respective plurality of pixels representing at least the portion of the first component, the plurality of temperatures comprising the first temperature associated with the first component; and the first temperature is associated with at least a first pixel of the plurality of pixels.

6. The method of claim 5, wherein: the respective plurality of pixels represent the first component and one or more additional components of the processing apparatus; and the method further comprises determining, in real-time, one or more additional temperatures associated with the one or more additional components of the processing apparatus based on one or more measurements obtained from one or more sensors disposed relative to the one or more additional components of the processing apparatus; wherein the performing of the first preventive measure is based at least on the first temperature or the one or more additional temperatures reaching the first threshold value.

7. The method of claim 5, wherein the first temperature is determined further based on at least a portion of the plurality of temperatures, the at least portion of the plurality of temperatures comprising one or more temperatures associated with one or more pixels adjacent to the first pixel.

8. The method of claim 1, further comprising: determining a rate of change of temperature associated with the first component of the processing apparatus over a period of time; and predicting, based at least on the rate of change of temperature, a future temperature associated with the first component; wherein the performing of the first preventive measure is further based on the future temperature reaching the first threshold value.

9. The method of claim 1, wherein the first preventive measure comprises:LAM1P015WO-11765-1WO issuance of a warning relating to degradation or failure of at least the first component of the processing apparatus; deactivation of at least the first component of the processing apparatus; shutting off a flow of a chemical through a chemical delivery line; flow of a purge gas; or a combination thereof.

10. The method of claim 1, further comprising, based on the first temperature associated with the first component reaching a second threshold value beyond the first threshold value, performing a second preventive measure that mitigates the degradation or failure of the processing apparatus.

11. The method of claim 10, wherein the first preventive measure comprises issuance of a warning relating to the degradation or failure associated with the processing apparatus, and the second preventive measure comprises deactivation of at least a portion of the processing apparatus.

12. The method of claim 1, further comprising estimating a health of a second component based on the first temperature associated with the first component.

13. The method of claim 12, wherein the second component comprises a remote plasma clean (RPC) source.

14. The method of claim 1, wherein the one or more measurements are received through an infrared-transparent window disposed on the first component.

15. A method of monitoring a processing apparatus, the method comprising: obtaining infrared data associated with a surface of a component of the processing apparatus based on one or more measurements by at least one infrared sensor disposed relative to the surface of the component; determining, in real-time, an infrared temperature associated with the surface of the component based on the infrared data; and performing a first preventive measure that mitigates degradation or failure of the processing apparatus based on (i) the infrared temperature associated with the surface of theLAM1P015WO-11765-1WO component reaching a first threshold value, (ii) a rate of change of the infrared temperature, or (iii) a combination thereof.

16. The method of claim 15, wherein the at least one infrared sensor is disposed at a prescribed angle and a prescribed distance relative to the surface of the component.

17. The method of claim 15, wherein: the infrared temperature associated with the surface of the component correlates with a temperature associated with an interior of the component; and the first threshold value is selected based on an acceptable deviation of the temperature associated with the interior of the component from a setpoint.

18. The method of claim 17, further comprising estimating the temperature associated with the interior of the component using a trained machine learning model.

19. The method of claim 18, wherein the trained machine learning model is configured to output the temperature associated with the interior of the component based on the infrared temperature associated with the surface of the component, an emissivity value, or a combination thereof.

20. The method of claim 17, wherein the first threshold value comprises a deviation from the setpoint which is less than the acceptable deviation.

21. The method of claim 17, wherein the interior of the component comprises a gas delivery line of the processing apparatus, and the surface of the component of the processing apparatus comprises a surface of a heat jacket that surrounds the gas delivery line.

22. The method of claim 15, further comprising performing a second preventive measure based on (i) the infrared temperature associated with the surface of the component reaching a second threshold value, (ii) the rate of change of the infrared temperature, or (iii) a combination of thereof.

23. The method of claim 15, wherein the first preventive measure comprises:LAM1P015WO-11765-1WO issuance of a warning relating to the degradation or failure of at least the component of the processing apparatus; deactivation of at least the component of the processing apparatus; shutting off a flow of a chemical through a chemical delivery line; flow of a purge gas; or a combination thereof.

24. The method of claim 15, wherein the infrared temperature is indicated by one or more pixels of an image representing at least a portion of the component of the processing apparatus and associated with an emissivity value.

25. The method of claim 24, wherein the one or more pixels comprise a plurality of pixels representative of a region of the at least portion of the component, and the infrared temperature comprises a value that is averaged spatially with respect to the plurality of pixels, averaged temporally over a period of time, or a combination thereof.

Images