Method and system for monitoring productivity of metrology group

Abstract

Methods and systems for evaluating individual semiconductor metrology tool productivity based on both individual tool productivity metrics and group productivity metrics are described herein. The productivity metrics associated with each individual tool are combined with productivity metrics associated with a group of tools to quickly and less mispositively identify problematic tools. In particular, in situations where productivity is driven by low-frequency events, tool productivity results are obtained significantly faster. Values of one or more accuracy metrics indicative of a confidence of measuring a ranking of individual tools in the population of tools are estimated. In addition, a probability of a fault event is predicted based on a difference between a predicted probability distribution of future fault events associated with individual tools of the group of measurement tools and an actual observed distribution of fault events.

Description

Technical Field

[0001] The described embodiments relate to metrology systems and methods, and more particularly, the described embodiments relate to methods and systems for improving measurement of semiconductor structures. Background Art

[0002] Semiconductor devices such as logic and memory devices are typically manufactured by a series of processing steps applied to a sample. Various features and multiple structural levels of the semiconductor device are formed by these processing steps. For example, photolithography is one type of semiconductor manufacturing process that is particularly involved in creating patterns on a semiconductor wafer. Additional examples of semiconductor manufacturing processes include, but are not limited to, chemical mechanical polishing, etching, deposition, and ion implantation. Multiple semiconductor devices can be manufactured on a single semiconductor wafer and then separated into individual semiconductor devices.

[0003] Metrology processes are used at various steps during the semiconductor manufacturing process to detect defects on wafers to facilitate higher yields. Optical and X-ray based metrology techniques offer the potential for high yields without the risk of sample destruction. Several metrology based techniques including scatterometry, reflectometry and ellipsometry implementations and associated analysis algorithms are commonly used to characterize critical dimensions, film thickness, composition, overlay and other parameters of nanoscale structures.

[0004] The performance, integration and reliability of semiconductor devices continue to improve over time due to improved process resolution and increasingly complex device structures. Improved process resolution can reduce the minimum critical dimension of fabricated structures. Process resolution is primarily driven by the wavelength of the light source used in the fabrication process. The latest extreme ultraviolet lithography (EUV) light sources produce a wavelength of 13.5 nanometers to enable fabrication of structural features smaller than 32 nanometers. In addition, more complex device structures (such as FinFET structures and vertical NAND structures) have been developed to improve overall performance, energy cost, integration and reliability.

[0005] As devices (such as logic and memory devices) move toward smaller nanoscale dimensions, characterization becomes more difficult. Devices incorporating complex three-dimensional geometries and materials with diverse physical properties contribute to characterization difficulties. In general, metrology systems are required to measure devices at more process steps and with higher precision.

[0006] In addition to accurate device characterization, measurement consistency across a range of measurement applications and a group of metrology systems with the same measurement target is also important. If measurement consistency degrades in a manufacturing environment, consistency between processed semiconductor wafers is lost and yield drops to unacceptable levels. Matching measurement results across applications and across multiple systems (i.e., tool-to-tool matching) ensures that measurements on the same wafer for the same application produce the same results.

[0007] The productivity of semiconductor manufacturing facilities is critical to achieving profitability in semiconductor manufacturing. Productivity is directly related to the productivity of individual tools. For example, a single underperforming tool creates a bottleneck that hinders the productivity of the entire production line. Therefore, it is critical to closely monitor the productivity of each tool and promptly resolve performance issues associated with each tool.

[0008] Traditionally, the productivity of each tool is monitored independently of the other tools in the group. Typically, the individual tool productivity metric is expressed in a statistical manner, such as a mean, standard deviation, etc. In addition, the decision on the need for intervention is based on the value of the individual tool productivity metric. In one example, the tool reset rate is calculated independently for each tool in the group, such as the number of tool resets per month. The tool reset rate characterizes the individual tool productivity. The tool reset rate of each respective tool is compared to a baseline and the performance of underperforming tools is addressed.

[0009] Assessing productivity based on individual tool productivity metrics lacks robustness. Typically, production tools operating in a manufacturing facility experience relatively few reset events. A single reset event can trigger an individual tool productivity metric to fall outside an acceptable baseline range of values. In other words, the signal provided by the individual productivity metrics is overcome by noise because the events that drive the signal value are so rare.

[0010] An attempt to address this problem is to simply evaluate individual tool productivity metrics over longer periods of time to improve the computational robustness. For example, the tool reset rate can be calculated as the average tool reset rate over several weeks or months. Unfortunately, this approach has several important limitations. First, extending the period over which individual tool productivity is evaluated delays the discovery of underperforming tools and reduces overall fab productivity over long periods of time. Second, extending the period over which individual tool productivity is evaluated leads to inaccuracies because the baseline range of acceptable values ​​may shift during the extended time interval.

[0011] As metrology systems evolve to measure devices at more process steps and with higher accuracy, assessment of population productivity becomes more complex and less efficient. It is desirable to improve methods and tools to reduce the time and cost associated with maintaining high productivity across a population of metrology tools. Summary of the invention

[0012] Methods and systems for evaluating the productivity of individual semiconductor metrology tools based on both individual tool productivity metrics and group productivity metrics are described herein. The accuracy, speed, and robustness of each individual tool productivity evaluation are improved by including both individual and group productivity metrics. The productivity metrics associated with each individual tool are combined with the productivity metrics associated with a group of tools to identify problem tools quickly and with fewer false positives. In particular, in situations where productivity is driven by low-frequency events, tool productivity results are obtained significantly faster.

[0013] The group productivity evaluation engine characterizes the productivity of each measurement tool in a group of measurement tools and ranks the measurement systems in order of productivity. The ranking can then be used by the user to guide decisions about tool repair and maintenance.

[0014] The productivity data set includes data indicative of individual tool performance characteristics collected from a plurality of individual tools of a group of semiconductor measurement tools. By way of non-limiting example, performance characteristics indicative of tool productivity include tool downtime rate, duration of tool downtime, tool reset rate, time between temporary resets, etc. Productivity metrics are used to numerically characterize individual tool and group productivity. In general, the values ​​of one or more individual tool productivity metrics that characterize the performance of each individual tool in a group of measurement tools are determined independently of the values ​​of one or more group productivity metrics that characterize the performance of the group of measurement tools.

[0015] An individual tool-based productivity metric associated with each individual tool is determined from data productivity data corresponding to each individual tool in the productivity dataset. Similarly, a group-based productivity metric is determined from data productivity data corresponding to a group of individual tools in the productivity dataset.

[0016] In some instances, individual tool productivity metrics and group productivity metrics are determined based on simple statistical measurements, such as the mean and standard deviation of a distribution of performance data, the median of a distribution of performance data, the harmonic mean of a distribution of performance data, the slope of a linear regression performed on a distribution of performance data, and the like.

[0017] In some other examples, individual tool productivity metrics and group productivity metrics are determined based on fitting performance data to an analytical function, such as a Gaussian function, a Poisson function, etc. In some of these examples, the productivity metric values ​​that characterize the performance of an individual tool or a group of tools are parameters of the analytical model.

[0018] In some other examples, individual tool productivity metrics and group productivity metrics are determined based on trained machine learning (ML) based models.

[0019] In one aspect, values ​​of one or more combined productivity metrics associated with each of the individual tools of the cluster of measurement tools are determined based on values ​​of one or more individual tool productivity metrics and values ​​of one or more cluster productivity metrics associated with each individual tool.

[0020] In some examples, the group productivity metric and the individual tool-based productivity metric are combined by selecting a relevant subset of both the group and individual tool-based metrics. In some of these examples, the combined productivity metric is determined by comparing the value of the individual tool productivity metric with the value of the corresponding group productivity metric. In one example, the difference between the average of the tool reset rates associated with each individual tool is compared to the average of the tool reset rates associated with all individual tools in the group. Each difference is a combined productivity metric value associated with the corresponding individual tool.

[0021] In some other examples, the combined productivity metric is determined based on a statistical distance between a distribution of productivity metric values ​​based on individual tools and a distribution of productivity metric values ​​based on a group. The statistical difference is used to quantify how much an individual tool differs from a group of tools.

[0022] In a further aspect, individual tools of the population of measurement tools are ranked based on the values ​​of one or more combined productivity metrics. If an individual tool performs poorly, the individual tool is selected for intervention, i.e., maintenance, repair, or both, in a ranking order determined by the values ​​of the one or more combined productivity metrics. Furthermore, the ranking of individual tools can be based on the one or more combined productivity metrics and the one or more individual productivity metrics.

[0023] In some examples, individual tools are ranked based on at least one combined productivity metric. In some other examples, individual tools are ranked based on at least one combined productivity metric and individual productivity metrics.

[0024] In another further aspect, the value of one or more accuracy metrics is estimated. The accuracy metrics indicate the confidence with which the ranking of individual tools in the population of measurement tools is measured.

[0025] In another further aspect, a probability of a future failure event associated with at least one individual tool of the measurement tool population is predicted based on a difference between a predicted probability distribution of failure events and an actual observed distribution of failure events.

[0026] The foregoing is an overview and therefore necessarily contains simplifications, generalizations, and omissions of details; therefore, those skilled in the art should appreciate that the "Summary" is provided for illustration only and is not intended to be limiting. Other aspects, inventive features, and advantages of the devices and / or processes described herein will be apparent in the non-limiting detailed description set forth herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a diagram illustrating an embodiment of an optical metrology tool for measuring characteristics of a sample according to exemplary methods presented herein.

[0028] Figure 2 is a diagram illustrating a group productivity assessment engine in one embodiment.

[0029] Figure 3 Depicted is a graph illustrating an average value of tool reset rates associated with each individual tool in a population of measurement tools.

[0030] Figure 4 Depicted is a graph illustrating the standard deviation of tool reset rates associated with each individual tool in a population of measurement tools.

[0031] Figure 5 Depicted is a graph illustrating a histogram of tool reset rates associated with a tool population.

[0032] Figure 6 Depicted is a graph illustrating the KL divergence associated with each individual tool of a population of measured tools for tool reset rate.

[0033] Figure 7 Description Figure 6 , in which the bars associated with individual tools having an average tool reset rate below the group average are drawn transparently, while the bars associated with individual tools having an average tool reset rate above the group average are shaded.

[0034] Figure 8 A graph depicting accuracy metrics associated with KL divergence calculations associated with each tool of a tool population.

[0035] Fig. 9 A flow chart illustrating a method 200 for evaluating productivity of a cluster of semiconductor measurement systems in at least one novel aspect. DETAILED DESCRIPTION

[0036] Reference will now be made in detail to background examples and some embodiments of the invention, examples of which are illustrated in the accompanying drawings.

[0037] Methods and systems for evaluating the productivity of individual semiconductor metrology tools based on both individual tool productivity metrics and group productivity metrics are described herein. The accuracy, speed, and robustness of each individual tool productivity evaluation are improved by including both individual and group productivity metrics. The productivity metrics associated with each individual tool are combined with the productivity metrics associated with a group of tools to identify problem tools quickly and with fewer false positives. In particular, in situations where productivity is driven by low-frequency events, tool productivity results are obtained significantly faster.

[0038] Figure 1 An exemplary metrology system 100 for performing measurements of structural features of semiconductor devices according to the exemplary methods presented herein is depicted. Figure 1, the metrology system 100 is configured as a broadband spectroscopic ellipsometer configured to perform measurements of structures within a measurement region 116 of a sample 120 disposed on a sample positioning system 140. However, in general, the metrology system 100 may be configured as any semiconductor metrology tool or inspection tool, including but not limited to optical-based semiconductor metrology tools, x-ray-based semiconductor metrology tools, electron beam-based semiconductor metrology tools, etc.

[0039] The metrology system 100 includes an illumination source 110 that generates a beam illumination light 117 that is incident on a wafer 120. In some embodiments, the illumination source 110 is a broadband illumination source that emits illumination light in the ultraviolet, visible, and infrared spectra. In one embodiment, the illumination source 110 is a laser sustained plasma (LSP) light source (also known as a laser driven plasma source). The pump laser of the LSP light source can be continuous wave or pulsed. The laser driven plasma source can generate significantly more photons than a xenon lamp across a wavelength range from 150 nanometers to 2000 nanometers. The illumination source 110 can be a single light source or a combination of multiple broadband or discrete wavelength light sources. The light generated by the illumination source 110 includes a continuous spectrum or a portion of a continuous spectrum, from ultraviolet to infrared (e.g., vacuum ultraviolet to mid-infrared). In general, the illumination light source 110 can include a supercontinuum laser source, an infrared helium-neon laser source, an arc lamp, or any other suitable light source.

[0040] In a further aspect, the illumination light is a broadband illumination light comprising a wavelength range spanning at least 500 nanometers. In one example, the broadband illumination light comprises wavelengths below 250 nanometers and wavelengths above 750 nanometers. Generally, the broadband illumination light comprises wavelengths between 120 nanometers and 3,000 nanometers. In some embodiments, broadband illumination light comprising wavelengths exceeding 3,000 nanometers may be employed.

[0041] like Figure 1 , the metrology system 100 includes an illumination subsystem configured to direct illumination light 117 to one or more structures formed on a wafer 120. The illumination subsystem is shown to include a light source 110, one or more optical filters 111, a polarization component 112, a field stop 113, an aperture stop 114, and illumination optics 115. The one or more optical filters 111 are used to control the light level, spectral output, or both from the illumination subsystem. In some examples, one or more multi-zone filters are used as optical filters 111. The polarization component 112 produces a desired polarization state exiting the illumination subsystem. In some embodiments, the polarization component is a polarizer, a compensator, or both and may include any suitable commercially available polarization component. The polarization component may be fixed, rotatable to different fixed positions, or rotate continuously. Although Figure 1The illumination subsystem depicted in FIG. 1 includes one polarization component, but the illumination subsystem may include more than one polarization component. Field stop 113 controls the field of view (FOV) of the illumination subsystem and may include any suitable commercially available field stop. Aperture stop 114 controls the numerical aperture (NA) of the illumination subsystem and may include any suitable commercially available aperture stop. Light from illumination source 110 is directed through illumination optics 115 to focus on one or more structures ( Figure 1 The illumination subsystem may include an optical filter 111, a polarization component 112, a field stop 113, an aperture stop 114, and illumination optics 115 of any type and configuration known in the art of spectroscopic ellipsometry, reflectometry, and scatterometry.

[0042] like Figure 1 , a beam of illumination light 117 passes through optical filter 111, polarization component 112, field stop 113, aperture stop 114, and illumination optics 115 as the beam propagates from illumination source 110 to wafer 120. Beam 117 illuminates a portion of wafer 120 at measurement point 116.

[0043] The metrology system 100 also includes a light collecting subsystem configured to collect light generated by the interaction between the one or more structures and the incident illumination beam 117. A beam of collected light 127 is collected from the measurement point 116 by a light collecting member 122. The collected light 127 passes through a collection aperture stop 123, a polarization element 124, and a field stop 125 of the light collecting subsystem.

[0044] The light collecting element 122 includes any suitable optical element for collecting light from one or more structures formed on the wafer 120. The collecting aperture stop 123 controls the NA of the light collecting subsystem. The polarizing element 124 analyzes the desired polarization state. The polarizing element 124 is a polarizer or a compensator. The polarizing element 124 can be fixed, can be rotated to different fixed positions, or can be continuously rotated. Although Figure 1 The collection subsystem depicted in FIG. 1 includes one polarization component, but the collection subsystem may include more than one polarization component. The collection field stop 125 controls the field of view of the collection subsystem. The collection subsystem takes light from the wafer 120 and directs the light through the collection element 122 and the polarization element 124 to focus on the collection field stop 125. In some embodiments, the collection field stop 125 is used as a spectrometer slit of a spectrometer of the detection subsystem. However, the collection field stop 125 may be located at or near the spectrometer slit of the spectrometer of the detection subsystem.

[0045] The light collection subsystem may include light collector 122, aperture stop 123, polarization element 124, and field stop 125 of any type and arrangement known in the art of spectroscopic ellipsometry, reflectometry, and scatterometry.

[0046] exist Figure 1 In the embodiment depicted in , the light collection subsystem directs light to a spectrometer 126. The spectrometer 126 generates an output in response to light collected from one or more structures illuminated by the illumination subsystem. In one example, the detectors of the spectrometer 126 are charge coupled devices (CCDs) that are sensitive to ultraviolet and visible light (e.g., light having a wavelength between 190 nanometers and 860 nanometers). In other examples, one or more of the detectors of the spectrometer 126 are photodetector arrays (PDAs) that are sensitive to infrared light (e.g., light having a wavelength between 950 nanometers and 2500 nanometers). However, in general, other detector technologies (e.g., position sensitive detectors (PSDs), infrared detectors, photovoltaic detectors, etc.) are contemplated. Each detector converts incident light into an electrical signal that indicates the spectral intensity of the incident light. In general, the spectrometer 126 generates an output signal 128 that indicates the spectral response of the structure under test to the illumination light.

[0047] The wafer stage 140 positions the wafer 120 relative to the ellipsometer. In some embodiments, the wafer stage 140 moves the wafer 120 in the XY plane by combining two orthogonal, translational movements (e.g., movements in the X and Y directions) to position the wafer 120 relative to the ellipsometer. In some embodiments, the wafer stage 140 is configured to control the orientation of the wafer 120 relative to the illumination provided by the optical ellipsometer in six degrees of freedom. In one embodiment, the wafer stage 140 is configured to control the azimuth angle AZ of the wafer 120 relative to the illumination provided by the optical ellipsometer by rotating about the z-axis. In general, the sample positioning system 140 may include any suitable combination of mechanical elements for achieving the desired linear and angular positioning performance, including but not limited to a goniometer, a hexapod, an angle stage, and a linear stage. The computing system 130 is communicatively coupled to the wafer stage 140 and transmits a motion command signal 141 to the wafer stage 140. In response, the wafer stage 140 positions the wafer 120 relative to the ellipsometer according to the motion control command.

[0048] The metrology system 100 also includes a computing system 130 for acquiring a signal 128 generated by the detector 126 and determining a property of the structure of interest based at least in part on the acquired signal. Figure 1 , a computing system 130 is configured to receive a signal 128 indicative of a measured spectral response of a structure of interest and estimate a value 129 of one or more parameters of interest (eg, CD, overlay, wafer tilt, etc.) based on the measured spectral response.

[0049] Figure 2A diagram depicting an embodiment of a group productivity assessment engine 150 for characterizing the productivity of each measurement tool in a group of measurement tools and ranking the measurement systems in order of productivity. The ranking can then be used by a user to guide decisions regarding tool repair and maintenance. In some embodiments, the computing system 130 is configured as the group productivity assessment engine 150 described herein. However, in general, any suitable computing system communicatively coupled to a group of measurement tools can be configured as the group productivity assessment engine 150 described herein.

[0050] like Figure 2 As depicted in , an exemplary group productivity assessment engine 150 includes a trained quality control encoder module 152 , a group-based productivity metric module 151 , an individual tool-based productivity metric module 152 , and a combined productivity metric module 153 and optionally includes a tool productivity ranking module 154 .

[0051] like Figure 2 , a productivity dataset 155 is received by the group productivity evaluation engine 150. The productivity dataset 155 includes data indicative of individual tool performance characteristics collected from a number of individual tools of the group semiconductor measurement tools. In some examples, the productivity dataset 155 may be transmitted from a metrology tool (e.g., the metrology tool 100), a network-accessible computing system configured to store productivity data collected from one or more metrology systems, a network-accessible data storage system configured to store productivity data collected from one or more metrology systems, or any combination thereof. By way of non-limiting example, performance characteristics indicative of tool productivity include tool downtime rate, duration of tool downtime, tool reset rate, time between temporary resets, and the like.

[0052] In one instance, Figure 1 The measurement system 100 depicted in is an individual tool of a group of semiconductor measurement tools used in a semiconductor manufacturing facility. However, in general, a group of measurement tools may include any number of the same or different measurement tools used in a semiconductor manufacturing facility.

[0053] Productivity metrics are used to numerically characterize individual tool and group productivity. Generally, the values ​​of one or more individual tool productivity metrics that characterize the performance of each individual tool in a group of measurement tools are determined independently of the values ​​of one or more group productivity metrics that characterize the performance of the group of measurement tools.

[0054] like Figure 2 As depicted, productivity dataset 155 is communicated to individual tool based productivity metrics module 152. Individual tool based productivity metrics module 152 determines one or more individual tool productivity metrics 158 associated with each individual tool from the productivity data corresponding to each individual tool in productivity dataset 155.

[0055] Similarly, productivity data set 155 is communicated to group-based productivity metrics module 151. Group-based productivity metrics module 151 determines one or more group productivity metrics 157 associated with the individual tool group from the productivity data corresponding to the individual tools of the individual tool group in productivity data set 155.

[0056] In some instances, individual tool productivity metrics and group productivity metrics are determined based on simple statistical measurements, such as the mean and standard deviation of the distribution of performance data, the median of the distribution of performance data, the harmonic mean of the distribution of performance data, the slope of a linear regression performed on the distribution of performance data, etc.

[0057] Figure 3 Depicted is a graph 170 illustrating the average of the tool reset rates associated with each individual tool of the group's 23 measurement tools. Each individual tool is assigned a tool identification number plotted on the x-axis. The average of the number of resets per unit time is plotted on the y-axis. Figure 4 Depicted is a graph 175 illustrating the standard deviation of the distribution of tool reset rates associated with each individual tool of the population of 23 measurement tools.

[0058] Figure 3 The average values ​​and Figure 4 The standard deviations depicted in are two statistically based individual tool productivity metrics 158 used to characterize the productivity of each of the 23 tools of the measurement tool population.

[0059] In some other examples, individual tool productivity metrics and group productivity metrics are determined based on fitting performance data to an analytical function, such as a Gaussian function, a Poisson function, etc. In some of these examples, the productivity metric values ​​that characterize the performance of an individual tool or a group of tools are parameters of the analytical model.

[0060] Figure 5 A graph 180 is depicted illustrating a histogram of tool reset rates associated with the 23 tools of the group. Figure 5 As depicted in FIG. 1 , the x-axis is subdivided into 11 different event bins. Each event bin represents a different number of resets per tool per unit time. The number of events corresponding to each event bin is plotted along the y-axis. For example, in a data set describing the number of resets over time in the 23 tools of the group, there are more than 350 events in which the resets are zero times per unit time (e.g., two weeks) in each individual tool of the 23 tools of the group. Similarly, there are more than 100 events in which the resets are once per unit time (e.g., two weeks) in each individual tool of the 23 tools of the group, and so on.

[0061] The distribution of reset events per unit time in the 23 tools of the group is described by the analysis function 171. Figure 5 As depicted in FIG. 1 , a gamma-Poisson function 171 is fitted to the tool reset rate data set to accurately describe the group productivity. In this example, the group productivity metric 157 that characterizes the tool reset rates of the 23 tools in the group is Figure 5 155. The fit parameters of the gamma-Poisson function 171 that fits the tool reset rate distribution plotted in FIG. 156 are shown in FIG. 157. Although the gamma-Poisson function may be used to describe the distribution of events across a group of tools, in general, any suitable mathematical function may be used, such as an exponential distribution, etc. In another example, a group productivity metric 157 that characterizes group performance (e.g., reset events per unit time) is the expected value and variance associated with a Gaussian fit of the productivity data set 155.

[0062] In some other examples, individual tool productivity metrics and group productivity metrics are determined based on trained machine learning (ML) based models. The ML based models are trained based on actual performance data, simulated performance data, or both. In some of these examples, productivity metric values ​​that characterize the performance of an individual tool or a group of tools are parameters of the ML based models. In some examples, incoming performance data is analyzed based on the trained ML model to obtain values ​​for the individual tool productivity metrics and group productivity metrics. In one example, principal component analysis is used to transform incoming performance data into productivity metric values ​​using the trained ML model.

[0063] In one aspect, values ​​of one or more combined productivity metrics associated with each of the individual tools of the cluster of measurement tools are determined based on values ​​of one or more individual tool productivity metrics and values ​​of one or more cluster productivity metrics associated with each individual tool.

[0064] like Figure 2 , the values ​​of the group productivity metric 157 and the values ​​of the individual tool productivity metrics 158 are transmitted to the combined productivity metric module 153. The combined productivity metric module 153 determines the values ​​of one or more combined productivity metrics 159 based on the values ​​of the group productivity metric 157 and the values ​​of the individual tool productivity metrics 158.

[0065] In some examples, the group productivity metric and the individual tool-based productivity metric are combined by selecting a relevant subset of both the group and individual tool-based metrics. In some of these examples, the combined productivity metric is determined by comparing the value of the individual tool productivity metric with the value of the corresponding group productivity metric. In one example, the difference between the average of the tool reset rates associated with each individual tool is compared to the average of the tool reset rates associated with all individual tools in the group. Each difference is a combined productivity metric value associated with the corresponding individual tool.

[0066] In some other examples, the combined productivity metric is determined based on a statistical distance between a distribution of productivity metric values ​​based on individual tools and a distribution of productivity metric values ​​based on a group. The statistical difference is used to quantify how much an individual tool differs from a group of tools.

[0067] In one example, the statistical distance is determined as the Kulbeck-Leibler (KL) divergence illustrated in equation (1), where P(x) is the discrete probability distribution of resets for a single tool and Q(x) is the discrete probability distribution of resets for a group.

[0068]

[0069] Figure 6 A graph 185 is depicted illustrating the KL divergence associated with each individual tool of the group of 23 measurement tools for tool reset rate. Each individual tool is assigned a tool identification number plotted on the x-axis. The KL divergence value is plotted on the y-axis. Figure 6 As depicted in , individual tools are plotted in descending order of KL divergence values. Tools with larger KL divergence values ​​have tool reset rate distributions that diverge from the group distribution of tool reset rates. Conversely, tools with smaller KL divergence values ​​have tool reset rate distributions that more closely compare to the group distribution of tool reset rates.

[0070] In a further aspect, individual tools of the population of measurement tools are ranked based on the values ​​of one or more combined productivity metrics. If an individual tool performs poorly, the individual tool is selected for intervention, i.e., maintenance, repair, or both, in a ranking order determined by the values ​​of the one or more combined productivity metrics. Furthermore, the ranking of individual tools can be based on the one or more combined productivity metrics and the one or more individual productivity metrics.

[0071] like Figure 2 , the values ​​of the combined productivity metric 159, the group productivity metric 157, and the individual tool productivity metric 158 are transmitted to the tool productivity ranking module 154. The tool productivity ranking module 154 ranks the individual tools in the order of urgency for initiating maintenance / repair operations based at least in part on the values ​​of the one or more combined productivity metrics. The tool productivity ranking 160 is transmitted to a memory, such as the memory 132.

[0072] In some examples, individual tools are ranked based on at least one combined productivity metric. Figure 6 As depicted in , individual tools are ranked based on KL divergence values. In one example, individual tools are selected for intervention, i.e., maintenance, repair, or both, based on the KL divergence values. Figure 6 In the example depicted in , tool 18 would be selected first for intervention, followed by tool 22, then tool 8, and so on.

[0073] In some other examples, individual tools are ranked based on at least one combined productivity metric and an individual productivity metric. For example, KL divergence provides a measure of how the distribution of an individual distribution compares to the distribution of a group. However, the performance of an individual tool with a relatively large KL divergence value may be abnormally good or abnormally poor. To solve this dilemma, in some examples, the mean value of the tool reset rate associated with each individual tool is compared with the mean value of the tool reset rate associated with all individual tools in the group. Tools with a mean value of tool reset rates below the group mean are considered acceptable, and tools with a mean value of tool reset rates above the group mean are considered to be intervened, i.e., maintained, repaired, or both. In some other examples, the standard deviation of the tool reset rate associated with each individual tool is compared with the standard deviation of the tool reset rate associated with the group. The standard deviation of the tool reset rate below the group is considered acceptable, and the tool with a standard deviation of the tool reset rate above the group is considered to be intervened, i.e., maintained, repaired, or both. In some other examples, the mean and standard deviation of the tool reset rate associated with each individual tool are compared with the mean and standard deviation of the tool reset rate associated with the group. Tools with both the mean and standard deviation of tool reset rates above the group are considered for intervention, ie, maintenance, repair, or both.

[0074] Figure 7 Description Figure 6 190 of the KL divergence associated with each individual tool of the group of 23 measurement tools of tool reset rate as illustrated in FIG. However, the bars associated with individual tools having an average value of tool reset rate below the group average are drawn transparently, while the bars associated with individual tools having an average value of tool reset rate above the group average are shaded. Figure 7 As shown in , although tools 18 and 22 have relatively high KL divergence values, the average of the tool reset rates is lower than the average of the two tools. Therefore, these tools perform exceptionally well. Figure 7 In the example depicted in , tool 8 would be selected first for intervention, followed by tool 3, then tool 17, and so on.

[0075] In general, tools may be ranked based on any number of productivity metrics. In one example, tools are ranked based on tool reset rate and repair time. In this example, tools that can be repaired faster may be advantageously prioritized over tools with poorer productivity to improve overall cluster performance in a shorter period of time. In another example, tools are ranked based on tool reset rate and the impact of the tool on overall factory productivity. In this example, tools that have a greater impact on overall factory productivity may be advantageously prioritized over tools with poorer individual productivity to improve overall cluster performance.

[0076] In some instances, user input is received by the group productivity assessment engine to determine which productivity metrics to use in the analysis and the relative importance of the selected productivity metrics. In addition, user input can also be used to determine the combination of individual and group metrics related to tool productivity monitoring. In general, many different tool characteristics are responsible for measuring the overall performance of the tool, and most of the relevant productivity metrics vary depending on the tool use case being measured. In some instances, minimizing tool downtime may be of primary importance. In other instances, minimizing temporary tool reset rates may be of primary importance.

[0077] In another further aspect, the value of one or more accuracy metrics is estimated. The accuracy metric indicates the confidence of the ranking of individual tools in the population of measurement tools. In some instances, a p-value analysis is used to estimate the p-value associated with the statistically derived productivity metric. In some instances, a goodness of fit analysis is used to estimate the value of one or more goodness of fit parameters (residual values, etc.) associated with the model-based productivity metric.

[0078] In one example, the uncertainty associated with the calculation of an average value of a productivity metric is high when there are not enough data points (eg, a tool is idle for a long period of time). Figure 8 Graph 195 depicts the accuracy metric values ​​associated with the mean and standard deviation calculations associated with the productivity metric (e.g., tool reset rate) associated with each of the 23 tools in the group. Figure 8 , the accuracy metric 196 associated with tool 21 is relatively small. In this example, the accuracy metric 196 is a p-value associated with the mean and standard deviation calculations associated with tool 21. A relatively low p-value indicates a high uncertainty / low confidence in approximating the mean and standard deviation using the available data set.

[0079] In another further aspect, the probability of a future failure event associated with at least one individual tool of the measurement tool population is predicted based on a difference between a predicted probability distribution of failure events and an actual observed distribution of failure events. In this way, predictions of future tool performance, such as tool downtime rates, tool reset rates, and time between resets, are enabled. Performance predictions enable early intervention to further improve productivity. Additionally, one or more accuracy metrics indicative of confidence in the prediction of future tool performance are calculated as described above. In this way, the confidence in the prediction of future failure events may be considered in determining whether early intervention actions should be taken.

[0080] In one example, the predicted number of future reset events associated with each tool is determined based on an analytical fit to the actual reset probability distribution. P(N) represents the predicted probability of having N resets. P(N) is calculated using individual tool statistics or group statistics. In addition, each tool in the group is known to have a known number of resets that have occurred in the past. The frequency of observed reset events ObsFreq(N) is determined based on the known reset history. The probability of future N resets is determined by comparing the predicted value P(N) with the observed value ObsFreq(N). The probability of future N resets is used as a prediction of the frequency of future reset events for each tool. In addition, a p-value analysis of the statistically derived productivity metric is used to estimate the confidence level of the prediction of the frequency of future reset events. A relatively low p-value indicates a high uncertainty / low confidence in the prediction of the frequency of future reset events.

[0081] In another aspect, the metrology tools comprising a group of metrology tools described herein may include metrology tools of the same or different types. By way of non-limiting example, individual tools of a group of metrology tools include any of a spectroscopic ellipsometer, a spectroscopic reflectometer, a soft x-ray reflectometer, a small angle x-ray scatterometer, an imaging system, a hyperspectral imaging system, a scattering superposition metrology system, etc. In one example, a group of 5 metrology tools may include 3 spectroscopic ellipsometry (SE) metrology tools and 2 SAXS metrology tools.

[0082] In general, an individual semiconductor measurement tool is any measurement tool used in a semiconductor manufacturing facility, including semiconductor metrology tools, semiconductor inspection tools, etc. An individual semiconductor measurement tool may be optically based, x-ray based, electron beam based, or any combination thereof. Furthermore, a group of individual semiconductor measurement tools may include one or more optically based semiconductor measurement tools, one or more x-ray based semiconductor measurement tools, one or more electron beam based semiconductor measurement tools, or any combination thereof.

[0083] like Figure 1 , system 100 includes a single measurement technique (i.e., SE). However, in general, system 100 may include any number of different measurement techniques. By way of non-limiting example, system 100 may be configured as a reflectance small-angle x-ray scatterometer, a soft x-ray reflectometer, a spectroscopic ellipsometer (including Mueller matrix ellipsometry), a spectroscopic reflectometer, a spectroscopic scatterometer, a superposition scatterometer, an angle-resolved beam profile reflectometer, a polarization-resolved beam profile reflectometer, a beam profile reflectometer, a beam profile ellipsometer, any single-wavelength or multi-wavelength ellipsometer, a hyperspectral imaging system, or any combination thereof. Furthermore, in general, measurement data collected by different measurement techniques and analyzed according to the methods described herein may be collected from multiple tools, a single tool integrating multiple techniques, or a combination thereof.

[0084] In further embodiments, system 100 may include one or more computing systems 130 for performing measurements of structures and estimating values ​​of parameters of interest according to the methods described herein. One or more computing systems 130 may be communicatively coupled to detector 116. In one aspect, one or more computing systems 130 are configured to receive measurement data 126 associated with measurements of a structure under test, such as a structure disposed on sample 120.

[0085] In yet another further aspect, the measurement results described herein can be used to provide active feedback to process tools (e.g., lithography tools, etching tools, deposition tools, etc.). For example, the values ​​of the measurement parameters determined based on the measurement methods described herein can be transmitted to the etching tool to adjust the etching time to achieve the desired etching depth. In a similar manner, etching parameters (e.g., etching time, diffusion rate, etc.) or deposition parameters (e.g., time, concentration, etc.) can be included in the measurement model to provide active feedback to the etching tool or deposition tool, respectively. In some examples, corrections to process parameters determined based on the measurement device parameter values ​​can be transmitted to the process tool. In one embodiment, the computing system 130 determines the values ​​of one or more parameters of interest. In addition, the computing system 130 transmits control commands to the process controller based on the determined values ​​of one or more parameters of interest. The control command causes the process controller to change the state of the process (e.g., stop the etching process, change the diffusion rate, etc.). In one example, the control command causes the process controller to adjust the focus of the lithography system, the dose of the lithography system, or both. In another example, the control command causes the process controller to change the etching rate to improve the measurement wafer consistency of the CD parameter.

[0086] In some examples, the measurement model is implemented as the ATmega 300 available from KLA-Tencor Corporation, Milpitas, California, USA. Elements of an optical critical dimension metrology system. In this way, a model is created and ready to use immediately after the spectra are collected by the system.

[0087] In some other examples, the measurement model is implemented, for example, by a computer program product available from K-Tec Corporation of Milpitas, California. The resulting trained model can be incorporated into a database accessible by the metrology system performing the measurement. Elements of the library.

[0088] Fig. 9 A method 200 for evaluating the productivity of a group semiconductor measurement system in at least one novel aspect is described. The method 200 is suitable for use by, for example, the present invention. Figure 1In one aspect, it should be appreciated that the data processing blocks of method 200 may be performed via pre-programmed algorithms executed by one or more processors of computing system 130 or any other general purpose computing system. It should be appreciated herein that specific structural aspects of metrology system 100 are not meant to be limiting, but rather should be interpreted as being for illustration only.

[0089] At block 201 , values ​​of one or more individual tool productivity metrics characterizing the performance of each individual tool in a population of measurement tools operating in a semiconductor fabrication facility are estimated.

[0090] At block 202, values ​​of one or more fleet productivity metrics characterizing performance of a fleet of measurement tools operating in a semiconductor fabrication facility are estimated.

[0091] At block 203, values ​​of one or more combined productivity metrics associated with each of the individual tools of the cluster of measurement tools are determined. The determined values ​​are based on the values ​​of the one or more individual tool productivity metrics associated with each individual tool and the values ​​of the one or more cluster productivity metrics.

[0092] At block 204 , individual tools of the fleet of measurement tools are ranked based on the values ​​of the one or more combined productivity metrics.

[0093] In a further embodiment, system 100 includes one or more computing systems 130 for performing measurements of semiconductor structures based on measurement data according to the methods described herein. One or more computing systems 130 may be communicatively coupled to one or more detectors, active optical elements, process controllers, and the like.

[0094] It should be appreciated that one or more steps described in the present invention may be performed by a single computer system 130 or, alternatively, a plurality of computer systems 130. In addition, the different subsystems of the system 100 may include computer systems suitable for performing at least a portion of the steps described herein. Therefore, the foregoing description should not be interpreted as limiting the present invention, but is merely illustrative.

[0095] In addition, the computer system 130 can be communicatively coupled to other components of the metering system in any manner known in the art. For example, one or more computing systems 130 can be coupled to a computing system associated with a detector. In another example, the detector can be directly controlled by a single computer system coupled to the computer system 130.

[0096] The computer system 130 of the system 100 may be configured to receive and / or obtain data or information from subsystems of the system (e.g., detectors and the like) through a transmission medium that may include wired and / or wireless portions. In this manner, the transmission medium may serve as a data link between the computer system 130 and other subsystems of the system 100.

[0097] The computer system 130 of the system 100 may be configured to receive and / or obtain data or information (e.g., measurement results, modeling inputs, modeling results, reference measurement results, etc.) from other systems through a transmission medium that may include wired and / or wireless parts. In this way, the transmission medium can act as a data link between the computer system 130 and other systems (e.g., memory onboard system 100, external memory, or other external systems). For example, the computing system 130 may be configured to receive measurement data from a storage medium (i.e., memory 132 or external memory) via a data link. For example, the measurement results obtained using the detector described herein may be stored in a permanent or semi-permanent memory device (e.g., memory 132 or external memory). In this regard, the measurement results may be input from the onboard memory or from the external memory system. In addition, the computer system 130 may send data to other systems via a transmission medium. For example, the measurement model or estimated parameter values ​​determined by the computer system 130 may be transmitted and stored in an external memory. In this regard, the measurement results may be output to another system.

[0098] Computing system 130 may include, but is not limited to, a personal computer system, a mainframe computer system, a workstation, a graphics computer, a parallel processor, or any other device known in the art. In general, the term "computing system" may be broadly defined to encompass any device having one or more processors that execute instructions from a memory medium.

[0099] Program instructions 134 implementing a method, such as the method described herein, may be transmitted via a transmission medium such as a wire, cable, or wireless transmission link. Figure 1 As illustrated in FIG. 1 , program instructions 134 stored in memory 132 are transmitted to processor 131 via bus 133. Program instructions 134 are stored in a computer readable medium such as memory 132. Exemplary computer readable media include read-only memory, random access memory, magnetic or optical disks, or tapes.

[0100] As described herein, the term "critical dimension" includes any critical dimension of a structure (e.g., bottom critical dimension, middle critical dimension, top critical dimension, sidewall angle, grating height, etc.), a critical dimension between any two or more structures (e.g., a distance between two structures), and a displacement between two or more structures (e.g., a superposition displacement between superimposed grating structures, etc.). The structure may include a three-dimensional structure, a patterned structure, a superimposed structure, etc.

[0101] As described herein, the term "critical dimension application" or "critical dimension measurement application" includes any critical dimension measurement.

[0102] As described herein, the term "metrology system" includes any system used at least in part to characterize a sample in any aspect, including measurement applications such as critical dimension metrology, overlay metrology, focus / dose metrology, and composition metrology. However, such technical terms do not limit the scope of the term "metrology system" described herein. Additionally, the system 100 can be configured for measurement of patterned wafers and / or unpatterned wafers. The metrology system can be configured as an LED inspection tool, an edge inspection tool, a backside inspection tool, a macro inspection tool, or a multi-mode inspection tool (involving data from one or more platforms simultaneously), and any other metrology or inspection tool that benefits from the techniques described herein.

[0103] Various embodiments are described herein for a semiconductor measurement system (e.g., an inspection system or a lithography system) that can be used to measure samples within any semiconductor processing tool. The term "sample" is used herein to refer to a wafer, a mask, or any other sample that can be processed (e.g., printed or inspected for defects) by means known in the art.

[0104] As used herein, the term "wafer" generally refers to a substrate formed of semiconductor or non-semiconductor materials. Examples include, but are not limited to, single crystal silicon, gallium arsenide, and indium phosphide. Such substrates are typically found and / or processed in semiconductor manufacturing facilities. In some cases, a wafer may include only a substrate (i.e., a bare wafer). Alternatively, a wafer may include one or more layers of different materials formed on a substrate. The one or more layers formed on a wafer may be "patterned" or "unpatterned." For example, a wafer may include multiple die having repeatable pattern features.

[0105] A "reticle" may be a reticle at any stage of a reticle manufacturing process or a complete reticle that may or may not be released for use in a semiconductor manufacturing facility. A reticle or "mask" is generally defined as a substantially transparent substrate having substantially opaque areas formed thereon and configured in a pattern. The substrate may include, for example, a glass material such as amorphous SiO2. The reticle may be placed over a resist-covered wafer during the exposure step of a photolithography process so that the pattern on the reticle may be transferred to the resist.

[0106] The one or more layers formed on a wafer may be patterned or unpatterned. For example, a wafer may include multiple dies, each with repeatable pattern features. The formation and processing of such material layers may ultimately result in a complete device. Many different types of devices may be formed on a wafer, and the term "wafer" as used herein is intended to encompass a wafer on which any type of device known in the art is fabricated.

[0107] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted by a computer-readable medium as one or more instructions or program codes. Computer-readable media include both computer storage media and communication media, which include any media that facilitates the transfer of a computer program from one location to another. Storage media may be any available media that can be accessed by a general or special-purpose computer. By way of example but not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices or any other media that can be used to carry or store desired program code components in the form of instructions or data structures and can be accessed by a general or special-purpose computer or a general or special-purpose processor. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0108] Although certain specific embodiments are described above for the purpose of teaching, the teachings of this patent document have general applicability and are not limited to the specific embodiments described above. Therefore, various modifications, adaptations and combinations of the various features of the described embodiments may be practiced without departing from the scope of the invention set forth in the claims.

Claims

1. A method comprising: estimating values ​​of one or more individual tool productivity metrics characterizing the performance of each individual tool of a population of metrology tools operating in a semiconductor fabrication facility; estimating values ​​of one or more cluster productivity metrics characterizing performance of the cluster of measurement tools operating in the semiconductor fabrication facility; determining a value of one or more combined productivity metrics associated with each of the individual tools of the group of measurement tools, wherein the determined value is based on the values ​​of the one or more individual tool productivity metrics associated with each individual tool and the values ​​of the one or more group productivity metrics; and The individual tools of the population of measurement tools are ranked based on the values ​​of the one or more combined productivity metrics.

2. A method according to claim 1, wherein the determining the value of one or more combined productivity metrics associated with each of the individual tools of the measurement tool group involves determining a statistical distance between the value of an individual tool productivity metric associated with an individual tool of the measurement tool group and the value of a group productivity metric associated with the measurement tool group.

3. The method according to claim 1, further comprising: An individual tool is selected for maintenance based on a value of the one or more combined productivity metrics associated with the individual tool.

4. The method according to claim 1, further comprising: determining a difference between the values ​​of the one or more individual tool productivity metrics associated with each individual tool and an average of the values ​​of the one or more individual tool productivity metrics associated with the individual tools comprising the population of measured tools; and An individual tool is selected for maintenance based on the determined difference and the values ​​of the one or more combined productivity metrics associated with the individual tool.

5. The method of claim 1, wherein the performance of each individual tool of the tool population is measured is tool downtime rate, duration of tool downtime, tool reset rate, time between scheduled resets, time between temporary resets, or any combination thereof.

6. The method of claim 1, wherein at least one of the one or more individual tool productivity metrics characterizing the performance of each individual tool of the fleet of measurement tools is a statistically based metric.

7. The method of claim 1, wherein at least one of the one or more individual tool productivity metrics characterizing the performance of each individual tool of the fleet of measurement tools is a parameter of an analytical or machine learning based model.

8. The method according to claim 1, further comprising: A value of an accuracy metric indicative of confidence in the ranking of individual tools in the population of measurement tools is estimated.

9. The method according to claim 1, further comprising: The probability of a failure event is predicted based on a difference between a predicted probability distribution of future failure events associated with at least one individual tool of the fleet of measurement tools and an actual observed distribution of the failure events.

10. The method of claim 1, wherein each individual tool of the population of measurement tools is any of a spectroscopic ellipsometer, a spectroscopic reflectometer, a soft x-ray reflectometer, a small angle x-ray scatterometer, an imaging system, a hyperspectral imaging system, and a scatter superposition metrology system.

11. A system comprising: an illumination source configured to provide an amount of illumination radiation to one or more structures disposed on the semiconductor wafer; a detector configured to receive an amount of collected radiation from the one or more structures in response to the amount of illuminating radiation and to generate a measurement signal indicative of the collected radiation; and One or more computer systems configured to: estimating values ​​of one or more individual tool productivity metrics characterizing the performance of each individual tool in a population of measurement tools operating in a semiconductor fabrication facility; estimating values ​​of one or more cluster productivity metrics characterizing performance of the cluster of measurement tools operating in the semiconductor fabrication facility; determining a value of one or more combined productivity metrics associated with each of the individual tools of the group of measurement tools, wherein the determined value is based on the values ​​of the one or more individual tool productivity metrics associated with each individual tool and the values ​​of the one or more group productivity metrics; and The individual tools of the population of measurement tools are ranked based on the values ​​of the one or more combined productivity metrics.

12. A system according to claim 11, wherein the determining the value of one or more combined productivity metrics associated with each of the individual tools of the measurement tool group involves determining a statistical distance between the value of the individual tool productivity metric associated with the individual tools of the measurement tool group and the value of the group productivity metric associated with the measurement tool group.

13. The system of claim 11, wherein the one or more computing systems are further configured to: An individual tool is selected for maintenance based on a value of the one or more combined productivity metrics associated with the individual tool.

14. The system of claim 11, wherein the one or more computing systems are further configured to: A value of an accuracy metric indicative of confidence in the ranking of individual tools in the population of measurement tools is estimated.

15. The system of claim 11, the one or more computing systems being further configured to: The probability of a failure event is predicted based on a difference between a predicted probability distribution of future failure events associated with at least one individual tool of the fleet of measurement tools and an actual observed distribution of the failure events.

16. A system comprising: an illumination source configured to provide an amount of illumination radiation to one or more structures disposed on the semiconductor wafer; a detector configured to receive an amount of collected radiation from the one or more structures in response to the amount of illuminating radiation and to generate a measurement signal indicative of the collected radiation; and A non-transitory computer-readable medium storing computer-readable instructions that, when executed by one or more processors of a computing system, cause the computing system to: estimating values ​​of one or more individual tool productivity metrics characterizing the performance of each individual tool in a population of measurement tools operating in a semiconductor fabrication facility; estimating values ​​of one or more cluster productivity metrics characterizing performance of the cluster of measurement tools operating in the semiconductor fabrication facility; determining a value of one or more combined productivity metrics associated with each of the individual tools of the group of measurement tools, wherein the determined value is based on the values ​​of the one or more individual tool productivity metrics associated with each individual tool and the values ​​of the one or more group productivity metrics; and The individual tools of the population of measurement tools are ranked based on the values ​​of the one or more combined productivity metrics.

17. A system according to claim 16, wherein the determining the value of one or more combined productivity metrics associated with each of the individual tools of the measurement tool group involves determining a statistical distance between the value of the individual tool productivity metric associated with the individual tools of the measurement tool group and the value of the group productivity metric associated with the measurement tool group.

18. The system of claim 16, the non-transitory computer-readable medium further storing computer-readable instructions that, when executed by the one or more processors, cause the computing system to: select an individual tool for maintenance based on a value of the one or more combined productivity metrics associated with the individual tool.

19. The system of claim 16, the non-transitory computer-readable medium further storing computer-readable instructions that, when executed by the one or more processors, cause the computing system to: estimate a value of an accuracy metric indicative of confidence in the ranking of individual tools in the group of measurement tools.

20. The system of claim 16, wherein the non-transitory computer-readable medium further stores computer-readable instructions that, when executed by the one or more processors, cause the computing system to: predict a probability of a failure event based on a difference between a predicted probability distribution of future failure events associated with at least one individual tool of the measurement tool group and an actual observed distribution of the failure events.

Images