Method for autonomously generating process formula of semiconductor process system based on reinforcement learning

Abstract

The invention discloses a method and a system for autonomously generating a process formula of a semiconductor process system based on reinforcement learning. The system comprises a comprehensive digital twinborn body which covers each subsystem digital twinborn body, a cavity plasma digital twinborn body and a process digital twinborn body, and the efficiency is improved by adopting a neural network model. By combining the policy neural network with the MCTS, the system can autonomously implement optimized trade-off in the process formulation.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 778,961, filed July 20, 2024. TECHNICAL FIELD

[0002] The present application relates to the field of semiconductor manufacturing, and in particular, to a method and system for autonomously generating process recipes for semiconductor process systems, including etching and deposition. The present application employs advanced digital twin technology, reinforcement learning (RL), and neural networks to optimize process parameters, improve the efficiency and accuracy of semiconductor manufacturing, and reduce manufacturing costs. The method aims to meet strict performance specifications while minimizing recipe time. BACKGROUND

[0003] Semiconductor manufacturing is a precise and complex field that requires precise control of numerous process parameters to ensure the quality and performance of the final product. As technology advances, the complexity of these processes increases, making it increasingly difficult to meet stringent performance requirements. Furthermore, the semiconductor manufacturing industry is facing rising costs due to the need for advanced equipment, materials, and the extensive trial-and-error methods traditionally used in developing process recipes.

[0004] In semiconductor manufacturing, a key issue is the difficulty in achieving desired performance within the shortest recipe time. Current methods rely heavily on experience, expert knowledge, and iterative testing, which not only consume time and resources but also often fail to fully optimize process parameters, making it difficult to achieve both high performance and minimal recipe time.

[0005] Digital twin technology enables the creation of a virtual mapping of a physical system, allowing for detailed simulation and analysis to provide potential solutions. However, the integration of digital twins into the recipe generation process is still limited, and more advanced methods are needed to handle the large number of variables and their complex interactions.

[0006] As a branch of Artificial Intelligence (AI), RL provides a promising approach for autonomously optimizing complex systems. RL can generate efficient and accurate process recipes by training neural networks to learn optimal strategies in simulated interactions with digital twins. The RL process involves extensive learning cases conducted in the background, allowing for extensive exploration of the parameter space. This iterative learning mechanism enables RL systems to strike a balance between high performance demands and shortest recipe time. Incorporating techniques such as Monte Carlo Tree Search (MCTS) into the RL framework further enhances the balance between exploration and exploitation, resulting in better optimization outcomes.

[0007] The present invention addresses these key challenges by providing a method and system for autonomously generating process recipes. The present invention leverages a comprehensive digital twin, RL, and neural networks to effectively optimize process parameters. By conducting extensive learning cases in the background, the RL system can continuously improve its policy neural network, achieving an optimal balance between performance and recipe time. The method of the present invention significantly reduces reliance on traditional empirical methods, shortens development cycles, and improves the overall efficiency and performance of semiconductor process systems while controlling manufacturing costs. The present invention aims to achieve the dual goals of desired performance and shortest recipe time, providing a powerful solution to meet modern semiconductor manufacturing demands. SUMMARY

[0008] In some embodiments, the present invention relates to an advanced method and system for autonomously generating process recipes for semiconductor process systems, particularly suitable for etching and deposition processes. The method employs a comprehensive digital twin, comprising multiple subsystem digital twins such as Radio Frequency (RF), gas, temperature, chamber plasma, surface flux, and process digital twins. To enhance computational efficiency, the present invention utilizes neural networks to construct the digital twin.

[0009] In some embodiments, the method involves all digital twins of the entire process system, including multiple subsystem digital twins. These subsystem digital twins are used to simulate specific aspects of the process system, providing accurate and detailed virtual mappings. By utilizing neural networks to construct the digital twin, the method can replicate the behavior of simulating the subsystem digital twins, enhancing computational efficiency and enabling faster and more efficient simulations.

[0010] In some embodiments, the RL process is employed to generate a process recipe. The RL process includes training a policy neural network and utilizing a MCTS algorithm. The policy neural network includes an input layer, multiple hidden layers, and an output layer for generating a probability distribution of selected process recipe parameters and step times. The output layer includes an output describing a softmax or logistic function for generating the probability distribution. The algorithm encourages more exploration, particularly for step times, to thoroughly examine the parameter space and optimize the process recipe.

[0011] In some embodiments, the RL process generates a total reward for each simulated process case. The total reward includes a performance first reward and a recipe time second reward. The second reward can be converted to the first reward by an exchange rate. The initial exchange rate is assigned by the AI engine and is gradually increased to make the RL process more focused on shortening the recipe time while still meeting the output specifications. Gradually increasing the exchange rate can make the RL process more robust over time, balancing the need for performance and the need to shorten the recipe time.

[0012] In some embodiments, the method is designed to be versatile and applicable to both etching and deposition systems. By employing the comprehensive digital twin and the RL process-based process recipe generation, the method can be adapted to a variety of semiconductor processing application scenarios. The process starts with initiating the RL process, generating simulated process cases and calculating rewards, then updating the weights of the policy neural network based on the total reward until the weights are essentially unchanged, and finally generating a process recipe from the updated policy neural network, which is used for the actual processing of the substrate.

[0013] In certain embodiments, the system components include an AI machine that contains hardware and software modules for optimizing AI applications. The AI machine includes an RL engine with an RL agent, a system digital twin, and an AI engine controller. The system digital twin integrates multiple subsystem digital twins (some of which are trained neural networks) to accurately simulate the entire process system. The integration of multiple subsystem digital twins can provide a comprehensive and detailed simulation environment, which is crucial for accurately generating a process recipe.

[0014] Furthermore, in certain embodiments, the method includes establishing nodes associated with states by the AI engine and expanding the nodes into a network of multiple nodes consisting of state-action pairs. The states are used to describe the substrate being processed, and the actions describe the steps of the process recipe. An enhanced exploration rather than exploitation algorithm is employed when generating actions, which can include an ε-greedy algorithm to ensure thorough exploration of the parameter space to obtain the optimal process recipe.

[0015] In some embodiments, the process system includes an etching process system and a deposition process system. The trained policy neural network can be derived from multiple sets of input-output specifications, which makes the system highly adaptable. Since the training can be performed in the background, large-scale, deep-level neural networks can be applied and massive data can be processed. Taking atomic layer etching (ALE) as an example, the method can develop a general ALE policy neural network applicable to different stack types, critical dimensions, and profile requirements. Therefore, the present application provides a wide range of implementations, including both specialized policy neural networks suitable for specific applications and more general policy neural networks suitable for multiple applications.

[0016] The present application employs a comprehensive digital twin and reinforcement learning combined with MCTS to autonomously generate process recipes, ensuring efficient, accurate, and adaptive semiconductor processing, and providing a robust solution for modern etching and deposition process systems. The RL process continuously optimizes process recipes by performing a large number of learning cases in the background, meeting strict performance specifications while achieving the shortest recipe time, thereby solving key problems in cost and efficiency in semiconductor manufacturing. BRIEF DESCRIPTION OF DRAWINGS

[0017] The following is a brief description of the drawings, which serves to enhance the clarity and understandability of the present application.

[0018] Figure 1 A schematic diagram of an exemplary process system of the present application.

[0019] Figure 2A A functional block diagram of an AI engine for autonomously generating process recipes.

[0020] Figure 2B A detailed functional module schematic diagram of an AI machine containing an AI engine.

[0021] Figure 3 A structural schematic diagram of a system digital twin.

[0022] Figure 4 A neural network representation of a system digital twin.

[0023] Figure 5 A flowchart example of mapping the ALE process to the RL processing process.

[0024] Figure 6 A schematic diagram of the policy neural network structure in the RL process.

[0025] Figure 7 A schematic diagram of an exemplary RL algorithm that uses the MCTS procedure to autonomously generate process recipes.

[0026] Figure 8 A flow chart for a recipe generation process for a RL.

[0027] Figure 9 A process flow chart for a recipe time reduction.

[0028] Figure 10 Table 1 lists design parameters describing the sub-system architecture and topology.

[0029] Figure 11 Table 2 summarizes the structure characteristics before and after ALE process. DETAILED DESCRIPTION

[0030] This section provides a detailed description of the specific embodiments of the present application, so that the reader can gain a thorough and complete understanding of the present application. It should be noted that although certain specific embodiments are described for the purpose of clear explanation of the key points of the present application, all modifications and alterations within the scope defined by the claims should be included in the scope of the present application. These specific embodiments highlight the innovative features of the present application that distinguish it from the prior art.

[0031] Figure 1 An embodiment of a process system 100 is shown. The system is suitable for a plasma enhanced etching or deposition process, such as RIE, ALE, PECVD or ALD. In some embodiments, the sub-systems related to plasma generation can be missing, thus converting the process system 100 into a thermal process system. The inventive concept is generic and can be applied to any type of semiconductor process system, the illustrated vacuum chamber plasma process system is merely an example and does not limit the scope of the present application.

[0032] The process system 100 includes a plasma process chamber 104, which is configured to maintain a vacuum environment suitable for plasma processing. The process system 100 also has a plasma source 106, which receives radio frequency power from an RF power generator 108 via a resonator 110. The plasma source 106 can be implemented in a variety of configurations, such as an inductively coupled plasma (ICP) source or a transformer coupled plasma (TCP) source.

[0033] The RF power generator 108 can operate at a single frequency or multiple frequencies (e.g., 13.56 MHz, 2.0 MHz and 40.0 MHz). The resonator 110 is used to match the output impedance of the RF power generator 108 to the impedance of the plasma process chamber 104 according to the transmission line impedance characteristics. The resonator 110 typically includes inductors and capacitors, and can be configured with a mechanically adjustable capacitor. In some embodiments, the resonator 110 does not include a mechanically adjustable capacitor.

[0034] Impedance matching can be achieved by varying the operating frequency of the RF power generator 108 and the resonator 110. During a process, the plasma can exhibit variable states of different impedance characteristics. To maintain efficient energy transfer and minimize the power reflected from the plasma process chamber 104 back to the resonator 110, the frequency can be fine-tuned for each state of the plasma to ensure that the resonator 110 remains in a resonant state.

[0035] The plasma process chamber 104 is also provided with a chuck 112 to support a substrate 114. Depending on the process requirements, the chuck 112 can be designed as an electrostatic chuck (ESC) or a vacuum chuck. When an ESC is used, the chuck 112 is electrically connected to the RF power generator 116 through the resonator 118. Similar to the resonator 110, the resonator 118 needs to be in a resonant state by adjusting the operating frequency. The operating frequency of the RF power generator 116 can be different from the RF power generator 108. For example, the RF power generator 116 can use a significantly lower frequency than the RF power generator 108.

[0036] The RF power generator 116 provides a bias to the chuck 112, which is transmitted through a DC blocking capacitor (not shown) configured in a standard manner in the art. In addition, a custom waveform generator 117 can also be used to provide a bias to the chuck 112. The custom waveform can significantly narrow the distribution of ion energies, which are generated by the ignition of the plasma 121 in the plasma process chamber 104. Depending on the specific implementation, the custom waveform generator 117 can be connected to the chuck 112 alone or in conjunction with the RF power generator 116 and the resonator 118 to provide the required bias.

[0037] The RF subsystem includes the RF power generator, the resonator, and the plasma source. The operation of the RF subsystem is managed by an RF controller 134, which communicates with and is subordinate to a system controller 132.

[0038] The plasma processing chamber 104 is configured with a gas distribution unit 122 for delivering process gas from a gas source 120 into the plasma processing chamber 104. The gas distribution unit 122 can take various forms such as a gas injector or showerhead, and can include lateral injection structures near the inner walls of the chamber. The gas source 120 typically draws gas from a central gas supply through a gas box, and is regulated in conjunction with valves, pressure regulators, and mass flow controllers (MFCs) to control the flow of gas into the chamber. In other embodiments, a delivery system (not shown) for delivering gaseous, liquid, or even solid precursors can also be employed.

[0039] The plasma processing chamber 104 is also provided with a pump 124 (which can be a turbo-molecular pump or other suitable type) for exhausting gas and byproducts from the plasma processing chamber 104. A valve 126 is typically provided on top of the pump 124 for regulating the chamber exhaust rate. Chamber pressure is monitored by a pressure gauge (not shown) that triggers adjustments to the set point of the actuator of the valve 126 to maintain a constant pressure suitable for vacuum-based plasma processing.

[0040] The gas distribution subsystem (including the gas distribution unit 122, gas source 120, pump 124, and valve 126) is governed by a gas controller 136. The gas controller 136 is connected to the system controller 132 to enable integrated management of the process system 100.

[0041] The plasma processing chamber 104 is also equipped with a temperature control subsystem to maintain the desired thermal conditions for the substrate and chamber. Figure 1 In the illustrated embodiment, the temperature of the chuck 112 is regulated by a temperature controller 138 that controls a heater 128, a chiller 130, and a temperature sensor (not shown). The chuck 112 can be designed as a multi-zone structure with each zone maintained at a different temperature. In addition, temperature control of other components within the plasma processing chamber 104 (such as the gas distribution unit 122 and the various chamber surfaces) can be implemented in a manner that is common in the industry. The temperature control subsystem is controlled by the temperature controller 138, which is connected to the system controller 132.

[0042] Figure 2AAn embodiment of the AI machine 200 is shown. In one implementation, the AI machine is a computer optimized for AI applications through advanced hardware and software modules. The hardware modules include advanced chips such as Graphics Processing Unit (GPU) 240 and High-Bandwidth Memory (HBM) 242. These components are integrated through advanced packaging technology to meet the ultra-high bandwidth requirement of AI applications. The software modules also include Compute Unified Device Architecture (CUDA) 244. These hardware and software modules enable the AI machine 200 to perform efficient parallel computation, for example, for the algorithms of RL.

[0043] The AI machine 200 also includes an AI engine 140 that is capable of autonomously training a policy neural network for generating process recipes. The AI engine 140 further includes an AI engine controller 202 to control the operation of the AI engine. The AI engine controller 202 can be implemented with the GPU 240, HBM 242, and CUDA 244. The AI engine 140 also includes an RL engine 206 that utilizes a system digital twin 204 that simulates the operation of a process system in a virtual environment and contains multiple subsystem digital twins 208 to autonomously generate process recipes through RL.

[0044] Figure 2B More detailed functional modules of the AI engine 140 are shown. The system digital twin 204 includes an RF digital twin 212 for simulating the operation of an RF subsystem, a gas digital twin 214 for simulating a gas subsystem, and a temperature digital twin 216 for simulating a temperature subsystem. The system digital twin 204 also includes a chamber plasma digital twin 218, a surface flux digital twin 220, and a process digital twin 222.

[0045] The RL engine 206 also includes an RL agent 224, which is typically a software program stored in the storage medium of the AI engine controller 202, responsible for performing the RL process. The RL agent 224 employs a policy neural network 226 and a MCTS program 228 to build a search tree and learns by evaluating the relationship between actions and total rewards. Each time a simulation case is completed through the system digital twin 204, a total reward calculator 234 receives inputs from a performance reward calculator 230 and a recipe time reward calculator 232 to perform total reward calculation.

[0046] After the action generated by the policy neural network 226 and the MCTS procedure 228 completes a simulated process case, the RL agent 224 calculates a first reward by a performance reward calculator 230. A cost function is constructed using a squared error function to measure the difference between the generated output parameters of the substrate and the target output parameters. Each term of the squared error function has an associated weight to reflect its relative importance. The RL agent 224 calculates a second reward by a recipe time reward calculator 232. The recipe time is the sum of the step times, which is the duration of a process step. The first reward and the second reward can be discretized into positive or negative integers, for example. The second reward can be converted to the first reward by an exchange rate 236, and then a total reward is generated by the total reward calculator 234.

[0047] The RL agent 224 can set an initial exchange rate before starting the RL process, and then increase the exchange rate to make the RL process more focused on reducing the recipe time.

[0048] In an embodiment, the Al machine 200 can be connected to the system controller 132 through a communication link. In another embodiment, the Al machine 200 can be connected to multiple system controllers of a group of process systems. In yet another embodiment, the Al engine 140 can be incorporated as a component of the system controller.

[0049] Figure 3 A workflow diagram for the system digital twin 204. The RF digital twin 212, the gas digital twin 214, and the temperature digital twin 216 take relevant process recipe parameters and subsystem / system design parameters as inputs. The RF digital twin 212 is used to simulate the RF subsystem, which includes at least the RF power generator and the resonator. In some embodiments, the RF subsystem also includes a custom waveform generator for biasing (although its operating frequency is usually not in the RF range). In an embodiment, the RF digital twin 212 includes SPICE models for the RF circuitry, which can determine the RF power delivered to the plasma source in a time step. Then, combined with the chamber structure parameters, a Maxwell’s equation solver is used to calculate the Electro Magnetic (EM) field distribution in the chamber.

[0050] The recipe parameters received by the RF digital twin 212 include the RF power and initial operating frequency of the steps. System and subsystem design parameters (e.g., RF circuit topology, component values, plasma source structure and parameters, chamber structure parameters, etc.) are typically stored in a storage medium of the AI engine controller 202. Table 1 lists a set of exemplary design parameters for an RF subsystem. The RF digital twin 212 can be used to determine the resonant frequency of the RF subsystem. In another implementation, multiple RF digital twins can be employed. For example, the plasma source and chuck bias can be modeled by different RF digital twins.

[0051] Similarly, the gas digital twin 214 replicates the functionality of the gas subsystem, including components such as the gas sources 120, gas distribution unit 122, pump 124, valves 126, and pressure gauges (not shown). The process recipe parameters received by the gas digital twin 214 include process gas flow rates, etc., for example, for an ALE process, the gas digital twin 214 receives first / second process gas flow rates and chamber pressure for the surface modification step and sputtering step, respectively. Design parameters for the gas delivery system include design parameters for the gas distribution unit (see Table 1 for examples), which, if a showerhead configuration is employed, include parameters such as its size, volume, injection channel / hole distribution and size, etc. In addition, the shape and size of the plasma process chamber are important input parameters for the gas digital twin 214. The outputs of the gas digital twin 214 include the three-dimensional gas distribution (e.g., density, partial pressure, flow rate, and residence time) within the gas distribution unit 122 and process chamber 104. In some implementations, the gas distribution within the lines from the gas sources 120 to the inlet of the gas distribution unit 122 is also modeled. Gas distribution simulation can be implemented using fluid dynamics methods based on finite element techniques or other advanced computational techniques.

[0052] The temperature digital twin 216 replicates the temperature subsystem, including the heater 128, the chiller 130, and the temperature sensors (not shown). In addition to the temperature control of the chuck, the temperature digital twin 216 can also integrate the temperature regulation functions of other chamber components, such as the gas distribution unit 122. The temperature digital twin 216 receives the process recipe parameters, such as the chuck temperature. In some cases, the chuck 112 can be divided into multiple temperature zones, each with a temperature specified by the process recipe individually. The input parameters of the temperature digital twin 216 also include the design parameters of the heater and the chiller (shown in Table 1). For the heater 128, the design parameters include its placement inside the chuck or other chamber components and the operating power range, as well as the thermal conductivities of various materials and their interfaces. The design parameters of the chiller can include the coolant type, the flow rate, and the number and location of the conduction channels (shown in Table 1). The digital twin can employ numerical simulation techniques, such as finite element method, to simulate the temperature distribution of the chuck, the substrate surface, and the inner surface of the plasma processing chamber.

[0053] It is important to note that the independent treatment of the digital twins 212, 214, and 216 can oversimplify the actual situation. For example, the RF power delivered to the chamber can affect the substrate surface temperature, and the interaction between such subsystem digital twins needs to be carefully considered.

[0054] The output values of each subsystem digital twin are input to the chamber plasma digital twin 218. At a specific time step of the process, the chamber plasma digital twin 218 simulates the plasma inside the chamber 104 and outputs the three-dimensional distribution of electrons, ions, and neutral particles. These distributions at specific time instances are functions of the electromagnetic field, the gas environment, and the temperature at that time instance, as well as the electron, ion, and neutral particle distributions at the previous time instance. Therefore, an iterative approach is needed to determine the electron / ion / neutral particle distributions. As shown in FIG. 2, the output of the chamber plasma digital twin at the current time step can be used as the input for its own calculation at the next time step, with each simulation event corresponding to a preset time step defined by the AI engine controller 202. Figure 3

[0055] After obtaining the three-dimensional distribution of ions and neutral particles, the surface flux digital twin 220 calculates and outputs the ion flux and the neutral particle flux towards the substrate surface. In addition, the surface flux digital twin 220 can also output the surface temperature of the substrate in coordination with the temperature digital twin 216. The plasma sheath layer above the substrate has a critical impact on the determination of the ion flux (which significantly affects the etching behavior). The formation mechanism of the plasma sheath layer is well known in the art and can be accurately modeled by the chamber plasma digital twin 218.

[0056] ​The output values of the surface flux digital twin 220 are input to the process digital twin 222 to simulate the processing in the plasma processing chamber 104. Updated substrate parameters or their states are input to the process digital twin 222. The process digital twin 222 determines its output using the current substrate parameter states. Figure 3 The illustrated flow represents a process snapshot of the plasma processing chamber 104 at a time step. Thus, the output of the process digital twin is the evolution of the structure at the time step.

[0057] At each time step, the ion flux and the neutral flux should be statistically accumulated. The detailed parameters of the ion and neutral distributions have important influences on the processing in the plasma processing chamber 104. For the ions, the energy and angular distributions are important and can vary during the step (possibly as a function of the substrate surface location). The output of the surface flux digital twin 220 should include such key details. Similarly, for the neutrals, the density, thermal energy, and activation energy are key parameters for processing at the substrate surface.

[0058] It is important to note that the designs of the RF digital twin, the gas digital twin, the temperature digital twin, the chamber plasma digital twin, the surface flux digital twin, and the process digital twin described herein are exemplary embodiments and can vary in implementation strategies. In some embodiments, the chamber plasma digital twin and the surface flux digital twin can be combined into a single digital twin. In other embodiments, the surface flux digital twin can be combined with the process digital twin. In addition, the RF subsystem digital twin can be split into multiple digital twins to represent the plasma source and the bias unit, respectively. Similarly, the temperature digital twin can be split into two or more independent digital twins, with at least one digital twin for the chuck and one digital twin for the gas distribution unit. All such variations are obvious and should be within the scope of the present disclosure.

[0059] When implementing the digital twins using neural networks, the same strategy of dividing the process system into subsystems can be followed.

[0060] Figure 4A process system is shown as an example of the system neural network 400. In this embodiment, the subsystem digital twin is implemented through multiple neural networks. The RF digital twin 212 serves as the basis for training the RF neural network 402. Using the plasma source 106 connected to the RF power generator 108 and the resonator 110 as an example, a SPICE model is first constructed to simulate the RF power generator 108, resonator 110, and their transmission line effects. The SPICE model outputs the initial AC current and voltage of the plasma source 106 coil, which requires an assumption of the initial impedance of the plasma 121. Subsequently, a numerical simulator applies Maxwell’s equations to predict the electromagnetic field distribution within the plasma process chamber 104.

[0061] The large amount of simulation data generated by the RF digital twin 212 serves as the training set for the RF neural network 402. The inputs to the neural network 402 include the RF circuit topology and related parameters, such as the values of inductors, capacitors, resistors, and transistors within the generator and resonator, as well as detailed modeling of effects and transmission lines. In addition, the RF neural network 402 takes into account chamber structure parameters, including dimensional specifications, locations of the chuck and gas distribution unit, and material properties of these components, as shown in the example of Table 1. Certain parameters are measurable, so they have greater weight in training the RF neural network 402. For example, sensors can track changes in current and voltage in the plasma source coil or reflected power at the resonator output node. B-dot sensors with multiple small coils can be placed within the chamber to map the magnetic field distribution in an experimental setup. The information obtained from these sensors not only provides the basis for the training process but also ensures that the RF neural network 402 closely matches the observed actual behavior.

[0062] When modeling the biasing portion of the RF subsystem using a neural network, the focus is on the initial electric field generated due to the application of RF power. Unlike the magnetic field involved in plasma generation, the biasing deals with the electric field that affects the surface of the substrate.

[0063] Regarding the gas dynamics problem within the process system 100, a gas distribution neural network 404 (which is informed by the gas digital twin 214) is needed. Numerical algorithms based on fluid dynamics are the basis for determining the gas distribution within the chamber 104. This complex interaction involves gas flowing in from the gas distribution unit 122, outflowing gas controlled by the pump 124 and valve 126, while the outflowing process is affected by the chamber conductivity and volume parameters. While numerical simulation provides accuracy, its requirements for computational resources and time make it necessary for real-time applications to require a more efficient method, so the gas distribution neural network 404 is established.

[0064] The gas distribution neural network 404 is trained with simulation data reflecting a variety of parameters, including gas type and flow rate, design of the gas distribution unit 122, capacity of the pump 124, setpoint of the valve 126 actuator, and chamber size and conductivity. Some of the design parameters are listed in Table 1. The gas distribution unit 122 is implemented with either a showerhead, a showerhead with a gas injector, or a combination thereof, which affects the gas distribution within the process chamber 104. The size, number, and distribution of the internal passages / pores of the showerhead and gas injector are important design parameters. The gas pressure within the process chamber is monitored by pressure gauges, and these measurements enhance the training of the gas distribution neural network 404, typically with higher weight than simulation data, to ensure the relevance of the model to the actual conditions.

[0065] Parallel to these developments is the temperature neural network 406 built based on the temperature digital twin 216. The temperature control neural network 406 is specifically designed to map the thermal profile within the plasma process chamber, particularly the thermal profile of the substrate surface. Its training is derived from numerical models that simulate the thermal interactions and distribution. The inputs to the temperature neural network 406 include the chuck and chamber parameters that affect the heat generation and heat conduction. In the case of an electrostatic chuck, the thermal properties of the electrostatic chuck and the heat conduction efficiency, which can be affected by the helium gas pressure as the medium, are critical. Other chamber specifications, such as size and structural materials, also affect the model. Temperature readings from sensors within the chuck 112 and chamber 104 provide valuable actual data, which can have higher weight than simulation data in training the temperature neural network 406 due to the direct measurement of the physical environment. The balance of simulation data and measured data ensures that each neural network closely models the actual process, enabling accurate predictions in the process system.

[0066] Figure 4 The complex structure of the system neural network 400 is illustrated, where the outputs of the RF neural network, gas neural network, and temperature neural network serve as inputs to the chamber plasma neural network 408. The chamber plasma digital twin 218 forms the basis of the chamber plasma neural network 408, enabling accurate characterization of the plasma within the etch chamber. To simulate the particle motion within the plasma, a Monte Carlo or numerical plasma simulator can be used to visualize the three-dimensional distribution of electrons, ions, and neutral particles. This is critical because electrons, being significantly lighter in mass, move much faster than ions, leading to the formation of a sheath layer at the inner surface of the chamber. The sheath layer plays a crucial role in the acceleration of ions towards the substrate, which is essential for sputtering but can have adverse effects during the surface modification phase.

[0067] The training of the chamber plasma neural network 408 incorporates simulation data to enable faster computation and higher efficiency. However, to enhance its predictive capability, the network can also ingest measurement data from the in-chamber sensors, such as optical emission spectrometers and hairpin sensors for measuring electron density. These measurement data are given higher weight than the simulation data to ensure that the output of the plasma neural network 408 is as close to the actual situation as possible.

[0068] The dynamic nature of the plasma environment is captured by the recurrent neural network (RNN) design of the chamber plasma neural network 408. This means that it can handle time-series data, taking a snapshot of the plasma state at a particular time and incorporating it into the model for future prediction. This is a continuous loop process, where the previous output of the neural network becomes part of the input data for the next time step, thereby simulating the continuous evolution of the plasma state.

[0069] When the chamber plasma neural network 408 completes the three-dimensional distribution calculation, the surface flux neural network 410 can determine the ion and neutral particle fluxes reaching the substrate surface. The ion and neutral particle fluxes, along with the surface temperature of the substrate, are then fed as inputs to the process neural network 412. The process neural network 412 can be trained based on data generated by the process digital twin 222. The output of the process neural network 412 also includes the structural evolution of the substrate.

[0070] Finally, the chamber plasma neural network 408 and the surface flux neural network 410 not only produce flux outputs, but also provide key information about the surface temperature by working in conjunction with the temperature neural network 406. The accumulated fluxes over the time step should also contain valuable information about the ion energy and angular distribution, as well as the neutral particle thermal and activation energy. These parameters are crucial for fine-tuning the process within the plasma chamber to achieve the desired etching precision and substrate surface quality.

[0071] It is noted that, Figure 4 A complete neural network implementation embodiment 400 of the system digital twin 204 is shown. In other embodiments or implementations, certain functional modules can not be implemented in the form of neural networks. For example, the surface flux neural network 410 can be an analytical model. Thus, the embodiment 400 is merely an example. There can be multiple variants of implementing selected building blocks of the system digital twin 204 by combining models, lookup tables, analytical models, numerical models, and Monte Carlo models. All such variants fall within the scope of the inventive concept.

[0072] The present disclosure illustrates a system and method for autonomously generating a process recipe by performing a RL process, using ALE process as an example. Figure 5 An ALE process flow 500 suitable for implementing the RL process is shown. The exemplary ALE process is generally performed in a cyclic manner, alternating between a surface modification step A and a sputtering step B. Note that steps A and B described herein are generally referred to as half cycles of the ALE process, which is different from the time steps described earlier for simulating the plasma behavior in the chamber, which is significantly shorter than the duration of steps A and B of the ALE process.

[0073] During the surface modification step A, the surface of the substrate 114 is chemically modified by chemically active neutral particles formed in a plasma generated by a plasma source driven by an RF power generator. A halogen gas such as chlorine is typically introduced to generate the neutral particles for this purpose. During this surface modification step, the bias of the chuck is typically set to zero to minimize the impact of ions on the substrate, thereby preserving the integrity of the ALE process.

[0074] Conversely, during the sputtering step B, an inert gas such as argon is introduced to generate high-energy ions to physically remove the chemically modified layer on the substrate by sputtering. At this time, a bias is typically applied to the chuck by the RF power generator and resonator.

[0075] Between steps A and B, a transition step 514 is employed to effect the process gas switching from step A (508) to step B (510); similarly, between steps B and A, a transition step 516 is employed to effect the process gas switching from step B (510) to step A (508). As shown, step A(a) represents step A at node a, and step B(a) represents step B at node a. Figure 5

[0076] In some cases, particularly when etching high aspect ratio structures, an additional deposition step C (512) can be added in conjunction with steps A and B. The step C is strategically inserted in the ALE cycle sequence, but at a lower frequency than steps A and B, and its main function is to protect the sidewalls of the etched structure from possible lateral etching due to the angular distribution of ions. Before performing step C, a transition step 518 is set to switch to the process gas required for step C; similarly, after completing step C, a transition step 520 is employed to switch back to the process gas required for step A. Step C(b) represents step C at node b.

[0077] In the present disclosure, all process steps and their corresponding step times need to be explicitly planned. The recipe time is the sum of all step times. The process recipe parameters and step times need to be co-optimized by the RL process.​

[0078] The ALE process operates in a cyclical manner, with each cycle consisting of at least one step A and one step B. For example... Figure 5 As shown, an ALE cycle begins in one state and completes in a new state. State 502 describes the substrate being processed. State a represents the state at node a. Specifically, in the ALE process, the state describes one or more structures in the substrate. The description of the state includes, but is not limited to, parameters describing the etched structure, such as depth, critical dimensions, profile, and loading, as shown in Table 2. State 502 is associated with node 504. Therefore, state a is associated with node a. The ALE cycle initially begins at a node with a state, performs action 506 by selecting process recipe parameters using policy neural network 226 and MCTS program 228, and completes at a new node with an updated state. Figure 5 In this context, action (a) represents the action triggered by the ALE recipe at node a.

[0079] It is important to note that a single node can generate multiple nodes through different actions. If the process formula parameters are continuous values, the number of new nodes may tend to be infinite; conversely, if the process formula parameters are discretized into a finite number of levels, the number of new nodes will also be limited accordingly.

[0080] Figure 5 In this example, a complete ALE cycle is considered as one action (for example only). In other embodiments, half-cycles may be used to separate nodes. In this case, the action may be surface modification step A, sputtering step B, or deposition step C. All such variations should fall within the scope of protection of this invention.

[0081] Figure 6 An exemplary policy neural network 226 is illustrated. Taking the ALE process as an example, the policy neural network 226 includes an input layer 602 for receiving the current node state and the desired output specification. It should be noted that the output specification here refers to the final requirement after the entire process is completed, not the requirement of a single step in the process. Using the output specification as one of the inputs to the policy neural network 226 makes it more versatile and adaptable to changes in the output specification. In other embodiments, the input only includes the state.

[0082] The policy neural network 226 further includes one or more hidden layers 604 for processing data from the input layer 602. The policy neural network 226 also includes an output layer 606, which may contain multiple parts, each containing several parameters describing a softmax or logistic function.

[0083] One portion of the output is used for process formulation parameters, such as...Figure 6 Process recipe parameters a (610) and process recipe parameters b (612) are shown. Only two parameters are shown here for illustration, but in practice, more process recipe parameters can be included. a (1), P a (2), P a (3), and P a (4) represent the probability distribution of parameter a at four discrete levels. b (1), P b (2), and P b (3) represent the probability distribution of parameter b at three discrete levels.

[0084] Another part of the output is used for step times, such as Figure 6 614 and 616 are shown. Only two parameters are shown here for illustration, but in practice, all step times (including the time for transition steps) should be included. ta (1), P ta (2), P ta (3), and P ta (4) represent the probability distribution of step A time at four discrete levels. tb (1), P tb (2), and P tb (3) represent the probability distribution of step B time at three discrete levels.

[0085] The output of the policy neural network 226 can not be directly a probability distribution. The output can be the parameters of a softmax function or a logistic function, and the corresponding probability distribution can be calculated from the output parameters.

[0086] In some implementations, the inclusion or exclusion of step C can be modeled by a binary discrete parameter of a logistic function (not shown). Figure 6

[0087] In addition, the output layer includes a value predictor 608 for predicting the value of the state based on the policy represented by the policy neural network under the current weights. The output value is represented as V(s), where V is the value and s is the current state.

[0088] ​In some embodiments, different combinations of ALE formulation parameters can be selected, and each parameter can be selected at different levels. If the process system 100 is used for other process types such as deposition, the parameter selection may differ. This embodiment is for illustrative purposes only and should not be considered a limitation of the invention. Furthermore, the selection of formulation parameters and levels can be dynamic, meaning that the formulation parameters and levels can be adjusted during RL execution. In one embodiment, after executing a predetermined number of simulation cases, the RL agent 224 can decide to narrow the parameter space and adjust the parameter range and levels to accelerate algorithm convergence. In some embodiments, old parameters may be discarded and new parameters enabled. In other embodiments, the entire set of formulation parameters can be selected and determined by executing the RL algorithm. The parameter range is related to the subsystem's capabilities and capacity and is stored in the storage medium of the AI ​​engine controller 202 of the AI ​​machine 200.

[0089] Figure 7 This demonstrates network 700 formed by unfolding the RL process through the execution of the MCTS procedure. (Example) Figure 7 As shown, nodes (such as 702) are represented by circles, and each node is associated with a state (such as S). a1 A parent node can spawn multiple child nodes after performing an action (such as a 704 error), for example, associated with state S. a1 The node can be accessed through action A a1-b1 Transition to associated state S b1 The nodes are described. The RL agent 224 manages the selection process through a policy neural network 226 and an MCTS program 228. For the ALE process, each action represents an ALE cycle with selected process recipe parameters (although a half-cycle is also an option). Action selection continues until a termination state is reached that satisfies the total reward calculation criteria, which are calculated by reward calculators 230 / 232 / 234. For example, in the ALE process, a reward may be calculated when a specific etching depth is reached.

[0090] In this invention, the designed reward system needs to consider a first reward for substrate performance and a second reward for recipe time generated by virtual processing. The second reward can be converted into the first reward through an exchange rate, and the total reward can be calculated after the conversion. The initial exchange rate can be set by the RL agent 224. When a new training cycle is started, the exchange rate can be gradually increased, thereby making the RL process more focused on shortening the recipe time.

[0091] Rewards can be designed based on a cost function. The cost function of a process case is typically expressed as the sum of squared error functions related to each output parameter of the virtual processed structure. This cost function can be defined as:

[0092] Where c is the cost, wi p i is a normalized output parameter, such as a critical dimension at a selected vertical coordinate in an ALE process, p itarget is a normalized target value of the output parameter, N is the parameter number. If multiple structures are evaluated, the cost function can be further expressed as: where C is the cumulative cost of the multiple structures, W j p j is the weight, c k is the cost of a certain structure. This method can cover multiple structures on one substrate (e.g. a 300 mm wafer), and can quantify various loading effects for different structures or different parts of a structure. The first reward can be designed as: R1= f1(c), [3]

[0093] where R1is the first reward, and f1is a function that determines the first reward based on the cost c. In one embodiment, the first reward R1can be designed as multiple discrete values based on the cost, for example, the cost range can be divided into 10 intervals, and each interval is represented by an integer.

[0094] The second reward can be designed as:

[0095] where R2is the second reward, f2is a function that determines the second reward based on the sum of the step times, N represents the number of steps, t k k represents the step time of step k.

[0096] The total reward can be expressed as: R total = R1+ E x R2 [5]

[0097] where E is the exchange rate.

[0098] In another embodiment, the cost function described in equation [1] can be added with a term that is the sum of the square function of the normalized step time multiplied by its weight. The RL process will optimize the cost function that contains both the output structure parameter effect and the step time effect. The weights can be used to adjust the relative importance of each parameter. In addition, the weight of the step time can be gradually increased to achieve the minimum recipe time while meeting the output specification of the structure parameters.

[0099] Each state-action pair (e.g. S a1 ,A a1-b1 ) is part of a state-action chain that obtains a test case of the total reward, and its access frequency will be updated synchronously. When a sufficient number of test cases are completed and the training period is ended, the average reward of each state-action pair can be calculated by dividing the cumulative reward by the access frequency.

[0100] The associated value of a node can be determined by the average reward of all state-action pairs originating from the node. These data can be used to train the policy neural network 226 to focus more on generating high-reward actions.

[0101] In some embodiments, the RL algorithm can be designed to favor exploration over exploitation. For example, in a new training cycle of the RL process, the initial weights of the policy neural network can be randomly assigned, which is an effective technique to avoid the RL process from getting stuck in a local optimum of the recipe parameter space.

[0102] In other embodiments, techniques such as the ε-greedy algorithm, which assigns a fraction of probability to a completely random distribution, can be used to expand the search tree, which is a well-known technique in the art.

[0103] In some embodiments, the techniques to favor exploration over exploitation can be applied specifically to the step time to fully explore the parameter space and achieve recipe time minimization while satisfying the structure parameter output specifications.

[0104] The present disclosure is illustrated by way of example using ALE. Generally, the number of nodes in the RL process can be extremely large. By continuously updating the weights to narrow down the range of actions, the policy neural network 226 can eventually become a deterministic output, and then a practical process recipe can be generated.

[0105] Figure 8 A flowchart of a process 800 is shown, which is a self-starting process to autonomously generate a process recipe through the RL process. The process 800 starts at step 802, where the RL agent 224 starts a training cycle of the RL process. A training cycle is represented as a network of nodes created by the MCTS procedure 228 of the policy neural network 226. Each training cycle contains a plurality of cases, each of which represents a virtual process simulation based on the system digital twin 204. For example, a case of an ALE process generates a complete ALE process that reaches a terminal state, where the structure on the substrate has satisfied a set of criteria (e.g., reaches a target etch depth), which typically involves a chain of actions and a number of intermediate states. A complete training cycle outputs a total reward associated with state-action pairs and node values, where the total reward includes the combined effects of the structure parameter effect and the recipe time effect.

[0106] At step 804, the policy neural network 226 is set with initial weights. In one embodiment, the weights are randomly assigned; in another embodiment, the weights are assigned based on a previous RL training cycle, so that the policy neural network 226 can continuously optimize to generate higher-reward actions.

[0107] Step 806: Establish an initial node of the network. The initial node is associated with a description of an initial state of the incoming substrate, a set of parameters of which are shown in Table 2. At this point, the RL agent 224 applies the policy neural network 226 to generate a probability distribution of selected recipe parameters and selected step times. Based on the probability distribution, an action with determined recipe parameters and associated step times is generated by the MCTS procedure 228 (typically a random number generator is used to generate the action based on the distribution). Subsequently, the RL agent 224 applies the action using the system digital twin 140 to generate a next node with a new state, and the process is repeated until the case is completed.

[0108] In step 808, the network is extended step by step using the policy neural network 226 and the MCTS procedure 228, with each state-action pair being associated with a visit count. Some state-action pairs are involved in multiple cases (reflected by the visit count).

[0109] In step 810, a first reward is calculated based on the performance reward calculator 230, and a second reward is calculated based on the recipe time reward calculator 232. Subsequently, a total reward is calculated by the total reward calculator 234, combining the first reward, the second reward, and a conversion rate.

[0110] If a state-action pair is involved in a particular case, the corresponding total reward is received in step 812. The reward is accumulated with the increase of the visit count, and the average reward of a particular state-action pair is calculated by dividing the accumulated reward by the visit count.

[0111] In step 814, the RL agent 224 evaluates whether the training period is completed. The decision can be made by evaluating the nodes in the network and the total number of completed cases, selected recipe parameters, step times, and discrete levels. If the result is negative, the RL agent 224 continues to expand the network. Otherwise, the RL agent 224 determines the value of each state in step 816. For each node associated with a state, the RL agent 224 has established a relationship between the state-action pair and its associated reward. The node value based on the current policy can be calculated as the average of the total rewards corresponding to all state-action pairs originating from the node.

[0112] In step 818, the RL agent 224 updates the weights of the policy neural network 226 based on all available state-action pairs. At each node, the state is input to the policy neural network 226 and a set of softmax / logistic function parameters is output. The output also includes a predicted value. The updated weights should cause the policy neural network to focus more on generating actions with higher values and to more accurately predict values. As the policy neural network 226 improves, it should become more deterministic in choosing the action from the set of available actions that produces the highest reward. This becomes a typical classification problem, so the cost function used to update the policy neural network 226 should include a cross-entropy loss function and a squared error function of the value. The policy neural network 226 can be trained by utilizing the rewards associated with all actions at a node. In one embodiment, earlier nodes can be given more weight during training to comply with the discounting rule.

[0113] In step 820, the RL agent 224 evaluates whether the weights have converged to a deterministic policy neural network. The evaluation can be performed by comparing the sum of the normalized weight change values to a target value. If the result is negative, the RL agent 224 can initiate a new training cycle to repeat the process and generate more data through further exploration. In one embodiment, an ε-greedy algorithm can be employed to enhance exploration over exploitation. In another embodiment, a new set of initial weights can be applied to the policy neural network 226. In another embodiment, the weights generated from a previous training cycle can be used with an ε-greedy algorithm. The ε-greedy algorithm can be employed exclusively for a selected step time to enhance exploration.

[0114] If the evaluation in step 820 results in a positive, the policy neural network 226 is finalized in step 822, and a process recipe can be generated. The finalized policy neural network 226 can be transmitted to the system controller 132.

[0115] The trained policy neural network 226 can be derived from multiple sets of input and output specifications. Since the training can be performed in the background, it is possible to apply a large, deep neural network and to process large amounts of data. This enables the development of a universal ALE policy neural network that is applicable to different material stack types, critical dimensions, and profile requirements. The present disclosure provides a wide range of implementations, including both specialized policy neural networks that are applicable to a specific application and more general policy neural networks that are applicable to multiple applications, all of which fall within the scope of the present disclosure.

[0116] Figure 9A process flow that minimizes recipe time while meeting the structure parameter output specification is demonstrated. The flow 900 starts at step 902 where an initial exchange rate is set by the RL agent 224, which is used to convert the second reward to the first reward, which can start from a lower value. The policy neural network 226 is trained in step 904, and the process recipe is generated accordingly by applying the flow 800. In step 906, the RL agent 224 evaluates whether the structure parameters have met the output specification. If the evaluation result is positive, the exchange rate is increased by the RL agent 224 in step 908 and the flow 800 is repeated. The flow continues until the performance of the output structure cannot meet the specification requirement. Then in step 910, the RL agent evaluates whether the policy neural network 226 has completed sufficient learning. If yes, the final policy neural network 226 is determined based on the last RL process and the process recipe is generated in step 912; otherwise, the learning process continues. In step 914, the final determined process recipe is deployed in the actual process system for processing of the substrate.

Claims

1. A method of generating a semiconductor process system process recipe, the method comprising: comprises: a) initiating a RL process by an AI engine to establish a process recipe for the process system, wherein the RL process further comprises generating a plurality of simulated process cases by a policy neural network and a MCTS procedure using a system digital twin; b) calculating a total reward for each simulated process case, wherein the total reward comprises a first reward calculated by a performance reward calculator and a second reward calculated by a recipe time reward calculator, wherein the second reward is convertible to the first reward by an exchange rate to calculate the total reward; c) updating weights of the policy neural network based on the total reward until a weight change is below a target value; d) generating a process recipe based on the updated policy neural network; and e) using the process system to actually process a substrate according to the generated process recipe. The policy neural network comprises an input layer, a plurality of hidden layers, and an output layer, wherein the output layer comprises an output describing a softmax or logistic function for generating a discrete level probability distribution of selected process recipe parameters and selected step times.

2. The method of claim 1, wherein, An initial exchange rate is set by an AI agent of the AI engine, and the exchange rate is gradually increased during the RL process to make the RL process more focused on reducing recipe time.

3. The method of claim 1, wherein, The method further comprises establishing a node associated with a state by the AI engine, and expanding the node into a network comprising a plurality of nodes consisting of state-action pairs, wherein the state describes a substrate being processed, and the action describes a step of a process recipe.

4. The method of claim 1, wherein, The method further comprises distributing the total reward to each state-action pair, and updating weights of the policy neural network based on the distributed total reward.

5. The method of claim 4, wherein, The method further comprises employing an enhanced exploration rather than exploitation algorithm for step times when generating actions, wherein the algorithm comprises an ε-greedy algorithm.

6. The method of claim 4, wherein, The AI engine is a component of an AI machine that is connected to a plurality of process systems through a communication link.

7. The method of claim 1, wherein, The AI engine is a component of a system controller of the process system.

8. The method of claim 1, wherein, The system digital twin comprises an RF digital twin, a gas digital twin, a temperature digital twin, a chamber plasma digital twin, a surface flux digital twin, and a process digital twin.

9. The method of claim 1, wherein, At least some of the digital twins are trained neural networks.

10. The method of claim 9, wherein, The process system comprises an etch process system and a deposition process system.

11. The method of claim 1, wherein, comprises:

12. An AI machine, characterized by, a plurality of hardware modules and software modules for optimizing AI applications; and an AI engine constructed based on the hardware modules and software modules, wherein the AI engine further comprises: a RL engine containing a RL agent for autonomously training a policy neural network through a RL process; a system digital twin for generating a plurality of simulated process cases; and an AI engine controller for coordinating operations of the AI engine, wherein the RL agent updates weights of the policy neural network based on a total reward generated from the plurality of simulated process cases to generate a process recipe. The total reward is calculated by the RL agent based on a performance reward calculator and a recipe time reward calculator. 13.The AI machine of claim 12, wherein, ​ 14.The AI machine of claim 13, wherein, The reward calculated by the recipe time reward calculator can be converted to the reward calculated by the performance reward calculator by an exchange rate. 15.The AI machine of claim 14, wherein, The RL agent sets an initial exchange rate and gradually increases the exchange rate during the RL learning process, making the RL process more focused on shortening the recipe time. 16.The AI machine of claim 12, wherein, The total reward is calculated based on a cost function that is the sum of the squared error of the normalized output parameter relative to the normalized output specification and the squared error of the normalized step time, where each term of the function is set with a weight. 17.The AI machine of claim 12, wherein, The system digital twin includes an RF digital twin, a gas digital twin, a temperature digital twin, a chamber plasma digital twin, a surface flux digital twin, and a process digital twin. 18.The AI machine of claim 17, wherein, Some of the digital twins further include a trained neural network. 19.The AI machine of claim 12, wherein, The RL engine further includes a MCTS procedure that is used in conjunction with the policy neural network to generate a probability distribution of selected recipe parameters and selected step times. 20.The AI machine of claim 12, wherein, The AI machine is connected to a plurality of process systems via a communication link, where the plurality of process systems includes etch and deposition process systems.

Images