An etching process control method and device based on digital twinning and deep reinforcement learning

Abstract

The present application relates to a kind of etching process control method and device based on digital twin and deep reinforcement learning, for controlling etching machine, comprising: constructing multiscale residual digital twin architecture;Based on variational trajectory embedding, extract non-Markov state features;Design and train deep reinforcement learning strategy network;Implementation simulation and physical environment difference correction;And execute real-time closed-loop control and dynamic intervention process.This application solves the problem of non-stationary control caused by chamber evolution in high aspect ratio etching and the difference between digital twin simulation and physical environment, can compensate system state drift in real time, improve etching control precision, stability and topography quality.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and semiconductor manufacturing technology, specifically to an etching process control method and apparatus based on digital twins and deep reinforcement learning. Background Technology

[0002] In the field of advanced semiconductor manufacturing processes, as process nodes continue to evolve towards 5 nanometers and below, the etching process for high aspect ratio (HAR) structures has become a key technological factor determining the performance and production yield of semiconductor devices. For example, in applications such as channel etching in 3D NAND flash memory or capacitor via etching in DRAM, the aspect ratio of the etched features typically exceeds 40:1, or even higher. Under such extreme geometric conditions, the physicochemical reaction mechanism of the etching process exhibits a high degree of complexity.

[0003] First, as the etching depth increases, the transport of ions and neutral particles used for material stripping is severely affected by geometrical shielding. The number of reactive particles entering the bottom of high aspect ratio structures decreases sharply, and the incident angle distribution of ions becomes distorted. This leads to a decrease in the etching rate at the feature bottom and easily causes morphological defects such as profile deviation, bottom shrinkage, or distortion.

[0004] Secondly, the etching process generates gaseous byproducts, such as fluorocarbon polymers. These byproducts not only dynamically deposit on the feature sidewalls to form a necessary passivation layer to protect them from lateral etching, but also inevitably accumulate on the inner walls of the reaction chamber, the electrostatic chuck, the electrodes, and the chamber top cover. This accumulation process is long-term and gradually alters the electromagnetic field distribution, gas flow dynamics, and plasma chemical equilibrium state inside the chamber. This slow and continuous change in the physical state of the chamber over time is known in the industry as the chamber evolution effect.

[0005] Because the chamber evolution effect is coupled with the evolution of microstructure features, the entire etching process exhibits non-Markovian and time-varying non-stationary characteristics. This means that the etching rate and morphology evolution at the current moment depend not only on the currently set process parameters, such as RF power, chamber pressure, and gas flow rate, but also on the historical accumulated state of the chamber and the preceding stages of the etching process.

[0006] Currently, most existing digital twin models are based on quasi-steady-state assumptions, typically using simplified physical equations or statistical models to predict etching results. These models struggle to accurately capture the continuous drift of plasma microdynamic parameters caused by byproduct accumulation. Meanwhile, traditional advanced process control methods primarily rely on linear regression models or shallow neural networks, which have limited control accuracy and robustness when dealing with high-dimensional, nonlinear, and high-aspect-ratio etching processes accompanied by random drift. Furthermore, while deep reinforcement learning techniques have shown promise in complex control applications, they often fail to converge when faced with dynamic drift in the system state transition function caused by chamber evolution, due to a lack of awareness of environmental non-stationarity. Moreover, the simulation and physical environment differences between digital twin models and real physical etching equipment due to model simplification widen with increasing process cycles, potentially leading to the failure of the closed-loop control system. Therefore, designing a control scheme that deeply integrates physical mechanisms and deep learning algorithms and effectively addresses non-stationary state drift is a current technical challenge in the field of semiconductor etching technology. Summary of the Invention

[0007] The present invention aims to provide a method and apparatus for controlling the etching process based on digital twins and deep reinforcement learning, which is used to solve the technical problems of non-stationary system state, non-Markov control, and the difference between simulation and physical environment between digital twin model and real physical environment caused by the accumulation of by-products and the evolution of chambers during high aspect ratio etching.

[0008] In one aspect, the present invention provides a method for controlling the etching process based on digital twins and deep reinforcement learning, the method comprising the following steps: Step 1: Construct a multi-scale residual digital twin architecture to simulate the evolution of macroscopic chambers and microscopic morphology and perform online calibration. The architecture includes a macroscopic simulation module, a microscopic morphology evolution module, and a physical residual calibration network. The macroscopic module outputs key physical quantities of plasma based on physical information neural networks, the microscopic module predicts high aspect ratio characteristic geometric boundaries based on implicit neural representations, and the calibration network introduces dynamic equations to model the machine state residuals and performs dynamic compensation. Step 2: Extract non-Markov state features based on variational trajectory embedding. Process sensor sampling data, action commands and morphological change trends within the process cycle through memory-enhanced encoders to extract temporal features, and compress the temporal features into decoupled latent state vectors. Step 3: Train a deep reinforcement learning policy network in the architecture. The policy network uses a distributed reinforcement learning framework combined with a multi-dimensional hierarchical reward function to find the optimal process trajectory. Step four: Implement simulation and physical environment difference correction based on domain randomization and adversarial transfer, and improve the robustness of strategy execution on physical machines by state distribution adversarial alignment. Step 5 involves executing a real-time closed-loop control and dynamic intervention process, including: collecting machine data to generate the current potential state vector, reasoning to obtain the optimal combination of process actions, and conducting a safety assessment before issuing instructions.

[0009] Furthermore, the macroscopic simulation module is a macroscopic plasma simulation module, which is constructed based on a physical information neural network. The physical information neural network embeds a set of partial differential equations describing plasma behavior as regularization constraints into its loss function. The set of partial differential equations includes Maxwell's equations, Boltzmann transport equations, and Navier-Stokes equations. The microscopic evolution module is a microscopic morphology evolution module, which predicts the geometric boundary by mapping the local particle flux to a spatial volume density change field and uses a scalar field function to represent the geometric boundary of the etching features. The physical residual calibration network uses a long short-term memory network to model the machine state residual sequence and calculates the residual between the actual plasma parameters and the module prediction values ​​based on the real-time acquired machine state variables. The kinetic equation is an equation based on the Langmuir adsorption model, and its discretized form is embedded in the hidden state transition function of the network.

[0010] Furthermore, the extraction of non-Markov state features based on variational trajectory embedding in step two specifically includes: constructing a memory-enhanced encoder using a Transformer architecture; using the self-attention mechanism of the memory-enhanced encoder to perform weighted aggregation of sensor sampling data, executed action commands, and morphological change trends over a past process cycle; applying a variational autoencoder to decouple the temporal features extracted by the memory-enhanced encoder for state-space decoupling representation; and introducing a decoupling regularization term to map the temporal features into a low-dimensional latent space to obtain a latent state vector, thereby forcing a portion of the latent state vector to specifically represent the current process parameter state, and another portion of the dimension to specifically represent the chamber evolution index reflecting the degree of chamber aging.

[0011] Furthermore, in step three, a deep reinforcement learning policy network oriented towards process stability is designed and trained. Specifically, this includes: constructing an agent using a distributed reinforcement learning framework, which models the entire probability distribution of future cumulative rewards; the distributed reinforcement learning framework is a quantile regression deep Q-network or an implicit quantile network; defining a multi-dimensional hierarchical reward function, which includes a first-level objective reward and a second-level constraint penalty, and combining the multi-dimensional hierarchical reward function to find the optimal process trajectory; the first-level objective reward includes the increase in etching depth and the morphology similarity score; the second-level constraint penalty includes penalties for severe sidewall distortion, penalties for bottom penetration risk, penalties for drastic jumps in RF power, and penalties for excessive single-step adjustment; and introducing a depth-weighted mechanism so that the closer the etching process is to the target depth, the higher the reward weight of the morphology similarity score.

[0012] Furthermore, the morphological similarity score is measured by the Fraser distance between the target profile curve and the actual predicted profile curve, and the evaluation dimensions include key dimensions, sidewall angles, and bottom twist.

[0013] Furthermore, step four involves implementing simulation and physical environment difference correction based on adversarial transfer, specifically including: introducing a symmetric adversarial transfer architecture, which designs an environment discriminator to distinguish whether a state-action-next state transition sequence comes from the residual digital twin architecture or from a real physical machine; during the learning process of the deep reinforcement learning policy network, adversarial learning is used to ensure that the state distribution triggered by the action sequence generated by it in the residual digital twin architecture cannot be accurately distinguished by the environment discriminator, specifically by minimizing the Jensen-Shannon divergence between the state distribution of the residual digital twin architecture and the state distribution of the real physical machine.

[0014] Furthermore, the simulation and physical environment difference correction based on domain randomization in step four specifically includes: during the training of the deep reinforcement learning policy network in the residual digital twin architecture, applying random perturbations conforming to a probability distribution to the physical parameters in the residual digital twin architecture, wherein the physical parameters include the reflection coefficient of ions on the wall, the wall adhesion coefficient, and the etchant concentration.

[0015] Furthermore, in step five, a safety assessment is performed before issuing the optimal combination of process actions. Specifically, this includes: introducing a safety assessment module based on contraction theory, wherein the safety assessment module predefines a safety invariant set containing the physical constraint boundaries of the process window; assessing whether the action to be executed will cause the system state to deviate from the safety invariant set; if a risk is detected in the action to be executed, then by solving a linear matrix inequality problem, the action is projected back to the nearest safe action boundary to generate a safety-verified action instruction.

[0016] In another aspect, the present invention provides an etching process control device based on digital twins and deep reinforcement learning, for controlling an etching machine. The device includes: a multi-source data integration unit for acquiring machine state acquisition variables from the etching machine; a digital twin computing server for running the multi-scale residual digital twin architecture to simulate the etching process in real time and perform online calibration; a temporal feature extraction engine for performing the extraction of non-Markovian state features based on variational trajectory embedding; an intelligent policy inference core equipped with a deep reinforcement learning policy model trained according to the above method for calculating optimal process adjustment parameters online based on real-time latent state vectors; and a real-time communication and safety intervention gateway for interfacing with the controller of the etching machine, issuing process commands, and running the safety assessment module based on contraction theory.

[0017] The technical solution provided by this invention, through residual digital twin and variational state decoupling technology, enables the system to effectively perceive and compensate in real time for the chamber evolution effect caused by the accumulation of by-products, solving the problem of precise control under system state drift and helping to reduce the deviation of process results during long-cycle etching. Simultaneously, by employing a memory-enhanced Transformer architecture, effective modeling of historical accumulation effects is achieved, solving the problem of nonlocal dependence of the current state on future evolution during the etching of high aspect ratio structures. Furthermore, through domain randomization and adversarial transfer training, the control strategy trained in a simulated environment can be better applied to real physical equipment, helping to shorten the R&D and debugging cycle of new processes. Addressing the physical limitations of the bottom of high aspect ratio structures, the depth-sensitive reward function and dynamic pulse control strategy help solve sidewall distortion and etching stop problems, improving the etching quality of devices under advanced processes. Moreover, the introduced safety intervention mechanism based on contraction theory ensures that deep reinforcement learning always remains within a preset process window during the exploration process, meeting the high stability requirements of semiconductor manufacturing. Attached Figure Description

[0018] Figure 1 A schematic diagram of an etching process control system architecture based on digital twin and deep reinforcement learning is provided according to an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of an etching process control method based on digital twin and deep reinforcement learning according to an embodiment of the present invention.

[0020] Figure 3 This is a detailed block diagram of a multi-scale residual digital twin architecture provided according to an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram illustrating the principle of a symmetric adversarial migration architecture provided according to an embodiment of the present invention.

[0022] Figure 5 This is a flowchart of real-time closed-loop control and safety intervention provided according to an embodiment of the present invention.

[0023] Figure 6 This is a comparison chart of high aspect ratio trench etching control performance provided according to an embodiment of the present invention.

[0024] Figure 7 This is a hardware system connection diagram of an etching process control device according to an embodiment of the present invention.

[0025] Figure 8 This is a schematic diagram of the wafer surface etching feature distribution according to an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments described in this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0030] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0031] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0032] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0033] Example 1 This embodiment provides a method for controlling the etching process based on digital twins and deep reinforcement learning, referring to... Figure 2 This method can be applied to Figure 1 In the control system shown. Figure 1This demonstrates an interactive closed loop between a physical etching machine 101 and a digital control system 102 constructed using the method of this invention. The digital control system 102 drives the digital twin environment by acquiring machine status data in real time and generates optimized process action instructions through deep reinforcement learning algorithms, which are then sent to the machine 101 for execution.

[0034] The following will elaborate on this in detail. Figure 2 The specific steps of the method shown are as follows.

[0035] Step S201: Construct a multi-scale residual digital twin architecture. This step is used to build a high-fidelity digital twin system that can not only simulate the core physicochemical reactions in the etching process, but also calibrate model biases online in a data-driven manner to cope with the evolution of the physical chamber. (Refer to...) Figure 3 The architecture specifically includes a macroscopic plasma simulation module 301, a microscopic morphology evolution module 302, and a physical residual calibration network 303.

[0036] Specifically, a sub-step of this process involves constructing a macroscopic plasma simulation module 301 based on a Physical Information Neural Network (PINN). This module incorporates partial differential equations from plasma hydrodynamics, such as Maxwell's equations describing electromagnetic field distribution and Boltzmann's equations describing particle transport, as penalty terms into the loss function of the deep neural network. This design ensures that while the network learns the mapping relationship between machine-acquired data and plasma states, its output must also satisfy fundamental physical conservation laws, thereby improving the model's generalization ability and prediction accuracy under unseen combinations of process parameters. The module's inputs are real-time process recipe parameters, such as the amplitude and frequency of RF power, bias voltage, type and flow rate of process gases (e.g., a mixture containing fluorocarbons C4F8 and oxygen O2), chamber pressure, and the temperature setpoint of the electrostatic chuck (ESC). The module's output 304 represents key plasma physical quantities at different radial positions on the wafer surface, including plasma density, electron temperature, ion energy distribution function (IEDF), and ion angular distribution function (IADF). These physical quantities will serve as inputs for subsequent microscopic morphology evolution modules. For example... Figure 8 As shown, the wafer surface etching feature distribution involved in the embodiments of the present invention includes the following components. Figure 8(a) shows a top view of wafer I01, which is a typical 300mm diameter wafer with a positioning notch I05 at the bottom for radial positioning. Based on the radial positional relationship, the wafer surface is divided into three concentric ring regions: the central region I02 corresponds to a radial range of 0 to 0.3R, the middle region I03 corresponds to a radial range of 0.3R to 0.6R, and the edge region I04 corresponds to a radial range of 0.6R to R, where R is the wafer radius. The figure uses a dot matrix with different filling densities to represent the etching feature distribution density of each region. The central region uses a high-density dark dot matrix, while the edge region uses a low-density light dot matrix, visually reflecting the spatial differences in etching uniformity. Four sampling points, P0, P1, P2, and P3, are set radially, corresponding to r=0, r=0.3R, r=0.6R, and r=R, respectively, for sampling positions of the plasma physical quantities output by the macroscopic plasma simulation module 301. Figure 8 (b) shows the radial etching rate distribution curve. The horizontal axis represents the normalized radial position r / R, and the vertical axis represents the normalized etching rate. The curve exhibits a typical distribution characteristic of high center and low edge, reflecting the radial non-uniformity of plasma density and ion flux on the wafer surface. This distribution characteristic is a key basis for optimizing etching uniformity. The digital twin module predicts this distribution curve through simulation, and the deep reinforcement learning module dynamically adjusts the process parameters based on the prediction results to improve etching uniformity.

[0037] Another sub-step of this process is to construct a micromorphological evolution module 302 based on implicit neural representation (INR). To overcome the accuracy and computational efficiency bottlenecks of traditional mesh-based or level-set methods in simulating high aspect ratio structures, this invention introduces implicit neural representation technology. This module utilizes a multilayer perceptron to learn a continuous function F(x, y, z, t), which maps any spatiotemporal coordinate (x, y, z, t) to a scalar value representing the matter density at that point. The isosurface where F(x, y, z, t) = 0 defines the geometric boundary of the etching feature at time t. This module receives local particle flux information from the macroscopic module output 304 and, based on a pre-established ion collision physics response model, converts the energy, flux density, and incident angle distribution of ions and neutral particles into the volume density change rate at spatial locations. For example, bombardment by high-energy ions leads to a decrease in volume density at a specific location, while the deposition of byproduct molecules leads to an increase in volume density. By integrating the evolution of the volume density field over time, the cross-sectional profile of high aspect ratio structures, such as changes in critical dimensions (CD), sidewall angles (SWA), and bottom morphology, can be accurately predicted.

[0038] Furthermore, to address the model mismatch issue caused by the chamber evolution effect, this step introduces a physical residual calibration network 303. The core task of this network is to learn and compensate for systematic deviations caused by slowly varying factors such as byproduct accumulation that the main physics model (i.e., the macroscopic plasma simulation module 301 and the microscopic morphology evolution module 302) fails to capture. Specifically, the system uses instrument state acquisition variables (SVIDs), such as the intensity of specific spectral lines monitored by optical emission spectroscopy (OES) and the impedance value of the RF matching network, to calculate in real time the residuals between the actual plasma parameters (obtained indirectly through sensors or surrogate models) and the predicted values ​​of the main physics model. Then, a long short-term memory network, i.e., LSTM (Long Short-Term Memory), is used to model the time series of this residual, thereby learning the dynamic trend of the chamber state drifting over time. Preferably, to enhance the interpretability of the model, this physical residual calibration network is constructed as a semi-physical, semi-empirical evolution operator. Specifically, this involves using simplified kinetic equations describing the deposition and desorption of byproducts on the chamber walls, such as the equation dθ / dt = k_a based on the Langmuir adsorption model. P (1-θ) - k_d θ is embedded as a physical constraint into the hidden state transition function of the LSTM network. Here, θ represents the wall coverage, P represents the partial pressure of byproducts, and k_a and k_d are the adsorption and desorption rate constants, respectively. Specifically, the equation is discretized to match its structure with the gating update logic of the LSTM. In this way, the residual terms learned by the network are no longer purely statistical fitting, but rather a simulation of the evolution of the physical state of the chamber walls. Finally, the learned residual terms are superimposed on the output of the macroscopic plasma simulation module, achieving dynamic online compensation for the digital twin model and ensuring its high synchronization with the state of the physical instrument.

[0039] Step S201 involves extracting non-Markovian state features based on variational trajectory embedding. Due to the non-Markovian nature of the etching system, the evolution of the current state depends on a series of past states and actions. This step compresses the high-dimensional historical time-series data into a low-dimensional, information-rich latent state vector for use by subsequent reinforcement learning algorithms.

[0040] Specifically, this step first constructs a memory-enhanced encoder, preferably using the Transformer architecture (a deep learning network architecture based on a self-attention mechanism). Its input is a time series consisting of machine sensor sampling data from the past K time steps, executed action commands (such as RF power settings), and topographic evolution indicators predicted by a digital twin model (such as etching depth and sidewall angles). The self-attention mechanism within the Transformer model dynamically assigns different weights to different moments in the sequence, effectively capturing the chamber state information of historical moments that have the greatest impact on the current state. This solves the problems of gradient vanishing and the difficulty of modeling long-range dependencies inherent in traditional Recurrent Neural Networks (RNNs).

[0041] Next, a variational autoencoder (VAE) is applied to decouple the temporal features extracted by the encoder for state-space representation. The features output by the encoder are fed into the encoding network of the VAE, which outputs a probability distribution in a low-dimensional latent space, typically the mean and variance of a Gaussian distribution. A latent state vector z is then sampled from this distribution. To give this vector a clear physical meaning, a decoupling regularization term is added to the training objective of the VAE, namely the Evidence Lower Bound (ELBO), for example, using the approach of β-VAE, to encourage different dimensions of the latent vector z to be independent of each other. Through supervised or semi-supervised learning, it is possible to further guide one dimension of z to specifically represent the current process parameter state, while another dimension represents a comprehensive chamber evolution index, which reflects the aging or contamination level of the chamber. In this way, the original non-stationary, non-Markovian process is mapped to a trajectory evolution problem in a static latent space, greatly simplifying the modeling and learning of reinforcement learning strategies.

[0042] Step S203: Design a deep reinforcement learning policy network for process stability. The goal of this step is to train an agent in the digital twin environment constructed in step S201, enabling it to generate the optimal sequence of process control actions based on the latent state features extracted in step S202.

[0043] Specifically, the agent constructed in this step employs a distributed reinforcement learning framework, preferably an implicit quantile network (IQN). Unlike traditional deep Q-networks (DQNs), which only predict the expected value of actions, IQNs can learn the probability distribution of the cumulative reward over the entire future, i.e., the reward distribution. This is crucial in semiconductor manufacturing scenarios because the etching process is accompanied by various random noise sources. By fully modeling the reward distribution, the agent can make risk-sensitive decisions. For example, between two actions with the same expected reward, the agent may tend to choose the action with a smaller variance in the reward distribution, i.e., a more stable outcome, thereby ensuring high stability and repeatability of the process.

[0044] To guide the agent in learning the desired control strategy, this step also defines a multi-dimensional hierarchical reward function. This reward function comprehensively considers the efficiency, accuracy, and stability of the process. The primary objective reward serves as the main driving force and includes two parts: first, the increase in etching depth, encouraging the agent to increase the etching rate; and second, a topographic similarity score, used to evaluate the geometric difference between the current predicted profile and the final target profile. Topographic similarity can be quantified using the Fréchet distance, which can more comprehensively reflect the similarity between two curves, superior to simple point-to-point distance comparison. The secondary constraint penalties are used to avoid undesirable process results, including: negative penalties for severe distortions such as sidewall bending and grooves; strong penalties for etching depths exceeding the substrate and posing a risk of bottom penetration; penalties for drastic jumps in parameters such as RF power or gas flow rate to ensure a smooth process transition; and penalties for excessive single-step action adjustments to prevent excessive burden on the actuator. In addition, to address the higher requirements for morphology control in the later stages of high aspect ratio etching, a depth-weighted mechanism is introduced. As the etching depth approaches the target value, the weight of the morphology similarity score in the total reward will dynamically increase, guiding the agent to perform more refined and conservative fine-tuning actions at the end of the etching process.

[0045] Step S204 involves implementing simulation-physical environment discrepancy correction based on domain randomization and adversarial transfer. Since there will always be modeling errors between the digital twin model and the real physical world—that is, discrepancies between the simulation and the physical environment—this step aims to improve the applicability and robustness of policies trained in the simulation environment on real machines.

[0046] Specifically, one sub-step involves employing domain randomization when training the agent in a digital twin environment. At the beginning of each training iteration, the system applies random perturbations to some key but difficult-to-measure physical parameters in the digital twin model. These parameters include, for example, the reflectivity of ions on feature sidewalls, the adhesion coefficient of byproducts on the wall surface, and the effective concentration of etchant at the bottom of the structure. By training on a range of parameters rather than fixed parameter points, the agent is forced to learn a robust policy that is insensitive to model uncertainty. Preferably, an active search strategy is used for domain randomization, i.e., an auxiliary optimization algorithm is deployed to periodically search the parameter space for the parameter combination that results in the worst performance of the current policy network, and the sampling frequency in these "difficult" regions is increased in subsequent training, thereby efficiently improving the overall robustness of the policy.

[0047] Another sub-step is to introduce a symmetric adversarial migration architecture, see reference Figure 4 The core of this architecture is an environment discriminator 401, a binary classifier tasked with distinguishing whether a state-action-next-state transition sequence originates from the digital twin environment 402 or the real physical machine 403. During training, the policy network (i.e., the agent 404) not only maximizes the cumulative reward but also acts as a generator. The simulated environment state distribution triggered by its generated action sequences needs to "deceive" the environment discriminator 401 as much as possible, making it unable to distinguish between them. Mathematically, this is equivalent to adding an adversarial loss term to the policy learning objective function, which minimizes the Jensen-Shannon divergence between the simulated and real environment state distributions. Through this adversarial training, the policy network actively avoids exploiting "shortcuts" in the digital twin model that deviate from reality, thereby forcing the digital twin model to align with the physical machine in key dynamic characteristics related to the control task, effectively bridging the gap between simulation and physical environments at the algorithmic level.

[0048] Step S205: Execute the real-time closed-loop control and dynamic intervention process. After completing offline training, the solidified policy network is deployed to the actual production environment to execute real-time control.

[0049] Specifically, within a preset control cycle (e.g., 50 milliseconds to 200 milliseconds), the system performs the following operations cyclically. First, it collects real-time machine status data (including SVID, sensor data, and spectral data) and encodes the original time-series data containing historical information into a current potential state vector using the variational trajectory embedding module described in step S202. Second, it inputs this potential state vector into a pre-trained policy network and, through a forward propagation calculation, obtains the optimal combination of process actions for the next moment, such as the fine-tuning of RF power or the percentage adjustment of gas ratio. To ensure computational efficiency, the policy inference process is executed on high-performance dedicated computing hardware. Then, before issuing the action command, a safety assessment module based on contraction theory is introduced. This module predefines a process safety invariant set, which is a multi-dimensional region in the process parameter space representing all permissible and safe operating states. The assessment module determines whether the action to be executed will cause the system state to deviate from this safety invariant set in the next moment. If a potential risk is detected, the system will quickly find a safe alternative action on the boundary of the safety invariant set that is closest to the original dangerous action by solving a linear matrix inequality (LMI) optimization problem. Finally, the safety-verified action commands are sent to the programmable logic controller (PLC) of the etching machine for execution. Simultaneously, the feedback data after the machine executes the action, such as actual changes in plasma parameters, serves as new input for online updating of the physical residual calibration network in step S201, thus forming a continuously learning and self-improving control loop. Figure 5As shown, the real-time closed-loop control and safety intervention process of this invention is executed cyclically within a preset control cycle, specifically including the following steps: The data acquisition layer D01 is responsible for real-time acquisition of machine status data, including multi-source heterogeneous information such as SVID parameters, sensor data, and spectral data. The acquired status data is transmitted to the strategy inference layer for processing. The strategy inference layer includes two core components: the variational trajectory embedding module D02 and the strategy network D03. The variational trajectory embedding module D02 encodes the acquired multi-dimensional status data into a latent state vector, which captures the essential characteristics of the current process status. The strategy network D03 receives the latent vector and performs forward propagation calculation, outputting the optimal combination of process actions as action candidates. The safety assessment layer performs safety verification on the action candidates based on contraction theory. The judgment module D04 evaluates whether the candidate action is within the predefined process safety invariant set. If the action is safe, it is directly transferred to the safety verification module D06 to form a verified action instruction; if there is a risk that may cause the state to deviate from the safety invariant set, the LMI optimization module D05 is triggered to solve for the nearest safe alternative action, which is also verified by D06 before being issued. The diagram uses dashed lines to represent abnormal paths for risk handling, distinguishing them from the normal, safe paths represented by solid lines. The PLC execution module D07 in the feedback layer receives safety-verified action commands and sends them to the etching machine for execution. The execution results are fed back to the physical residual calibration network D08 for online updates. The updated model parameters are transmitted back to the data acquisition layer through the feedback loop (shown by the dotted lines in the diagram), achieving closed-loop optimization of the entire control system. This closed-loop control cycle is typically set to 50-200 milliseconds to ensure the system can quickly respond to changes in process conditions and intervene safely in a timely manner.

[0050] Example 2 This embodiment provides an etching process control device based on digital twins and deep reinforcement learning, used to control an etching machine. This device is the hardware implementation of the method described in Embodiment 1. Specifically, the device includes: a multi-source data integration unit, a digital twin computing server, a temporal feature extraction engine, an intelligent policy inference core, and a real-time communication and security intervention gateway. Figure 7As shown, the etching process control device provided in this embodiment of the invention adopts a distributed hardware architecture to realize the collaborative work of various functional units. The overall system consists of three parts: the etching machine, the main control device, and auxiliary equipment. The etching machine side includes two core components: a sensor network and a programmable logic controller (PLC). The sensor network is responsible for real-time acquisition of process parameters such as temperature, pressure, and flow rate, and transmits the acquired data to the control device via a high-speed data bus. The PLC, as the core controller of the machine's execution layer, interacts bidirectionally with the control device through the SECS / GEM communication protocol. The main body of the control device is marked with a dashed box and integrates five functional units: The multi-source data integration unit H01, located at the top, is responsible for aggregating heterogeneous information from multiple sources, including SVID parameters, spectral monitoring data, and wafer metrology data from the sensor network; the digital twin computing server H02 is equipped with high-performance computing resources such as GPU arrays and TPUs to run a digital twin simulation architecture; the time-series feature extraction engine H03 implements efficient computation of the variational trajectory embedding module in hardware; the intelligent policy inference core H04 carries a trained deep reinforcement learning policy model, outputting the optimal process control policy based on the extracted time-series features; and the real-time communication and safety intervention gateway H05 interfaces with the machine's PLC via the SECS / GEM protocol and has a built-in safety assessment module to verify the safety of candidate control actions. On the auxiliary equipment side, an optical emission spectrometer OES (H06) and an RF probe RF Probe (H07) are installed, transmitting spectral data and RF signals to the multi-source data integration unit via sensor data links, respectively. In the diagram, the thick solid line represents the high-speed data bus, the thin solid line represents the SECS / GEM bidirectional communication protocol, the dashed line represents the sensor data link, and the arrows indicate the direction of data flow. This hardware architecture ensures that the control system achieves an efficient closed-loop process of data acquisition, feature extraction, strategy reasoning, and command issuance within a control cycle of 50 to 200 milliseconds.

[0051] The multi-source data integration unit is responsible for connecting to the control system and sensor network of the etching machine. It acquires various types of data in real time through a high-speed data bus, including machine status acquisition variables (SVID), spectral monitoring data from optical emission spectrometers or broadband RF probes, and wafer metrology data (such as film thickness) from previous processes.

[0052] The digital twin computing server is the computing core of the device, used to run the multi-scale residual digital twin architecture described in step S201 of Embodiment 1 to simulate the etching process in real time and perform online calibration. Preferably, the server is configured with a high-performance graphics processing unit (GPU) array to accelerate the solution of the macroscopic plasma simulation module based on physical information neural networks in parallel, and to use a tensor processing unit (TPU) to perform efficient inference calculations on the microscopic morphology evolution module based on implicit neural representations.

[0053] The temporal feature extraction engine implements the variational trajectory embedding module described in step S202 of Example 1 in hardware or firmware. This engine can compress and encode the high-dimensional raw data stream from the multi-source data integration unit in real time to generate low-dimensional non-Markov latent state vectors for use by subsequent modules.

[0054] The intelligent policy inference core is equipped with a deep reinforcement learning policy model trained offline through steps S203 and S204 in Example 1. This core receives the latent state vector generated by the temporal feature extraction engine and performs fast online inference to calculate the optimal process adjustment parameters for the current state.

[0055] The real-time communication and security intervention gateway serves as the interface between the device and the physical machine. On one hand, it is responsible for distributing the process instructions generated by the intelligent strategy inference core to the machine's programmable logic controller via standard industrial protocols such as the Semiconductor Equipment Communications Standard / Generic Equipment Model (SECS / GEM). On the other hand, it integrates the security assessment module based on contraction theory described in step S205 of Embodiment 1, which performs a final security check before the instructions are issued, ensuring that all output actions are within the preset process window and hardware constraints, thus guaranteeing the safety and stability of the production process.

[0056] Example 3 This embodiment specifically illustrates the application of the method of the present invention in a high aspect ratio (HAR) trench etching process. The goal of this process is to fabricate vertical trenches with an aspect ratio greater than 60:1 on a semiconductor substrate for cell arrays of three-dimensional memory devices. The main technical challenge of this process is the long etching time of a single wafer, which leads to significant chamber evolution effects. Fluorocarbon polymer byproducts generated during etching continuously accumulate on the chamber walls and electrodes, causing a slow drift of the plasma state, which in turn leads to a decrease in trench sidewall verticality (SWA) and a deviation of the bottom critical dimension (CD) from the target value, and even etching stoppage. Conventional process control methods based on fixed formulations or simple feedback loops cannot model and compensate for this non-stationary, non-Markovian system drift, making it difficult to achieve consistent process results in this application.

[0057] To address the aforementioned issues, this embodiment employs the control method described in this invention. First, a multi-scale spatiotemporally decoupled residual digital twin model is constructed and initialized for the target trench structure. The macroscopic plasma simulation module receives the initial process recipe, including the gas flow rates of C4F6 and O2, the RF source power and bias power, and the chamber pressure. The microscopic morphology evolution module is initialized based on the target geometric profile. After the process begins, the physical residual calibration network receives SVID data in real time, such as the CF2 radical spectral line intensity and the impedance value of the RF matching network, from the optical emission spectrometer (OES), for online learning and compensation of model prediction biases caused by changes in the chamber wall state.

[0058] During the etching process, the system operates on a 100-millisecond control cycle. In each cycle, the Transformer encoder in the variational trajectory embedding module processes the SVID data from the past 5 seconds, the sequence of executed actions, and the SWA and CD evolution data predicted by the digital twin, generating a low-dimensional latent state vector. The decoupled "chamber evolution index" dimension in this vector quantifies the current degree of polymer accumulation on the chamber walls.

[0059] The latent state vector is input into a pre-trained policy network based on an Implicit Quantile Network (IQN). The reward function of this policy network is specially designed to impose heavy penalties on deviations in SWA below 89.5 degrees and CD deviations from the target value exceeding 3%, and to increase the weight of morphology accuracy rewards in the latter half of the etching process through a depth-weighted mechanism. Based on the input "chamber evolution index," the policy network can output fine-grained compensatory actions. For example, when an increase in the chamber evolution index is detected, indicating excessive passivation layer growth, the policy network outputs a combination of actions: in the following pulse cycle, increasing the O2 flow rate by 0.5 standard cubic centimeters per minute (sccm) while simultaneously increasing the bias RF power by 5 watts to enhance the ion bombardment energy at the bottom of the trench, removing excess polymer and thus maintaining the verticality of the sidewalls.

[0060] Before being issued, the action command was verified by a safety assessment module based on contraction theory to confirm that its parameter change rate was within the safe range of the machine hardware. After the command was executed, new machine status data was collected to update the residual network and generate the control action for the next moment, forming a closed loop. By applying this method, the final trench profile SWA was stabilized at 89.7 degrees, and the bottom CD deviation was controlled within 2% of the target value, effectively overcoming the chamber drift problem in long-cycle etching processes. Figure 6 The figure shows a performance comparison between the method of this invention and the conventional control method in high aspect ratio trench etching. The horizontal axis represents the number of process cycles, the left vertical axis represents the sidewall angle SWA (in degrees), and the right vertical axis represents the bottom critical dimension CD deviation (in percentage). The figure contains four curves: a solid circular curve represents the sidewall angle variation of the method of this invention, a dashed circular curve represents the sidewall angle variation of the conventional method, a solid square curve represents the CD deviation of the method of this invention, and a dashed square curve represents the CD deviation of the conventional method. As can be seen from the figure, the method of this invention compensates for the chamber evolution effect in real time through the physical residual calibration network in the multi-scale spatiotemporally decoupled residual digital twin architecture, enabling the sidewall angle SWA to be stably maintained around 89.7 degrees, close to the ideal target value of 90 degrees, with a fluctuation range not exceeding 0.3 degrees; simultaneously, the bottom critical dimension CD deviation is always controlled within 2% of the target value, meeting the requirements of high-precision manufacturing. In contrast, traditional control methods, unable to detect and compensate for systematic deviations caused by chamber evolution effects such as byproduct accumulation, result in a gradual decrease in sidewall angles to below 87 degrees and a gradual increase in CD deviation to 6% to 8% as the process cycle increases, severely impacting etching quality. This comparative result verifies the effectiveness of the present invention in modeling the evolution of the physical state of the chamber walls and dynamically compensating for the digital twin model using a physical residual calibration network.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for controlling the etching process based on digital twins and deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Construct a multi-scale residual digital twin architecture to simulate the evolution of macroscopic chambers and microscopic morphology and perform online calibration. The architecture includes a macroscopic simulation module, a microscopic morphology evolution module, and a physical residual calibration network. The macroscopic module outputs key physical quantities of plasma based on physical information neural networks, the microscopic module predicts high aspect ratio characteristic geometric boundaries based on implicit neural representations, and the calibration network introduces dynamic equations to model the machine state residuals and performs dynamic compensation. Step 2: Extract non-Markov state features based on variational trajectory embedding, and process sensor sampling data, action commands and morphological change trends within the process cycle through memory-enhanced encoders to compress long-term states into decoupled potential state vectors. Step 3: Train a deep reinforcement learning policy network in the architecture. The policy network uses a distributed reinforcement learning framework combined with a multi-dimensional hierarchical reward function to find the optimal process trajectory. Step four: Implement simulation and physical environment difference correction based on domain randomization and adversarial transfer, and improve the robustness of strategy execution on physical machines by state distribution adversarial alignment. Step 5 involves executing a real-time closed-loop control and dynamic intervention process, including: collecting machine data to generate the current potential state vector, reasoning to obtain the optimal action combination, and conducting a safety assessment before issuing instructions.

2. The method according to claim 1, characterized in that, In step one, the macroscopic simulation module is a macroscopic plasma simulation module, which is constructed based on a physical information neural network. The physical information neural network embeds a set of partial differential equations describing plasma behavior as regularization constraints into its loss function. The set of partial differential equations includes Maxwell's equations, Boltzmann transport equations, and Navier-Stokes equations. The microscopic evolution module is a microscopic morphology evolution module, which predicts the geometric boundary by mapping the local particle flux to a spatial volume density change field and uses a scalar field function to represent the geometric boundary of the etching features. The physical residual calibration network uses a long short-term memory network to model the machine state residual sequence and calculates the residual between the actual plasma parameters and the module prediction values ​​based on the real-time acquired machine state variables. The kinetic equation is an equation based on the Langmuir adsorption model, and its discretized form is embedded in the hidden state transition function of the network.

3. The method according to claim 1, characterized in that, Step two, which involves extracting non-Markovian state features based on variational trajectory embedding, specifically includes: A memory-enhanced encoder based on the Transformer architecture is constructed. Through the self-attention mechanism of the memory-enhanced encoder, the sensor sampling data, the action commands, and the morphological change trends during the process cycle are weighted and aggregated. A variational autoencoder is applied to decouple the temporal features extracted by the memory-enhanced encoder into a state space. By introducing a decoupling regularization term, the temporal features are mapped to a low-dimensional latent space to obtain a latent state vector. This forces a portion of the latent state vector to specifically represent the current process parameter state, while another portion specifically represents the chamber evolution index that reflects the degree of chamber aging.

4. The method according to claim 1, characterized in that, The design and training of the deep reinforcement learning policy network for process stability in step three specifically includes: Construct an agent using a distributed reinforcement learning framework, which models the entire probability distribution of future cumulative rewards. The distributed reinforcement learning framework is a quantile regression deep Q network or an implicit quantile network. A multi-dimensional hierarchical reward function is defined, which includes a primary objective reward and a secondary constraint penalty. The optimal process trajectory is found by combining the multi-dimensional hierarchical reward function. The primary objective reward includes the increase in etching depth and the morphology similarity score. The secondary constraint penalty includes penalties for severe sidewall distortion, penalties for bottom penetration risk, penalties for drastic jumps in RF power, and penalties for excessive single-step adjustment. A depth-weighted mechanism is introduced, so that the closer the etching process is to the target depth, the higher the reward weight of the morphology similarity score.

5. The method according to claim 4, characterized in that, The morphological similarity score is measured by the Fraser distance between the target profile curve and the actual predicted profile curve, and the evaluation dimensions include key dimensions, sidewall angles, and bottom torsion.

6. The method according to claim 1, characterized in that, Step four involves implementing simulation and physical environment difference correction based on adversarial transfer, specifically including: A symmetric adversarial migration architecture is introduced, which is designed with an environment discriminator to distinguish whether a state-action-next state transition sequence comes from the residual digital twin architecture or from a real physical machine. During the learning process of the deep reinforcement learning policy network, adversarial learning is used to make the state distribution triggered by the action sequence generated by it in the residual digital twin architecture impossible to be accurately distinguished by the environment discriminator. Specifically, this is achieved by minimizing the Jensen-Shannon divergence between the state distribution of the residual digital twin architecture and the state distribution of the real physical machine.

7. The method according to claim 1 or 6, characterized in that, Step four involves implementing domain-based randomization-based simulation and physical environment difference correction, specifically including: During the training of the deep reinforcement learning policy network in the residual digital twin architecture, random perturbations conforming to a probability distribution are applied to the physical parameters in the residual digital twin architecture. The physical parameters include the reflection coefficient of ions on the wall, the wall adhesion coefficient, and the etchant concentration.

8. The method according to claim 1, characterized in that, Step five, which involves conducting a safety assessment before issuing the optimal combination of process actions, specifically includes: A safety assessment module based on contraction theory is introduced, which predefines a safety invariant set containing the physical constraint boundaries of the process window. The system assesses whether the action to be executed will cause the system state to deviate from the safe invariant set. If the action to be executed is found to be risky, the action is projected back to the nearest safe action boundary by solving a linear matrix inequality problem to generate a safe-verified action instruction.

9. An etching process control device based on digital twin and deep reinforcement learning, used to control an etching machine, characterized in that, The device includes: A multi-source data integration unit is used to acquire machine status acquisition variables from the etching machine. A digital twin computing server is used to run the multi-scale residual digital twin architecture of claim 1 to simulate the etching process in real time and perform online calibration; A temporal feature extraction engine is used to perform the temporal feature extraction based on variational trajectory embedding to extract non-Markovian state features as described in claim 1; The intelligent policy reasoning core is equipped with a deep reinforcement learning policy model trained according to the method described in claim 1, which is used to calculate the optimal process adjustment parameters online based on the real-time potential state vector. A real-time communication and security intervention gateway is used to interface with the controller of the etching machine, issue process instructions, and be configured to execute the security assessment steps as described in claim 8.

Images