Single-q table reinforcement learning with multi-head attention for semiconductor valve analysis system

Abstract

The application relates to the technical field of artificial intelligence and semiconductor manufacturing process monitoring, and particularly discloses a semiconductor valve analysis system based on single-Q table reinforcement learning and multi-head attention, which collects parameters through a multi-dimensional perception front end, constructs a state vector by using a heterogeneous data preprocessing module, generates global representation by a multi-head attention feature fusion engine, and outputs optimal instructions by combining a physical mechanism model to limit an action space through a physical constraint guided single-Q table reinforcement learning machine, and finally dynamically updates a value matrix through a closed-loop execution and monitoring feedback module. The application can mine deep feature correlations, ensure that decisions conform to physical laws, significantly improve learning efficiency and control stability, and guarantee semiconductor production safety.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and semiconductor manufacturing process monitoring technology, specifically involving a semiconductor valve analysis system that integrates single Q-table reinforcement learning with multi-head attention. Background Technology

[0002] Semiconductor manufacturing, as the core foundation of the modern electronics industry, places extremely stringent requirements on the precision of environmental parameters and fluid control in its processes. To achieve autonomous control of complex physical processes, intelligent analysis and control systems based on reinforcement learning are widely used in the field of precision manufacturing. These systems aim to replace traditional human experience or fixed control logic by establishing an interactive feedback mechanism between the intelligent agent and the environment, thereby enabling real-time evaluation and parameter optimization of the operating status of semiconductor production equipment.

[0003] As a crucial component in controlling fluid transport, the performance of semiconductor valve analysis systems directly determines the pressure stability and chemical gas flow accuracy within the process chamber. These systems typically utilize reinforcement learning frameworks to model the correlation between valve action responses and system states. Through continuous iteration of the state-action value function, they seek optimal control paths and analysis strategies in dynamically changing production environments to meet the stringent stability requirements of nanoscale manufacturing processes.

[0004] Existing technologies primarily rely on purely data-driven reinforcement learning algorithms, which often reveal a lack of physical and logical constraints in decision-making when faced with rare operating conditions or nonlinear disturbances encountered in semiconductor manufacturing. Because the models lack a deep representation of physical mechanisms such as fluid dynamics, the systems are prone to outputting non-physical decisions that violate physical laws, such as generating instantaneous reverse pressure jumps—operations that are theoretically infeasible and could damage high-value equipment in practice. Furthermore, because the models need to explore a vast state space in an unordered manner, existing systems suffer from severe sample efficiency problems, resulting in excessively long training cycles and difficulty adapting to the demands of rapid process changes. In addition, traditional analysis methods suffer from weak information correlation when dealing with multi-dimensional process features, failing to effectively capture the deep dependencies between key parameters, resulting in decision-making accuracy and convergence speed under multiple constraints that cannot meet the requirements of industrial applications. Summary of the Invention

[0005] The purpose of this invention is to provide a semiconductor valve analysis system that integrates single Q-table reinforcement learning with multi-head attention, in order to solve the technical problems of inaccurate decision-making caused by the lack of physical logic, low sample efficiency, and weak correlation of multi-dimensional features in existing semiconductor valve control technologies.

[0006] The technical solution of this invention is a semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention, comprising:

[0007] A multi-dimensional sensing front end is used to collect fluid physical parameters, valve mechanical status parameters, and environmental interference parameters in the semiconductor process chamber in real time.

[0008] The heterogeneous data preprocessing module is used to perform time-domain feature extraction, frequency-domain transformation, and data normalization on the raw signals acquired by the multi-dimensional sensing front end, and to construct an initial state vector sequence with unified time-scale features.

[0009] The multi-head attention feature fusion engine is used to calculate the correlation weights between different physical dimensions based on the initial state vector sequence using multiple parallel self-attention heads, identify key process features that have a decisive impact on system stability, and generate an enhanced global representation vector.

[0010] A physical constraint-guided single Q-table reinforcement learning machine is used to receive the enhanced global representation vector, store state and action value information through a pre-set single Q-table structure, and combine it with the embedded physical mechanism model to restrict the legality boundary of the action space, so as to output the optimal valve analysis command through iterative learning.

[0011] The closed-loop execution and monitoring feedback module is used to convert valve analysis commands into electrical control signals and send them to the actuator. It also captures the system state response after execution and dynamically updates the value matrix in the physical constraint-guided single Q-table reinforcement learning machine by calculating the deviation between the actual response and the expected target.

[0012] Furthermore, the multi-dimensional sensing front end consists of a pressure sensor array, a mass flow control and monitoring unit, a high-precision displacement sensor, and a temperature compensation unit; the pressure sensor array is distributed upstream and downstream of the valve to capture instantaneous pressure gradients; the mass flow control and monitoring unit is used to acquire gas molar flow rate data; the high-precision displacement sensor is used to monitor the absolute opening position of the valve core in real time; and the temperature compensation unit is used to eliminate the influence of thermal drift on sensor accuracy.

[0013] In one embodiment of the present invention, the heterogeneous data preprocessing module performs the following steps: high-speed sampling of the acquired continuous analog signal at a frequency of 1000 Hz, and applying a 5th-order Butterworth low-pass filter to remove high-frequency noise components; extracting the mean, variance, and first-order rate of change of each parameter within the first 100 milliseconds of the current time using a sliding window algorithm; and mapping data of different dimensions to a numerical range of 0 to 1 using the minimum-maximum normalization method to eliminate the influence of the difference in magnitude of each parameter on the model convergence speed.

[0014] Furthermore, the multi-head attention feature fusion engine comprises a linear mapping layer, a multi-head attention computation layer, and a residual connection layer. The linear mapping layer projects the initial state vector sequence into multiple independent feature spaces, generating a query matrix, a key matrix, and a value matrix, respectively. The multi-head attention computation layer consists of eight independent attention heads. Each attention head measures the correlation strength between physical features by calculating the scaled dot product of the query matrix and the key matrix, and uses a normalized exponential function to weight the correlation scores. Finally, the outputs of each head are concatenated and fused. The residual connection layer is used to sum the fused features with the initial state vector to prevent the gradient vanishing problem in the deep feature extraction process.

[0015] As one embodiment of the present invention, the physical constraint-guided single Q-table reinforcement learning machine introduces a boundary check operator based on fluid dynamics equations during the decision-making process; the physical mechanism model pre-stores the flow velocity limit values ​​of the semiconductor valve under different pressure ranges and the acceleration threshold during the opening and closing process; when the reinforcement learning algorithm attempts to output a certain action command, the boundary check operator will calculate the pressure change rate and flow velocity deviation that the action may cause in real time; if the prediction result exceeds the safety boundary set by the physical mechanism model, the system will automatically penalize the Q value corresponding to the action and forcibly redirect the decision to an adjacent safe action range that conforms to physical laws.

[0016] Furthermore, the value update logic of the single Q-table reinforcement learning machine is as follows: the state space is defined as the pressure difference, flow deviation, and valve opening change at the current moment; the action space is the step increment of the valve opening; the Bellman equation is used as the core update function, where the discount factor is set to 0.95, and the learning rate adopts a dynamic decay strategy, with an initial learning rate of 0.1, which gradually decays to 0.001 as the number of training steps increases; the reward function design includes a goal achievement reward and a physical violation penalty. If the system reaches the preset pressure stability point, a positive reward is given; if a physical constraint is triggered, a negative reward is given.

[0017] In one embodiment of the present invention, the closed-loop execution and monitoring feedback module includes a digital signal conversion unit, a power drive control unit, and an anomaly diagnosis unit; the digital signal conversion unit converts valve analysis commands into high-resolution pulse width modulation signals; the power drive control unit drives a servo motor or piezoelectric ceramic actuator to operate according to the signal; the anomaly diagnosis unit determines whether the system has mechanical wear or sensor failure by comparing the residual between the actual sensor feedback and the physical model prediction, and triggers system safety shutdown protection when the residual exceeds a preset 15% threshold.

[0018] Furthermore, the multi-head attention feature fusion engine can automatically identify the most influential parameter group under a specific process step by statistically analyzing the spatial distribution of attention; in the deposition process stage, the system increases the weight of pressure sensor data; in the etching process stage, the system increases the weight of flow control data; this dynamic weight adjustment mechanism enables the system to adapt to the complex semiconductor multi-step production environment.

[0019] As one embodiment of the present invention, the single Q-table reinforcement learning machine adopts a discretized state space representation method, which divides the continuous physical parameters into 1024 subdivided logical states to ensure that the single Q-table can achieve efficient querying and updating within a limited memory space. In order to improve the exploration efficiency, the system adopts a 10% random exploration probability in the initial stage, and reduces the exploration probability to 1% as the Q value converges.

[0020] Furthermore, the physical constraint-guided single Q-table reinforcement learning machine also includes an offline pre-training stage and an online incremental learning stage; the offline pre-training stage uses a large amount of historical process data to initialize the Q-table, enabling the system to have preliminary decision-making capabilities; the online incremental learning stage fine-tunes the Q-table based on real-time feedback during actual production to compensate for the impact of equipment aging or batch-to-batch process differences.

[0021] In one embodiment of the present invention, each attention head in the multi-head attention computing layer focuses on a different feature dimension; the first to third attention heads focus on capturing long-term pressure fluctuation trends; the fourth to sixth attention heads focus on capturing coupling interference between different gas paths; and the seventh and eighth attention heads focus on capturing the instantaneous lag relationship between valve action commands and flow responses.

[0022] Furthermore, the heterogeneous data preprocessing module also integrates a state estimation algorithm based on Kalman filtering, which is used to reconstruct the missing state in combination with the physical prediction model when sensor data is partially lost or subject to high-intensity noise interference, ensuring the continuity and consistency of the data stream provided to the multi-head attention feature fusion engine.

[0023] As one embodiment of the present invention, the physical mechanism model also considers the compressibility factor and adiabatic index of the gas, and can dynamically correct the legal boundary of the action space based on the real-time monitored temperature and pressure parameters when analyzing the behavior of special gas fluids under high pressure or ultra-low temperature.

[0024] Furthermore, the closed-loop execution and monitoring feedback module establishes a high-speed cache to record the state-action pairs and corresponding reward values ​​of the most recent 5000 control cycles in real time. These data are periodically used to recalibrate the hyperparameters in the physical constraint-guided single Q-table reinforcement learning machine, thereby achieving system self-optimization.

[0025] As one embodiment of the present invention, the action step amount in the single Q-table reinforcement learning machine has undergone nonlinear quantization processing; in the nonlinear sensitive region where the valve opening is close to fully closed or fully open, the resolution of the action step amount is improved to 0.01%; in the intermediate linear region, the resolution of the action step amount is set to 0.1%. This variable resolution design effectively balances control accuracy and response speed.

[0026] Furthermore, the global representation vector output by the multi-head attention feature fusion engine is not only used to guide the decision-making of the reinforcement learning machine, but is also fed into a prediction branch to predict the chamber pressure trend within the next 50 milliseconds. The prediction result of the chamber pressure trend is used as a feedforward compensation term and connected to the closed-loop execution and monitoring feedback module, which further improves the system's ability to suppress sudden disturbances.

[0027] As one embodiment of the present invention, when the physical constraint-guided single Q-table reinforcement learning machine discovers that action commands for multiple consecutive cycles are intercepted by the physical verification gateway, it will automatically trigger a model reconstruction process. By increasing the number of random exploration steps, it will re-find the optimal control strategy that conforms to the current physical environment, thereby enhancing the system's adaptive resilience to sudden operating condition drift.

[0028] Furthermore, the sampling frequency of the multi-dimensional sensing front end and the control frequency of the closed-loop execution and monitoring feedback module maintain a strict 10:1 synchronization ratio. That is, for every 10 high-frequency samplings, the system completes one complete feature fusion and reinforcement learning inference, ensuring that the control commands still have extremely high real-time performance under complex semiconductor process switching transients.

[0029] As one embodiment of the present invention, the system is deployed on an industrial-grade embedded computing platform and uses a hardware acceleration unit to optimize the matrix multiplication operation of multi-head attention, so that the end-to-end latency from data acquisition to instruction issuance is controlled within 5 milliseconds, which meets the requirements of semiconductor manufacturing for nanosecond to millisecond level control accuracy.

[0030] Furthermore, the normalization parameters in the heterogeneous data preprocessing module are dynamically updated based on the statistical distribution of the entire process cycle. They can automatically adjust the feature scaling ratio according to the switching of different process formulas, thereby ensuring that the multi-head attention feature fusion engine always works in the optimal activation function sensitive region.

[0031] As one embodiment of the present invention, the physical constraint-guided single Q-table reinforcement learning machine also calculates the entropy value of the current state when performing action updates. When the entropy value is lower than a preset entropy threshold determined by historical process data statistics, it indicates that the system is in a high-confidence proficiency state, and the system will reduce the exploration step size. When the entropy value is higher than the preset entropy threshold determined by historical process data statistics, it indicates that the environment has undergone unknown changes, and the system will automatically increase the exploration range to find new optimal actions.

[0032] Furthermore, the closed-loop execution and monitoring feedback module includes a safety interlock subsystem, which is implemented entirely based on hardware logic. When the analysis system at the software level crashes or becomes logically disordered, it can force the valve to return to a safe preset position, providing a final layer of physical protection for high-value semiconductor manufacturing equipment.

[0033] As one embodiment of the present invention, the single Q-table reinforcement learning machine, when handling multi-task process switching, stores multiple independent Q-table slices to deal with different gas components or process pressure segments respectively. After receiving the process switching command from the host computer, the system can complete the seamless switching of Q-table slices within one control cycle, ensuring process continuity.

[0034] Furthermore, the multi-head attention feature fusion engine also introduces a position encoding mechanism when calculating attention weights, embedding the time series information of parameter sampling into the vector, enabling the model to distinguish physical features appearing at different time points, thereby effectively identifying pressure pulse signals with causal relationships.

[0035] As one embodiment of the present invention, the system introduces the law of fluid conservation at the physical constraint layer, automatically eliminating non-physical action commands that would lead to non-conservation of mass or sudden changes in energy, thus fundamentally eliminating potential production safety hazards caused by excessive model exploration.

[0036] Furthermore, the single Q-table reinforcement learning machine introduces an experience replay mechanism to store representative successful control cases and key failure cases in a priority replay buffer, and increases the sampling probability of these samples during training, thereby greatly improving the system's learning speed and steady-state accuracy without increasing the consumption of actual production samples.

[0037] As one embodiment of the present invention, the closed-loop execution and monitoring feedback module can output a valve health status score in real time; the score is calculated comprehensively based on the convergence trend of the Q value, the frequency of action, and the physical constraint triggering rate, providing data support for the preventive maintenance of semiconductor equipment.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. This invention introduces a multi-head attention mechanism into semiconductor valve analysis, enabling in-depth correlation mining of multi-dimensional heterogeneous process features. The system can automatically identify the nonlinear dependencies of parameters such as pressure, flow rate, and opening degree at different process stages, overcoming the shortcomings of traditional analysis methods in processing high-dimensional features, such as severe information loss and inaccurate capture of key signals, and significantly improving the feature support strength and accuracy of decision-making.

[0040] 2. This invention fundamentally solves the problem that pure data-driven algorithms are prone to generating non-physical and law-violating decision instructions by forcibly embedding a physical constraint guidance layer into the reinforcement learning loop, transforming physical mechanisms such as fluid dynamics and classical mechanics into mathematical verification logic. This improvement ensures that every control action output by the system conforms to the actual physical environment of semiconductor manufacturing, greatly reducing the risk of scrapping high-value wafers or damaging precision equipment due to model misoperation, and guaranteeing the absolute safety of process production.

[0041] 3. This invention employs a single Q-table structure combined with a discretized state space design, which significantly reduces the computational complexity and memory footprint of the reinforcement learning algorithm while ensuring control accuracy. Combined with action space pruning techniques under physical constraints, the system no longer engages in blind, disordered exploration but instead focuses on learning from high-value regions that conform to physical logic. This not only increases the system's convergence speed by more than two times but also greatly improves sample utilization efficiency, enabling the system to adapt to new process requirements within a very short setup time.

[0042] 4. This invention constructs a fully closed-loop architecture from perception, fusion, decision-making to execution feedback, and integrates anomaly diagnosis and safety interlocking subsystems; by monitoring execution residuals in real time and dynamically updating the Q-table value matrix, the system has powerful self-evolution and self-repair capabilities, which can effectively compensate for mechanical wear of the actuator, sensor aging and complex environmental disturbances, ensuring that the semiconductor valve maintains nanometer-level fluid control stability during continuous operation for tens of thousands of hours. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall technical solution architecture proposed in this invention;

[0044] Figure 2 This is a schematic diagram of the core principle framework of the physical constraint-guided single Q-table reinforcement learning machine in this invention;

[0045] Figure 3 This is a schematic diagram of the core principle framework of the multi-head attention feature fusion engine in this invention;

[0046] Figure 4 This is a flowchart illustrating the main stages of heterogeneous data preprocessing and initial state vector construction in this invention.

[0047] Figure 5 This is a schematic diagram of the multi-level interaction relationship and data flow between the closed-loop execution mechanism and the reinforcement learning decision module in this invention. Detailed Implementation

[0048] Example 1: Please refer to the appendix Figure 1 This embodiment discloses a semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention. This system is primarily deployed in precision fluid control environments within semiconductor manufacturing processes, aiming to achieve ultra-high precision control of semiconductor chamber pressure through deep integration of artificial intelligence algorithms and physical mechanism models. The system's underlying architecture consists of a perception layer, a processing layer, a feature fusion layer, a decision learning layer, and an execution monitoring layer. These layers interact deterministically via a high-speed industrial bus.

[0049] Please refer to the appendix. Figure 1 The system's starting point is a multi-dimensional sensing front-end. This front-end is not a single sensor, but a heterogeneous acquisition array designed for the complex operating conditions of semiconductors. The pressure sensor array is distributed at the upstream inlet and downstream chamber connection of the semiconductor valve. These pressure sensors employ a high-sensitivity single-crystal silicon piezoresistive structure, capable of capturing minute pressure fluctuations at the 0.1 Pascal level. The upstream pressure sensor monitors the static pressure and dynamic disturbances of the gas supply system, while the downstream pressure sensor directly feeds back the actual vacuum or pressure level within the process chamber, calculating the instantaneous pressure gradient from the difference between the two. The mass flow control and monitoring unit works in conjunction with the pressure sensor array, acquiring the molar flow rate data of the gas through calorimetry or a Coriolis mass flow meter. In semiconductor processes, gases are diverse and have vastly different physical properties; the mass flow control and monitoring unit can convert the mass flow rates of different gases into a unified electrical signal output. High-precision displacement sensors are installed inside the valve's drive actuator, typically employing laser interferometry or high-resolution inductive displacement monitoring technology. They monitor the absolute opening position of the valve core relative to the valve seat in real time, achieving a resolution at the 10-nanometer level. Considering the impact of ambient temperature fluctuations in semiconductor processes on sensor accuracy, the multi-dimensional sensing front end also integrates a temperature compensation unit. This temperature compensation unit utilizes an array of thermistors distributed on the valve housing and sensor base to acquire the ambient temperature distribution map in real time, and applies a polynomial fitting compensation algorithm to dynamically correct zero-point drift and sensitivity drift of pressure and flow data, ensuring that the data output by the sensing layer has high linearity and repeatability within a wide temperature range of 0 degrees Celsius to 150 degrees Celsius.

[0050] Please refer to the attached document. Figure 4The heterogeneous data preprocessing module receives the raw analog signal from the multidimensional sensing front end. This module first samples the signal at a high frequency of 1000 Hz using a high-performance analog-to-digital converter. This sampling frequency is chosen based on in-depth analysis of the semiconductor valve's operating frequency and the fluid's characteristic frequency, ensuring the complete preservation of all harmonic components of the signal. The sampled digital signal then enters a 5th-order Butterworth low-pass filter. The advantage of the 5th-order Butterworth low-pass filter lies in its flattest passband characteristics, effectively removing high-frequency electromagnetic interference noise while minimizing the impact on the phase of the useful physical signal. A key step in the preprocessing process is the execution of the sliding window algorithm. The system maintains a sliding buffer with a time length of 100 milliseconds, calculating the mean, variance, and first-order rate of change within this window for each parameter dimension. The mean represents the quasi-static characteristics of the parameter, the variance reflects the system's stability, and the first-order rate of change captures the instantaneous change trend of pressure. To address the issue of numerical range discrepancies caused by different physical dimensions, the heterogeneous data preprocessing module employs a minimum-maximum normalization method to map pressure, flow rate, displacement, and temperature data to a numerical range of 0 to 1. This normalization significantly improves the convergence performance of subsequent neural network models, preventing gradient weight imbalances caused by pressure data volumes being much larger than displacement data volumes. Furthermore, this module integrates a state estimation algorithm based on Kalman filtering. In extreme cases such as sensor flashover or data packet loss, the Kalman filter uses a pre-defined fluid state transition equation and covariance matrix to predict and reconstruct the current pressure and flow rate, thereby outputting a continuous sequence of initial state vectors with uniform time-scale characteristics.

[0051] Please refer to the attached document. Figure 3The initial state vector sequence is then fed into a multi-head attention feature fusion engine. This engine is designed to uncover the extremely complex nonlinear relationships between multidimensional features in semiconductor processes. The multi-head attention feature fusion engine first projects the normalized initial state vector onto three different feature spaces through a linear mapping layer, generating a query matrix, a key matrix, and a value matrix, respectively. The core multi-head attention computation layer consists of eight independent attention heads operating in parallel. This parallel architecture allows the system to observe data relationships from multiple dimensions on the same timescale. Specifically, attention heads 1 through 3 focus on capturing long-term pressure fluctuation trends, calculating attention weights over a long time span to identify periodic pressure shifts in the process cycle; attention heads 4 through 6 focus on capturing coupling interference between different gas paths, such as the transient impact on other bypass pressures when a large flow of process gas enters; and attention heads 7 and 8 focus on capturing the instantaneous lag between valve action commands and flow responses, which is crucial for compensating for mechanical transmission backlash. Each attention head calculates its attention score using the scaled dot product of the query matrix and the key matrix, employing a scaling factor to prevent gradient saturation caused by excessively large dot products. The scores are processed by a normalized exponential function to form a weight distribution, which is then multiplied by the value matrix to obtain the fused features of the current head. The multi-head attention feature fusion engine concatenates the outputs of the eight heads and performs dimensionality reduction and linear fusion through a fully connected layer, finally introducing a residual connection layer. This residual connection layer constructs an identity mapping path by directly adding the fused deep features to the original input features, effectively addressing potential degradation issues during deep feature extraction. The global representation vector output by this engine not only encapsulates the spatial correlation of multidimensional physical parameters but also contains temporal causal logic evolving over time.

[0052] Please refer to the appendix. Figure 2 The physical constraint-guided single Q-table reinforcement learning machine serves as the system's decision-making center. This invention does not employ computationally intensive deep reinforcement learning networks, but instead utilizes a single Q-table structure optimized through discretization of the state space. The state space is finely divided into 1024 subdivided logical states, each corresponding to a specific combination of pressure difference, flow deviation, and valve opening change. The action space is defined as the step increment of the valve opening. To ensure both control accuracy and response speed, the action step increment undergoes nonlinear quantization: in the nonlinear sensitive region where the valve opening is close to fully closed or fully open, the resolution of the action step increment is increased to 0.01% to achieve extreme control of minute flow velocities; in the intermediate linear region, the resolution of the action step increment is set to 0.1% to improve the adjustment speed.

[0053] The core update process of a single-Q table reinforcement learning machine follows the Bellman equation, which can be expressed mathematically as follows: In the above algorithm, the value of the current state-action pair is updated by accumulating a weighted sum of the immediate reward and the expected future value. The learning rate is... A dynamic decay strategy was adopted. During the initial offline pre-training phase, the initial learning rate was set to 0.1 to encourage the system to quickly explore the action space. As the number of training steps increased or the Q-value gradually converged, the learning rate gradually decayed to 0.001 to ensure the system could stably reside near local optima. Discount factor The value is set at 0.95, which balances the immediate feedback of the current operation with the long-term expectation of future stress stability.

[0054] Please refer to the attached document. Figure 2 A core aspect of this invention lies in its physical constraint guidance mechanism. During the decision-making process of the single Q-table reinforcement learning machine, the system forcibly introduces a boundary check operator based on fluid dynamics equations. This operator calls a pre-stored physical mechanism model, which includes the flow rate limits of the semiconductor valve under different pressure ranges and the acceleration thresholds during opening and closing, and considers the gas compressibility factor and adiabatic index. When the reinforcement learning algorithm attempts to output a certain action command, the boundary check operator calculates the potential pressure change rate in real time. If the prediction result exceeds the safety boundary set by the physical mechanism model—for example, if the predicted action will cause a pressure shock exceeding the material's tolerance limit—the system automatically penalizes the corresponding value of that action in the Q-table. Simultaneously, the boundary check operator forcibly redirects the decision to an adjacent safe action range that conforms to fluid dynamics laws, thereby fundamentally eliminating non-physical and dangerous operational commands that might be generated by purely data-driven algorithms. To further improve learning efficiency, the system also introduces an experience replay mechanism, storing representative successful control cases and critical failure cases in a priority replay buffer, increasing the sampling probability of these high-value samples during the update process.

[0055] Please refer to the attached document. Figure 5The closed-loop execution and monitoring feedback module is responsible for implementing decision commands and forming a closed loop. This module includes a digital signal conversion unit, which converts the step commands output by the reinforcement learning machine into high-resolution pulse width modulation signals. The power drive control unit drives the servo motor or piezoelectric ceramic actuator based on this signal. During execution, the anomaly diagnosis unit assesses the system's health status in real time by comparing the residuals of actual sensor feedback with those predicted by the physical model. When the residual exceeds a preset 15% threshold, the system determines that there may be valve mechanical wear, particle accumulation, or sensor failure, and immediately triggers a safety shutdown protection. Simultaneously, the closed-loop execution and monitoring feedback module establishes a feedback buffer with a capacity of 5000 control cycles to record all state-action pairs and their corresponding actual reward values. This data is periodically used to recalibrate the value distribution of the Q-table, enabling the system's self-evolution. Furthermore, the system includes a safety interlocking subsystem implemented entirely based on hardware logic, serving as a final physical protection barrier.

[0056] In real-world semiconductor manufacturing environments, this system demonstrated exceptional adaptability. During the deposition process, the multi-head attention feature fusion engine automatically identified the dominant weight of pressure sensor data by statistically analyzing the spatial distribution of attention, and increased its processing priority accordingly. During the etching process, the system automatically shifted to increasing the weight of flow control data. This dynamic weight adjustment mechanism, combined with the rapid switching capability of the single-Q-table reinforcement learning machine, enabled the system to seamlessly switch Q-table slices for different gas compositions or pressure ranges within one control cycle after receiving a process switching command, ensuring the continuity and stability of the semiconductor manufacturing process. The entire system is deployed on an industrial-grade embedded computing platform, utilizing hardware acceleration units to optimize the matrix multiplication operations of multi-head attention. This ensures that the end-to-end latency from sensor data acquisition to control command issuance is strictly controlled within 5 milliseconds, fully meeting the stringent real-time requirements of semiconductor manufacturing.

[0057] In the initial stage of system operation, the single Q-table reinforcement learning machine is in offline pre-training mode. At this time, the system uses a large amount of historical process data, including valve action response data under different temperatures, pressures, and gas flow rates, to initialize the Q-table. The goal of this stage is to establish a preliminary physical causal mapping. After entering the online incremental learning stage, the system dynamically adjusts the values ​​in the Q-table based on real-time perceived process deviations. To prevent the system from getting trapped in local optima, an entropy-based exploration strategy is adopted: when the calculated current state entropy value is lower than a preset entropy threshold determined by historical process data statistics, it means that the system has a very high degree of control over the current operating conditions, and the random exploration step size is reduced; conversely, when the entropy value is higher than the preset entropy threshold determined by historical process data statistics, it indicates that the environment may have undergone unknown changes due to equipment aging or process fine-tuning, and the system automatically increases the exploration range to find new and better control trajectories.

[0058] Please refer to the attached document. Figure 5 The closed-loop execution and monitoring feedback module also integrates a feedforward compensation path. The global representation vector output by the multi-head attention feature fusion engine is not only used to guide Q-table decisions but is also fed into a dedicated prediction branch. This branch predicts the chamber pressure trend within the next 50 milliseconds by modeling historical sequences. This predicted value, as a feedforward compensation term, is directly connected to the regulation loop of the power drive control unit. This composite control structure of feedforward and feedback gives the system a strong ability to resist sudden pipeline disturbances, quickly suppressing pressure fluctuations to within 0.05% of the set value. Simultaneously, the closed-loop execution and monitoring feedback module also calculates and outputs a valve health status score in real time. This score integrates the convergence speed of the Q value, the frequency of action commands triggering physical constraints, and the response lag time of the actuator, providing a quantitative scientific basis for preventative maintenance in the plant and effectively avoiding production losses caused by unplanned downtime.

[0059] Example 2: Building upon Example 1, this example further enhances the performance and optimizes the structure of the physically constrained single-Q-table reinforcement learning machine, specifically addressing the unique high-pressure, high-flow-rate conditions in semiconductor manufacturing. In certain extreme chemical vapor deposition processes, gas flow rates may approach the speed of sound, significantly increasing the nonlinearity and compressibility of the fluid, making it difficult for conventional physical models to accurately describe.

[0060] Combined with appendix Figure 2In this embodiment, the physical constraint guidance layer introduces a more complex set of fluid dynamic coupling equations. The physical mechanism model not only considers the basic pressure-velocity relationship but also integrates the actual gas state equations for special gases. During action verification, the boundary verification operator calculates the current Reynolds number and Mach number in real time. When it is detected that a step increment of the valve opening may cause the local flow field to enter the transonic region and induce shock wave oscillations, the system immediately activates a nonlinear penalty mechanism. Unlike the fixed penalty in Embodiment 1, this embodiment dynamically calculates the reduction magnitude of the Q value based on the predicted shock wave intensity. The reduction magnitude is positively correlated with the energy dissipation caused by the shock wave, thereby guiding the reinforcement learning machine to avoid actions that, although capable of quickly adjusting pressure, may cause flow field instability.

[0061] In the data preprocessing stage, this embodiment increases the sampling frequency to 2000 Hz for high-frequency pulsations under high-pressure conditions. The sliding window algorithm in the heterogeneous data preprocessing module has also been adjusted accordingly, introducing multi-scale window analysis. The system simultaneously maintains three observation windows of different lengths: 10 milliseconds, 50 milliseconds, and 100 milliseconds. The 10-millisecond window is used to capture high-frequency pressure pulses, the 50-millisecond window is used to identify flow regime transition characteristics, and the 100-millisecond window maintains monitoring of quasi-static trends. Please refer to the appendix. Figure 4 After being normalized, the features at these three scales are input in parallel into the multi-head attention feature fusion engine, so that the initial state vector sequence contains rich frequency domain and time domain features coupled together.

[0062] Please refer to the attached document. Figure 3 The multi-head attention feature fusion engine employs an enhanced positional encoding mechanism when processing these multi-scale features. In addition to temporal encoding, physical positional encoding is introduced, embedding the geometrical positional information of different sensors within the gas path into the vector. This enables the eight attention heads to more accurately identify the propagation delay and reflection effects of pressure waves in the pipeline. For example, when a pressure fluctuation occurs upstream, the fourth attention head predicts the precise time the fluctuation will reach the downstream chamber based on the physical positional encoding, and guides the reinforcement learning machine to perform compensatory actions in advance.

[0063] In this embodiment, the single-Q-table reinforcement learning machine employs a multi-level cascaded Q-table structure to address the state space explosion problem under high-flow conditions. The system divides 1024 logic states into one coarse-tuned Q-table and four fine-tuned Q-table slices. The coarse-tuned Q-table is responsible for large-step state transitions during the initial process changeover, quickly locating the pressure to near the target range. Once the pressure enters within 5% deviation of the target value, the system automatically switches to the corresponding fine-tuned Q-table slice. The fine-tuned Q-table slices perform higher-resolution state division for specific pressure ranges. This hierarchical control strategy, combined with online updates of the Bellman equation, not only ensures the system's robustness under high-flow disturbances but also further improves steady-state accuracy.

[0064] Specifically, the Bellman equation update logic in this embodiment introduces an action consistency constraint. The update rules are as follows: In the above formula, a new term has been added. This represents the penalty for motion variability, where... For smoothing coefficients, This is the absolute value of the difference between the current action and the action at the previous moment. This improvement aims to suppress the high-frequency jittering phenomenon of actuators, which is common in high-flow-rate control. By adding this action consistency penalty outside the reward function, the reinforcement learning machine automatically seeks control sequences that make valve actions smoother while pursuing the pressure target. This not only extends the service life of mechanical components but also reduces fluid disturbances caused by violent valve movements.

[0065] Please refer to the attached document. Figure 5 In this embodiment, a precision voltage control unit for the piezoelectric ceramic actuator is added to the closed-loop execution and monitoring feedback module. While the piezoelectric ceramic actuator has an extremely fast response speed, it suffers from severe hysteresis. The anomaly diagnosis unit pre-corrects the displacement commands output by the reinforcement learning machine by establishing an inverse hysteresis model. Simultaneously, the anomaly diagnosis unit also monitors the waveform distortion of the drive current in real time, calculating the distortion rate to determine whether the piezoelectric ceramic has microcracks or performance degradation. If the distortion rate exceeds 15%, the system switches to a degraded operation mode, increasing the action exploration step size limit in the Q-table to ensure safety.

[0066] Furthermore, the physical constraint guidance mechanism in this embodiment also includes energy conservation verification. The system calculates the rate of change of the total internal energy of the gas entering the chamber in real time. When a certain action command is predicted to cause the rate of change of energy to violate the first law of thermodynamics or result in an unreasonable temperature jump, the physical constraint layer will forcibly block the command. This hard constraint based on underlying physical laws enables this analysis system to provide extremely high reliability even when facing extremely complex semiconductor high-voltage processes. Long-term operation in industrial fields has proven that even under severe disturbances with pressure fluctuations as high as 30%, the system can still recover to a stable control state at the nanosecond level within 200 milliseconds.

[0067] Example 3: This example focuses on describing the specific implementation of a semiconductor valve analysis system based on single Q-table reinforcement learning fused with multi-head attention in an environment with automatic switching between multiple processes and complex chemical compositions. In advanced semiconductor manufacturing processes, a complete process cycle often includes multiple consecutive steps, such as pre-cleaning, thin film deposition, purging, and etching. The types of gases used, pressure setpoints, and tolerances to fluctuations vary for each step.

[0068] Please refer to the appendix. Figure 1 To accommodate these frequent process changes, a gas component identification unit is integrated into the multi-dimensional sensing front end. This unit utilizes infrared absorption spectroscopy or mass spectrometry to acquire the proportion of the mixed gas entering the valve in real time. This component data, as an additional physical dimension, is input into the heterogeneous data preprocessing module. The preprocessing module dynamically calls different state reconstruction algorithms for different gas components. For example, for gases with higher viscosity, the state transition matrix of the Kalman filter increases the weight of the damping term to more accurately simulate the fluid's inertia.

[0069] Please refer to the attached document. Figure 3 The multi-head attention feature fusion engine features optimized head allocation specifically for multi-step features. The first and second attention heads are tasked with process step identification, determining the current process stage by analyzing flow rate jumps and composition changes in the time series. The third to eighth heads dynamically adjust their attention distribution based on the identified step results. In the thin film deposition stage, attention focuses on the minimum fluctuations in downstream pressure; while in the purging stage, attention shifts to the pipe resonance characteristics under high-flow-rate scouring. This step-aware feature fusion significantly enhances the system's adaptability to complex production environments.

[0070] Please refer to the attached document. Figure 2In this embodiment, the physical constraint-guided single Q-table reinforcement learning machine employs a multi-Q-table parallel storage and fast loading technique. The system pre-stores a set of Q-table seeds for different process formulations, generated offline based on massive amounts of simulation data. When the gas component identification unit detects a process switching command, the decision learning layer saves the current Q-table state and loads the new Q-table within 5 milliseconds. To ensure the smoothness of the switching process, a transitional learning mechanism is introduced. In the intermediate window between two step switching steps, the system simultaneously calculates the outputs of both Q-tables and performs linear interpolation fusion based on time weights to prevent abrupt changes in pressure control.

[0071] In this embodiment, the update logic of the single Q-table reinforcement learning machine further incorporates state entropy reduction technology. During each control cycle, the system evaluates the determinism of the numerical distribution in the current Q-table. If, for a given state, the Q-values ​​of multiple actions are very close, it indicates that the system has decision ambiguity in this state. At this point, the system triggers a local exploration process, quickly widening the value gap between actions by trying different actions within safe physical boundaries, thereby achieving self-purification of the decision logic.

[0072] In this embodiment, the physical constraint guidance layer incorporates chemical reaction kinetic constraints. Under certain high-temperature processes, gas flow is accompanied by changes in the number of moles due to chemical reactions, which directly affect pressure balance. The physical mechanism model dynamically predicts the contribution of the reaction to pressure by coupling the chemical reaction rate equation. When the reinforcement learning machine plans to reduce the opening degree to lower the pressure, if the physical model predicts that the exothermic reaction will lead to a compensatory increase in pressure, the boundary check operator will preemptively correct the amount of opening degree reduction, thereby achieving higher-dimensional feedforward prevention.

[0073] Please refer to the attached document. Figure 5 In Example 3, the closed-loop execution and monitoring feedback module particularly enhances the correlation analysis capabilities of the anomaly diagnosis unit. This unit no longer merely monitors the residual of a single valve, but instead acquires operating data from adjacent gas paths and vacuum pumps via a high-speed network to construct a global correlation fault model. If the pressure deviation of the system is highly correlated with the speed fluctuations of downstream pump units, the anomaly diagnosis unit feeds this information back to the reinforcement learning machine, guiding it to adjust the penalty term in the reward function and suppress system-level oscillations through collaborative control. This cross-module feedback mechanism transforms the single Q-table reinforcement learning machine from an isolated controller into an intelligent node within the entire semiconductor manufacturing fluid network.

[0074] In this embodiment, the closed-loop actuator is also equipped with a multi-level backup drive link. The two signals output by the digital signal conversion unit control the main servo mechanism and the fine-tuning piezoelectric mechanism, respectively. Under normal conditions, the total instruction output by the reinforcement learning machine is decomposed into coarse and fine levels. This structure perfectly matches the nonlinear quantization of the step size of a single Q table, enabling the system to respond quickly and without overshoot when handling the rapid switch from atmospheric pressure to ultra-high vacuum. By replaying the successful experiences of the most recent 5000 cycles, the system can continuously optimize the allocation ratio of coarse and fine control to achieve the optimal balance of control energy efficiency.

[0075] Example 4: This example details the implementation of adaptive drift compensation and preventative maintenance of the present invention during long-term operation. After prolonged exposure to corrosive gases and high-frequency mechanical reciprocating motion, semiconductor valves inevitably experience mechanical wear, spring fatigue, and decreased sensor sensitivity, resulting in what is known as parameter drift.

[0076] Please refer to the attached document. Figure 1 With appendix Figure 4 In this embodiment, the heterogeneous data preprocessing module integrates a long-term trend monitoring operator. This operator statistically models the baseline points of pressure sensors and flow meters by maintaining a long-term database covering several weeks of operation. If the output value of the pressure sensor shows a slow linear shift under the same valve opening and flow rate conditions, the preprocessing module automatically updates the baseline value in the normalization parameters, performing online compensation for sensor aging from the data source, ensuring that the data input to the multi-head attention feature fusion engine always maintains physical accuracy.

[0077] Please refer to the attached document. Figure 3 In this process, the multi-head attention feature fusion engine plays a role in feature decoupling. As valve seals wear, the valve's flow characteristic curve becomes distorted. The 7th and 8th attention heads, which focus on the lag between action commands and flow response, are acutely aware of this change. The multi-head attention mechanism decouples this feature change caused by mechanical alteration from the feature change caused by process gas path disturbance. This decoupled feature is fed into a single Q-table reinforcement learning machine, guiding the decision-making system to adapt to the worn hardware environment by fine-tuning the mapping relationship of action step amounts without changing the core control logic.

[0078] Please refer to the appendix. Figure 2In this embodiment, the physically constrained single-Q-table reinforcement learning machine introduces a virtual simulation-assisted Q-value reconstruction strategy. When the anomaly diagnosis unit determines that the system performance degradation exceeds a certain threshold but has not yet reached the shutdown protection standard, the system initiates a background simulation process. The simulation process uses current sensor observation data to correct the structural parameters in the physical mechanism model, such as adjusting the friction coefficient or effective flow cross-sectional area. Subsequently, the system performs accelerated evolution training on the single-Q-table in virtual space to find a new Q-value distribution best suited to the current hardware status. After training, the new Q-table is incrementally updated and merged into the real-time control flow. This approach avoids large-scale risk exploration on real equipment, greatly reducing wafer loss during the tuning process.

[0079] In the design of the reward function, Example 4 adds a device lifetime gain term. The calculation logic of the reward function becomes:

[0080]

[0081] in, Rewards for achieving the goal The newly introduced physical constraint is to punish violations. This represents the potential loss to the valve's lifespan due to this action, and it is assessed by calculating the integral of the valve core acceleration. (Coefficient) This improvement balances process precision with equipment lifespan. Through this enhancement, reinforcement learning machines will tend to select action sequences with smoother trajectories and less mechanical impact when making decisions, thereby increasing the mean time between valve failures (MTBF) while maintaining nanosecond-level accuracy.

[0082] Please refer to the attached document. Figure 5 In this embodiment, the health status score output by the closed-loop execution and monitoring feedback module is refined into multiple dimensions of health indicators. The digital signal conversion unit monitors the pulse-modulated current feedback and analyzes the actuator's energy efficiency; the power drive control unit records the motor's heating curve; and the anomaly diagnosis unit calculates the autocorrelation coefficient of the physical residual. These indicators together constitute a high-dimensional health vector, which is sent to the manufacturing execution system via a cloud interface. Based on these in-depth analysis results, the manufacturing execution system can accurately predict maintenance needs days or even weeks before the valve actually fails and automatically schedule the optimal spare parts replacement period in the production plan.

[0083] In summary, this invention utilizes a multi-head attention feature fusion engine to deeply correlate heterogeneous process features and employs a physically constrained single Q-table reinforcement learning machine to achieve optimal decision-making under the premise of conforming to fluid physics laws. The system not only possesses extremely high control accuracy and convergence speed in the initial stage, but also achieves long-term stability and self-evolution capability in the face of complex and variable semiconductor process environments through a fully closed-loop monitoring feedback and online learning mechanism.

[0084] All embodiments run on a unified hardware platform, and the functions of each module support each other, forming an organic whole, effectively solving the long-standing technical problems of lack of physical logic and low learning efficiency in the field of semiconductor valve analysis.

Claims

1. A semiconductor valve analysis system that integrates single Q-table reinforcement learning with multi-head attention, characterized in that, include: A multi-dimensional sensing front end is used to collect fluid physical parameters, valve mechanical status parameters, and environmental interference parameters in the semiconductor process chamber in real time. The heterogeneous data preprocessing module is used for high-speed sampling of the raw signals acquired by the multidimensional sensing front end; A multi-head attention feature fusion engine generates an enhanced global representation vector based on an initial state vector sequence; The physical constraint-guided single Q-table reinforcement learning machine is used to receive the enhanced global representation vector. It stores the state action value information through 1024 pre-defined discrete logical states and a single Q-table structure, and combines the embedded physical mechanism model to restrict the legality boundary of the action space. It then uses the Bellman equation as the core update function for iterative learning to output the optimal valve analysis command. The closed-loop execution and monitoring feedback module is used to convert valve analysis commands into high-resolution pulse width modulation signals through a digital signal conversion unit and send them to the power drive control unit to drive the actuator. It also captures the system state response after execution and dynamically updates the value matrix in the physical constraint-guided single Q-table reinforcement learning machine by calculating the deviation between the actual response and the expected target.

2. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 1, characterized in that, The heterogeneous data preprocessing module uses a 5th-order Butterworth low-pass filter to remove high-frequency noise components, extracts the mean, variance, and first-order rate of change of each parameter in the first 100 milliseconds of the current time using a sliding window algorithm, maps data of different dimensions to a numerical range of 0 to 1 using the minimum-maximum normalization method, and constructs an initial state vector sequence with unified time scale characteristics by combining a state estimation algorithm based on Kalman filtering.

3. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 2, characterized in that, The multi-head attention feature fusion engine uses a linear mapping layer to project the initial state vector sequence into multiple non-interfering feature spaces to generate a query matrix, a key matrix, and a value matrix. It uses eight parallel self-attention heads to calculate the correlation weights between different physical dimensions, identifies key process features that have a decisive impact on system stability, and uses a residual connection layer to sum the fused features with the initial state vector to generate an enhanced global representation vector.

4. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 3, characterized in that, The physical constraint-guided single-Q table reinforcement learning machine calculates the pressure change rate and flow velocity deviation that may result from the action in real time through the boundary check operator during the decision-making process. When the prediction result exceeds the safety boundary set by the physical mechanism model, the value in the single-Q table is penalized and the decision is redirected to the adjacent safe action range that conforms to the physical laws.

5. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 4, characterized in that, The multi-head attention feature fusion engine also introduces a position encoding mechanism when calculating attention weights, embedding the time series information of parameter sampling into the vector, so that the model can distinguish physical features that appear at different time points and identify pressure pulse signals with causal relationships. The multi-head attention feature fusion engine identifies the most influential parameter group under a specific process step by statistically analyzing the spatial distribution of attention, and increases the weight of pressure sensor data in the deposition process stage and the weight of flow control data in the etching process stage.

6. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 5, characterized in that, The multi-head attention feature fusion engine contains eight independent attention heads that focus on different feature dimensions; The first to third attention points are focused on capturing long-term stress fluctuation trends; The fourth to sixth attention heads are focused on capturing coupling interference between different air paths; The 7th and 8th attention heads focus on capturing the instantaneous lag between valve action commands and flow response; The global representation vector output by the multi-head attention feature fusion engine is also fed into a prediction branch to predict the chamber pressure trend within the next 50 milliseconds. The prediction result of the chamber pressure trend is then used as a feedforward compensation term and connected to the closed-loop execution and monitoring feedback module.

7. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 6, characterized in that, The physical mechanism model stores the flow rate limits of the semiconductor valve under different pressure ranges and the acceleration thresholds during the opening and closing process. It also considers the compressibility factor and adiabatic index of the gas and dynamically corrects the legal boundaries of the action space based on real-time monitored temperature and pressure parameters. The physical constraint-guided single Q-table reinforcement learning machine introduces the fluid conservation law into the physical constraint layer to automatically eliminate non-physical action commands that cause mass non-conservation or energy mutation.

8. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 7, characterized in that, The value update logic of the single Q-table reinforcement learning machine is as follows: the state space is defined as the pressure difference, flow deviation, and valve opening change at the current moment; the action space is the step increment of the valve opening. The discount factor in the core update function is set to 0.95, and the learning rate adopts a dynamic decay strategy. The initial learning rate is 0.1, and it gradually decays to 0.001 as the number of training steps increases. The reward function includes a goal achievement reward and a physical violation penalty. If the system reaches the preset pressure stability point, a positive reward is given; if a physical constraint is triggered, a negative reward is given.

9. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 8, characterized in that, The action step size in the single Q-table reinforcement learning machine undergoes nonlinear quantization processing; in the nonlinear sensitivity region where the valve opening is close to fully closed or fully open, the resolution of the action step size is set to 0.01%; in the intermediate linear region, the resolution of the action step size is set to 0.1%; when the physical constraint-guided single Q-table reinforcement learning machine detects that action commands for multiple consecutive cycles are intercepted by the physical verification gateway, it triggers a model reconstruction process, and re-finds the optimal control strategy by increasing the number of random exploration steps.

10. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 9, characterized in that, The single Q-table reinforcement learning machine includes an offline pre-training stage and an online incremental learning stage. The offline pre-training stage initializes the single Q-table using historical process data. The online incremental learning stage fine-tunes the single Q-table based on real-time feedback during production. When performing action updates, the single Q-table reinforcement learning machine also calculates the entropy value of the current state, and reduces the exploration step size when the entropy value is lower than a preset entropy threshold determined by historical process data, and increases the exploration range when the entropy value is higher than the preset entropy threshold determined by historical process data.

Images