Semiconductor valve analysis system integrating single Q table reinforcement learning and multi-head attention

Abstract

The invention relates to the technical field of artificial intelligence and semiconductor manufacturing process monitoring, and particularly discloses a semiconductor valve analysis system integrating single Q table reinforcement learning and multi-head attention. Global representation is generated through a multi-head attention feature fusion engine, a single Q table reinforcement learning machine guided by physical constraints is combined with a physical mechanism model to limit an action space so as to output an optimal instruction, and finally, a value matrix is dynamically updated through a closed-loop execution and monitoring feedback module. According to the invention, deep feature association can be mined, decision is ensured to accord with physical rules, learning efficiency and control stability are significantly improved, and semiconductor production safety is guaranteed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and semiconductor manufacturing process monitoring, and particularly relates to a single Q-table reinforcement learning semiconductor valve analysis system with multi-head attention. BACKGROUND

[0002] As a core cornerstone of modern electronic industry, the process flow of semiconductor manufacturing has almost strict requirements for the precision of environmental parameters and fluid control. In order to realize autonomous regulation and control of complex physical processes, intelligent analysis and control systems based on reinforcement learning are widely used in the field of precision manufacturing, aiming to replace traditional manual experience or fixed control logic by establishing an interactive feedback mechanism between agents and the environment, to realize real-time evaluation and parameter optimization of the running state of semiconductor production equipment.

[0003] As a key link for controlling fluid transportation, the performance of the semiconductor valve analysis system directly determines the pressure stability in the process chamber and the flow accuracy of chemical gas. Such a system usually uses a reinforcement learning framework to model the association between the action response of the valve and the system state, and through the continuous iteration of the state-action value function, it seeks the optimal control path and analysis strategy in the dynamically changing production environment to meet the stringent stability requirements of nanoscale manufacturing processes.

[0004] The existing technology mainly relies on pure data-driven reinforcement learning algorithms, which often expose the defect of lacking physical logic constraints when facing rare working conditions or nonlinear disturbances in the semiconductor manufacturing process. Due to the lack of deep representation of physical mechanisms such as fluid dynamics in the model, the system is prone to output non-physical decision instructions that violate physical laws, such as pressure instantaneous reverse jump, which is theoretically infeasible and can cause high-value equipment damage in practice. At the same time, due to the need for unordered exploration in a large state space, existing systems face serious problems of low sample efficiency, leading to long training cycles and difficulty in adapting to fast switching process requirements. In addition, traditional analysis methods have the disadvantage of weak information correlation when dealing with multi-dimensional process characteristics, and cannot effectively capture the deep dependency relationship between key parameters, resulting in that the decision accuracy and convergence speed under multiple constraint conditions cannot meet the requirements of industrial-level applications. SUMMARY

[0005] The purpose of the present application is to provide a single Q-table reinforcement learning semiconductor valve analysis system with multi-head attention to solve the technical problems of lack of physical logic, low sample efficiency, and weak correlation between multi-dimensional characteristics in existing semiconductor valve control technology, which leads to inaccurate decision-making.

[0006] The technical solution of the present application is a single Q-table reinforcement learning semiconductor valve analysis system with multi-head attention, which comprises:

[0007] A multi-dimensional perception front-end is used to synchronously collect fluid physical parameters, valve mechanical state parameters and environmental interference parameters in a semiconductor process chamber in real time.

[0008] A heterogeneous data preprocessing module is used to perform time domain feature extraction, frequency domain transformation and data normalization processing on original signals collected by the multi-dimensional perception front-end, and construct an initial state vector sequence with uniform time scale features.

[0009] A multi-head attention feature fusion engine is used to calculate the correlation weights between different physical dimensions based on the initial state vector sequence, identify key process features that have a decisive influence on system stability by using a plurality of self-attention heads in parallel, and generate an enhanced global representation vector.

[0010] A physical constraint guided single Q-table reinforcement learning machine is used to receive the enhanced global representation vector, store state-action value information through a preset single Q-table structure, and combine an embedded physical mechanism model to limit the legality boundary of the action space, so as to output an optimal valve analysis instruction through iterative learning.

[0011] A closed-loop execution and monitoring feedback module is used to convert the valve analysis instruction into an electrical control signal and send it to an execution mechanism, and synchronously capture the system state response after execution, and dynamically update the value matrix in the physical constraint guided single Q-table reinforcement learning machine by calculating the deviation between the actual response and the expected target.

[0012] Further, the multi-dimensional perception front-end is composed of a pressure sensor array, a mass flow control monitoring unit, a high-precision displacement sensor and a temperature compensation unit. The pressure sensor array is distributed upstream and downstream of the valve to capture the instantaneous pressure gradient. The mass flow control monitoring unit is used to obtain the molar flow data of the gas. The high-precision displacement sensor is used to monitor the absolute opening position of the valve core in real time. The temperature compensation unit is used to eliminate the influence of thermal drift on the accuracy of the sensor.

[0013] As an embodiment of the present application, the heterogeneous data preprocessing module performs the following steps: high-speed sampling of the collected continuous analog signals at a frequency of 1000 Hz, and applying a 5th order Butterworth low-pass filter to remove high-frequency noise components; extracting the mean value, variance and first-order change rate of each parameter within the current time and 100 milliseconds before the current time through a sliding window algorithm; and using the min-max normalization method to map data of different dimensions to a value interval of 0 to 1, so as to eliminate the influence of the magnitude difference of each parameter on the convergence speed of the model.

[0014] Further, the multi-head attention feature fusion engine comprises a linear mapping layer, a multi-head attention calculation layer and a residual connection layer; the linear mapping layer projects the initial state vector sequence into a plurality of mutually independent feature spaces to respectively generate a query matrix, a key matrix and a value matrix; the multi-head attention calculation layer is composed of 8 independent attention heads; each attention head measures the correlation strength between each physical feature by calculating the scaled dot product of the query matrix and the key matrix, and uses a normalization exponential function to weight the correlation score; finally, the output results of each head are spliced and fused; the residual connection layer is used to sum the fused features and the initial state vector to prevent the gradient vanishing problem in the deep feature extraction process.

[0015] As an embodiment of the present application, the physical constraint guided single Q-table reinforcement learning machine introduces a boundary check operator based on the fluid dynamics equation in the decision-making process; the physical mechanism model pre-stores the flow rate limit value of the semiconductor valve in different pressure intervals and the acceleration threshold value in the opening and closing process; when the reinforcement learning algorithm attempts to output a certain action instruction, the boundary check operator will calculate the pressure change rate and flow rate deviation that the action may cause in real time; if the prediction result exceeds the safety boundary set by the physical mechanism model, the system will automatically punish the Q value corresponding to the action and forcibly redirect the decision to the adjacent safe action interval that conforms to the physical law.

[0016] Further, the value update logic of the single Q-table reinforcement learning machine is as follows: the state space is defined as the pressure difference, the flow deviation degree and the valve opening change at the current time; the action space is the step increment of the valve opening; the Bellman equation is used as the core update function, wherein the discount factor is set to 0.95, the learning rate adopts a dynamic decay strategy, the initial learning rate is 0.1, and gradually decays to 0.001 as the training step number increases; the design of the reward function includes target achievement reward and physical violation penalty, if the system reaches the preset pressure stable point, a positive reward is given, and if the physical constraint limit is triggered, a negative reward is given.

[0017] As an embodiment of the present application, the closed-loop execution and monitoring feedback module comprises a digital signal conversion unit, a power drive control unit and an abnormal diagnosis unit; the digital signal conversion unit converts the valve analysis instruction into a high-resolution pulse width modulation signal; the power drive control unit drives the servo motor or piezoelectric ceramic actuator to act according to the signal; the abnormal diagnosis unit compares the residual error between the actual sensor feedback and the physical model prediction to determine whether the system has mechanical wear or sensor failure, and triggers the system safety shutdown protection when the residual error exceeds the preset 15% threshold.

[0018] Further, the multi-head attention feature fusion engine can automatically identify the most influential parameter group under a specific process step sequence by performing statistics on the spatial distribution of attention. In the deposition process phase, the system increases the weight of the pressure sensor data; in the etching process phase, the system increases the weight of the flow control data. This dynamic weight adjustment mechanism enables the system to adapt to complex semiconductor multi-step production environments.

[0019] As an embodiment of the present application, the single Q-table reinforcement learning machine adopts a discretized state space representation method, which divides continuous physical parameters into 1024 subdivided logical states to ensure efficient query and update of the single Q-table in limited memory space. To improve exploration efficiency, the system uses a random exploration probability of 10% in the initial stage and reduces the exploration probability to 1% as the Q value converges.

[0020] Further, the physical constraint guided single Q-table reinforcement learning machine also includes an offline pre-training phase and an online incremental learning phase. The offline pre-training phase initializes the Q-table using a large amount of historical process data, enabling the system to have preliminary decision-making capabilities. The online incremental learning phase fine-tunes the Q-table based on real-time feedback during actual production to compensate for the effects of equipment aging or process differences between batches.

[0021] As an embodiment of the present application, each attention head in the multi-head attention calculation layer focuses on different feature dimensions. The first to third attention heads focus on capturing long-term pressure fluctuation trends. The fourth to sixth attention heads focus on capturing coupling interference between different gas paths. The seventh and eighth attention heads focus on capturing the instantaneous lag relationship between valve action instructions and flow responses.

[0022] Further, the heterogeneous data preprocessing module also integrates a state estimation algorithm based on Kalman filtering, which is used to reconstruct the missing state when there is partial loss or high-intensity noise interference in the sensor data, ensuring that the data stream provided to the multi-head attention feature fusion engine has continuity and consistency.

[0023] As an embodiment of the present application, the physical mechanism model also considers the compressibility factor and the adiabatic index of the gas. When analyzing the behavior of high-pressure or ultra-low temperature special gas fluids, the legal boundaries of the action space can be dynamically corrected based on real-time monitoring of temperature and pressure parameters.

[0024] Further, the closed-loop execution and monitoring feedback module records the state-action pairs and corresponding reward values of the last 5000 control cycles in real time by establishing a cache area. These data are regularly used to recalibrate the hyperparameters in the physical constraint guided single Q-table reinforcement learning machine, achieving self-optimization of the system.

[0025] As an embodiment of the present application, the action step quantity in the single Q table reinforcement learning machine is subjected to nonlinear quantization processing; in the nonlinear sensitive region where the valve opening approaches full closing or full opening, the resolution of the action step quantity is improved to 0.01%; in the middle linear region, the resolution of the action step quantity is set to 0.1%, and this variable resolution design effectively balances the control accuracy and response speed.

[0026] Further, the global representation vector output by the multi-head attention feature fusion engine is not only used to guide the decision of the reinforcement learning machine, but also sent to a prediction branch for predicting the chamber pressure trend in the next 50 milliseconds. The prediction result of the chamber pressure trend is used as a feedforward compensation term to access the closed-loop execution and monitoring feedback module, further improving the system's ability to suppress sudden disturbances.

[0027] As an embodiment of the present application, when the single Q table reinforcement learning machine guided by physical constraints discovers that continuous multiple cycles of action instructions are intercepted by the physical verification gateway, a model reconstruction process is automatically triggered to find the optimal control strategy that meets the current physical environment by increasing the number of random exploration steps, thereby enhancing the system's adaptive resilience to sudden operating condition drift.

[0028] Further, the sampling frequency of the multi-dimensional perception front end and the control frequency of the closed-loop execution and monitoring feedback module maintain a strict 10:1 synchronization ratio, that is, for every 10 high-frequency samplings, the system completes one complete feature fusion and reinforcement learning inference, ensuring that the control instructions still have extremely high real-time performance under complex semiconductor process switching transients.

[0029] As an embodiment of the present application, the system is deployed on an industrial-grade embedded computing platform, and the hardware acceleration unit is used to optimize the matrix multiplication operation of multi-head attention, so that the whole-link delay from data acquisition to instruction issuance is controlled within 5 milliseconds, meeting the demand of semiconductor manufacturing for nanosecond to millisecond control accuracy.

[0030] Further, the normalization parameters in the heterogeneous data preprocessing module are dynamically updated based on the statistical distribution of the whole process cycle, which can automatically adjust the feature scaling ratio according to the switching of different process recipes, so as to ensure that the multi-head attention feature fusion engine always works in the optimal activation function sensitive region.

[0031] As an embodiment of the present application, the physical constraint guided single Q table reinforcement learning machine also calculates the entropy value of the current state when updating the action; when the entropy value is lower than the preset entropy threshold determined by the historical process data statistics, it indicates that the system is in a skilled state with high confidence, at which time the system will reduce the exploration step; when the entropy value is higher than the preset entropy threshold determined by the historical process data statistics, it indicates that the environment has changed unknownly, and the system will automatically increase the exploration range to find new optimal actions.

[0032] Further, the closed-loop execution and monitoring feedback module includes a safety interlocking subsystem, which is completely realized based on hardware logic and can forcibly restore the valve to a safe preset position when the analysis system at the software level is dead or logically confused, thereby providing the last layer of physical protection for high-value semiconductor production equipment.

[0033] As an embodiment of the present application, the single Q table reinforcement learning machine stores multiple independent Q table slices to cope with different gas components or process pressure segments when processing multi-task process switching, and after receiving the process switching instruction from the upper computer, the system can complete the seamless switching of the Q table slice within one control cycle, ensuring process continuity.

[0034] Further, the multi-head attention feature fusion engine also introduces a position encoding mechanism when calculating the attention weight, embeds the time sequence information of the parameter sampling into the vector, so that the model can distinguish physical features appearing at different time points, thereby effectively identifying pressure pulse signals with causal relationship.

[0035] As an embodiment of the present application, the system automatically eliminates non-physical action instructions that can cause mass non-conservation or energy mutation by introducing the law of conservation of fluid in the physical constraint layer, thereby eliminating production safety hazards caused by excessive exploration of the model from the bottom principle.

[0036] Further, the single Q table reinforcement learning machine stores representative successful control cases and key failure cases in a priority playback buffer by introducing an experience replay mechanism, and increases the sampling probability of these samples during the training process, thereby greatly improving the learning speed and steady-state accuracy of the system without increasing the consumption of actual production samples.

[0037] As an embodiment of the present application, the closed-loop execution and monitoring feedback module can output a valve health state score in real time; the score is calculated based on the convergence trend of the Q value, the action frequency and the physical constraint triggering rate, thereby providing data support for preventive maintenance of semiconductor equipment.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] 1. The present application realizes deep correlation mining of multi-dimensional heterogeneous process characteristics by introducing a multi-head attention mechanism into semiconductor valve analysis; the system can automatically identify the nonlinear dependence of parameters such as pressure, flow rate, and opening degree at different process stages, overcoming the information loss and inaccurate key signal capture of traditional analysis methods when dealing with high-dimensional characteristics, and significantly improving the feature support strength and accuracy of decision-making.

[0040] 2. The present application fundamentally solves the problem of non-physical and law-violating decision instructions produced by pure data-driven algorithms by forcibly embedding a physical constraint guidance layer in the reinforcement learning loop, converting physical mechanisms such as fluid dynamics and classical mechanics into numerical test logic; this improvement ensures that each control action output by the system conforms to the actual physical environment of semiconductor manufacturing, greatly reducing the risk of high-value wafer rejection or precision equipment damage due to model misoperation, and ensuring the absolute safety of process production.

[0041] 3. The present application uses a single Q table structure combined with a discretized state space design to significantly reduce the computational complexity and memory usage of the reinforcement learning algorithm while ensuring control accuracy; in combination with the action space pruning technique under physical constraints, the system no longer performs blind and unordered exploration, but focuses on targeted learning in high-value areas that conform to physical logic; this not only improves the convergence speed of the system by more than 2 times, but also greatly improves the sample utilization efficiency, enabling the system to adapt to new process demands in a very short warm-up time.

[0042] 4. The present application constructs a full closed-loop architecture from perception, fusion, decision-making to execution feedback, and integrates an abnormality diagnosis and safety interlocking subsystem; by monitoring execution residuals in real time and dynamically updating the Q table value matrix, the system has strong self-evolution and self-repairing capabilities, which can effectively compensate for mechanical wear of the execution mechanism, sensor aging, and complex environmental disturbances, ensuring that the semiconductor valve maintains nanoscale fluid control stability during continuous operation of tens of thousands of hours. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the overall technical scheme architecture diagram proposed by the present application;

[0044] Figure 2 is the core principle framework diagram of the single Q table reinforcement learning machine with physical constraint guidance in the present application;

[0045] Figure 3 is the core principle framework diagram of the multi-head attention feature fusion engine in the present application;

[0046] Figure 4 is the main stage flow framework diagram of heterogeneous data preprocessing and initial state vector construction in the present application;

[0047] Figure 5 is the multi-level interaction relationship and data flow diagram of the closed-loop actuator and reinforcement learning decision module in the application. DETAILED DESCRIPTION

[0048] Embodiment 1: please refer to the attached Figure 1 The embodiment discloses a semiconductor valve analysis system based on single Q table reinforcement learning and multi-head attention. The system is mainly deployed in a precise fluid control environment in a semiconductor manufacturing process, and aims to realize ultra-high precision regulation and control of the pressure in a semiconductor chamber through deep integration of artificial intelligence algorithms and physical mechanism models. The system is divided into a perception layer, a processing layer, a feature fusion layer, a decision learning layer, and an execution monitoring layer from the bottom layer architecture. Deterministic data interaction is performed between the layers through a high-speed industrial bus.

[0049] Please refer to the attached Figure 1 The starting end of the system is a multi-dimensional perception front end. The multi-dimensional perception front end is not a single sensor, but a heterogeneous collection array designed for complex semiconductor working conditions. Among them, the pressure sensor array is distributed at the upstream inlet of the semiconductor valve and the downstream chamber connection. These pressure sensors adopt a high-sensitivity single-crystal silicon piezoresistive structure, which can capture small pressure fluctuations of 0.1 Pascal. The upstream pressure sensor is responsible for monitoring the static pressure and dynamic disturbance of the gas supply system, and the downstream pressure sensor directly feeds back the actual vacuum degree or pressure level in the process chamber. The instantaneous pressure gradient is calculated by the difference between the two. The mass flow control monitoring unit works with the pressure sensor array. It obtains the molar flow data of the gas through the calorimetric principle or the Coriolis mass flowmeter. In semiconductor processes, there are many types of gases with large differences in physical properties. The mass flow control monitoring unit can convert the mass flow of different gases into a unified electrical signal output. High-precision displacement sensors are installed inside the valve driving actuator, usually using laser interference or high-resolution inductive displacement monitoring technology, which monitors the absolute opening position of the valve core relative to the valve seat in real time, with a resolution of 10 nanometers. Considering the influence of environmental temperature fluctuations on sensor accuracy in semiconductor processes, the multi-dimensional perception front end also integrates a temperature compensation unit. The temperature compensation unit uses a thermistor array distributed on the valve housing and sensor base to obtain the environmental temperature distribution in real time, and applies a polynomial fitting compensation algorithm to dynamically correct the zero drift and sensitivity drift of the pressure and flow data, ensuring that the data output by the perception layer has high linearity and repeatability in a wide temperature range of 0 to 150 degrees Celsius.

[0050] Please refer to the attached Figure 4, the isomeric data preprocessing module receives the original analog signals from the multi-dimensional perception front-end. The module first performs high-speed sampling on the signals at a frequency of 1000 Hz through a high-performance analog-to-digital converter. The choice of this sampling frequency is based on in-depth analysis of the operating frequency of the semiconductor valve and the characteristic frequency of the fluid, ensuring that the harmonic components of the signal can be completely retained. The sampled digital signal enters a 5th order Butterworth low-pass filter. The advantage of the 5th order Butterworth low-pass filter is that it has the flattest passband characteristics, which can effectively remove high-frequency electromagnetic interference noise while minimizing the impact on the phase of useful physical signals. The key step in the preprocessing process is the execution of the sliding window algorithm. The system maintains a sliding cache with a time length of 100 milliseconds, and for each parameter dimension, it calculates the mean, variance and first-order change rate within the window. The mean represents the quasi-static characteristics of the parameter, the variance reflects the stability of the system, and the first-order change rate captures the instantaneous mutation trend of the pressure. In order to solve the problem of numerical range difference caused by different physical dimensions, the isomeric data preprocessing module applies the min-max normalization method to map pressure, flow, displacement, temperature and other data to the numerical interval of 0 to 1. This normalization process can significantly improve the convergence performance of the subsequent neural network model, preventing the imbalance of gradient weights caused by the fact that the pressure data is much larger in magnitude than the displacement data. In addition, the module also integrates a state estimation algorithm based on Kalman filtering. In the extreme case of sensor transient flashover or data packet loss, the Kalman filter uses the pre-set fluid state transition equation and covariance matrix to predict and reconstruct the current pressure and flow, thereby outputting an initial state vector sequence with uniform time stamp characteristics and continuous.

[0051] Please refer to the attached drawings Figure 3The initial state vector sequence is then fed into a multi-head attention feature fusion engine. The engine is designed to exploit the extremely complex nonlinear correlations between multiple dimensions of semiconductor processes. The multi-head attention feature fusion engine first projects the normalized initial state vector into three different feature spaces through a linear mapping layer, generating query matrices, key matrices, and value matrices, respectively. The core multi-head attention calculation layer is composed of eight independent attention heads in parallel. This parallel architecture allows the system to observe data correlations from multiple dimensions at the same time scale. Specifically, the first to third attention heads focus on capturing long-term pressure fluctuation trends. They calculate attention weights over a longer time span to identify periodic pressure shifts in the process cycle. The fourth to sixth attention heads focus on capturing coupling interference between different gas paths, such as the transient impact on other bypass pressures when a large flow process gas is switched in. The seventh and eighth attention heads focus on capturing the instantaneous lag relationship between valve action instructions and flow responses, which is crucial for compensating for mechanical transmission gaps. Each attention head uses the scaled dot product of the query matrix and the key matrix to calculate the attention score, and a scale factor is used to prevent gradient saturation caused by excessively large dot product results. The score is processed by a normalization exponential function to form a weight distribution, and finally multiplied by the value matrix to obtain the fusion features of the current head. The multi-head attention feature fusion engine concatenates the output results of the eight heads and reduces the dimension through a fully connected layer and linear fusion, and finally introduces a residual connection layer. The residual connection layer constructs an identity mapping path by directly adding the fused deep features to the original input features, effectively solving the degradation problem that may occur during deep feature extraction. The global representation vector output by the engine not only condenses the spatial correlation of multiple physical parameters but also contains the time-evolving causal logic.

[0052] Please refer to the attached drawings Figure 2 The physical constraint-guided single Q-table reinforcement learning machine is the decision center of the system. Instead of using a deep reinforcement learning network with a huge amount of computation, the invention uses a single Q-table structure optimized through a discretized state space. The state space is finely divided into 1024 subdivided logical states, each corresponding to a specific combination of pressure difference, flow deviation, and valve opening change. The action space is defined as the step increment of the valve opening. In order to ensure control accuracy while considering response speed, the action step size has undergone nonlinear quantization processing: in the nonlinear sensitive area where the valve opening is close to full closure or full opening, the resolution of the action step size is increased to 0.01% to achieve extreme control of small flow rates; in the middle linear area, the resolution of the action step size is set to 0.1% to improve the adjustment speed.

[0053] The core update process of the single Q-table reinforcement learning machine follows the Bellman equation, which has the following mathematical expression: In the above algorithm logic, the value of the current state-action pair is updated by accumulating the weighted sum of the immediate reward and the future expected value. Among them, the learning rate A dynamic decay strategy is adopted. In the initial offline pre-training stage of system startup, the initial learning rate is set to 0.1 to promote the system to quickly explore the action space; as the training step increases or the Q value gradually converges, the learning rate gradually decays to 0.001 to ensure that the system can stably reside near the local optimal solution. The discount factor is set to 0.95, which balances the immediate feedback of the current operation and the long-term expectation of the stability of the future pressure.

[0054] Please refer to the attached Figure 2 One of the cores of the present application is the physical constraint guiding mechanism. In the decision-making process of the single Q table reinforcement learning machine, the system forcibly introduces a boundary check operator based on the fluid dynamics equation. The operator calls a pre-stored physical mechanism model, which includes the flow rate limit value of the semiconductor valve in different pressure intervals and the acceleration threshold value in the opening and closing process, and considers the compressibility factor and the adiabatic index of the gas. When the reinforcement learning algorithm tries to output a certain action instruction, the boundary check operator will calculate the pressure change rate that the action may cause in real time. If the prediction result exceeds the safety boundary set by the physical mechanism model, for example, the predicted action will cause a pressure impact exceeding the material bearing limit, the system will automatically punish and reduce the corresponding value of the action in the Q table. At the same time, the boundary check operator will forcibly redirect the decision to the adjacent safe action interval that conforms to the laws of fluid mechanics, thereby fundamentally eliminating the non-physical and dangerous operation instructions that may be produced by the pure data-driven algorithm. In order to further improve the learning efficiency, the system also introduces an experience replay mechanism, which stores representative successful control cases and key failure cases in a priority replay buffer, and increases the sampling probability of these high-value samples in the update process.

[0055] Please refer to the attached Figure 5, the closed-loop execution and monitoring feedback module is responsible for implementing the decision instructions and forming a closed loop. The module includes a digital signal conversion unit that converts the step command output by the reinforcement learning machine into a high-resolution pulse width modulation signal. The power drive control unit drives the servo motor or piezoelectric ceramic actuator according to this signal. During execution, the abnormality diagnosis unit assesses the system health in real time by comparing the actual sensor feedback with the residual error predicted by the physical model. When the residual error exceeds the preset 15% threshold, the system determines that there may be valve mechanical wear, particle accumulation or sensor failure, and immediately triggers a safety shutdown protection. At the same time, the closed-loop execution and monitoring feedback module records all state-action pairs and corresponding actual reward values by establishing a feedback buffer area with a capacity of 5000 control cycles. These data are regularly used to recalibrate the value distribution of the Q table, achieving self-evolution of the system. In addition, the system also includes a safety interlock subsystem completely implemented based on hardware logic, as the last physical protection barrier.

[0056] In the actual semiconductor production environment, the system shows excellent adaptability. In the deposition process stage, the multi-head attention feature fusion engine automatically identifies that the weight of the pressure sensor data should dominate by statistical attention distribution, and improves the corresponding processing priority; while in the etching process stage, the system will automatically switch to improve the weight of the flow control data. This dynamic weight adjustment mechanism combined with the rapid switching capability of the single Q table reinforcement learning machine enables the system to complete seamless switching of Q table slices for different gas components or pressure segments within 1 control cycle after receiving the process switching instruction, ensuring the continuity and stability of semiconductor manufacturing processes. The entire system is deployed on an industrial-grade embedded computing platform, and the hardware acceleration unit is used to optimize the matrix multiplication operation of multi-head attention, so that the whole link delay from perception data acquisition to control instruction issuance is strictly controlled within 5 milliseconds, fully meeting the stringent requirements of semiconductor manufacturing for extremely high real-time performance.

[0057] In the initial stage of system operation, the single Q-table reinforcement learning machine is in offline pre-training mode. At this time, the system initializes and assigns values to the Q-table using a large amount of historical process data, including valve action response data under different temperatures, different pressures, and different gas flow rates. The goal of this stage is to allow the system to establish a preliminary physical causal mapping. After entering the online incremental learning stage, the system dynamically adjusts the values in the Q-table based on real-time perceived process deviations. To prevent the system from falling into a local optimum, the system uses an exploration strategy based on entropy: when the calculated current state entropy is lower than the preset entropy threshold determined by historical process data statistics, it means that the system has very high control over the current working condition, and the random exploration step size is reduced; when the entropy is higher than the preset entropy threshold determined by historical process data statistics, it indicates that the environment may have undergone unknown changes due to equipment aging or process fine-tuning, and the system will automatically increase the exploration range to find new and better control trajectories.

[0058] Please refer to the attached Figure 5 The closed-loop execution and monitoring feedback module also integrates a feedforward compensation path. The global representation vector output by the multi-head attention feature fusion engine is not only used to guide Q-table decision-making, but also sent to a dedicated prediction branch. This branch predicts the chamber pressure trend within the next 50 milliseconds by modeling historical sequences. This predicted value serves as a feedforward compensation term and is directly input into the power drive control unit's regulation loop. This compound control structure of feedforward plus feedback makes the system highly resistant to sudden pipeline disturbances and can quickly suppress pressure fluctuations within 0.05% of the set value. At the same time, the closed-loop execution and monitoring feedback module also calculates and outputs a valve health status score in real time. This score takes into account the convergence speed of Q-values, the frequency of action instructions triggering physical constraints, and the response lag time of the actuator, providing a quantitative scientific basis for preventive maintenance in the factory and effectively avoiding production losses caused by unplanned downtime.

[0059] Embodiment 2: Based on Embodiment 1, for the special high-pressure and high-flow conditions in semiconductor manufacturing, the physical constraint-guided single Q-table reinforcement learning machine is further enhanced and structurally optimized. In some extreme chemical vapor deposition processes, the flow rate of the gas may approach the speed of sound, at which point the nonlinearity and compressibility of the fluid significantly increase, making it difficult for conventional physical models to accurately describe.

[0060] Combined with the attached Figure 2The physical constraint guide layer in this embodiment introduces more complex fluid dynamics coupled equations. The physical mechanism model not only considers the basic pressure and flow rate relationship, but also integrates the actual gas state equation for special gases. When performing action verification, the boundary verification operator will solve the current Reynolds number and Mach number in real time. When detecting that the step increment of the valve opening may cause the local flow field to enter the transonic speed region and trigger shock oscillation, the system will immediately start the nonlinear penalty mechanism. Unlike the fixed value penalty in embodiment 1, this embodiment dynamically calculates the reduction amplitude of Q value according to the predicted shock intensity. The reduction amplitude is positively correlated with the energy dissipation caused by the shock, so as to guide the reinforcement learning machine to avoid actions that can quickly adjust the pressure but cause flow field instability.

[0061] In the data preprocessing stage, this embodiment increases the sampling frequency to 2000 Hz for high-frequency pulsation under high-pressure working conditions. The sliding window algorithm in the heterogeneous data preprocessing module is also adjusted accordingly, and multi-scale window analysis is introduced. The system maintains three observation windows of different lengths, 10 milliseconds, 50 milliseconds and 100 milliseconds. The 10-millisecond window is used to capture high-frequency pressure pulses, the 50-millisecond window is used to identify flow regime transition characteristics, and the 100-millisecond window is used to maintain monitoring of the static trend. Please refer to the attached Figure 4 After normalization, the features of these three scales are input into the multi-head attention feature fusion engine in parallel, so that the initial state vector sequence contains rich frequency and time domain feature coupling.

[0062] Please refer to the attached Figure 3 When processing these multi-scale features, the multi-head attention feature fusion engine adopts an enhanced position encoding mechanism. In addition to time sequence encoding, physical position encoding is also introduced, which embeds the geometric position information of different sensors in the gas path into the vector. This enables the 8 attention heads to more accurately identify the propagation delay and reflection effect of pressure waves in the pipeline. For example, when pressure fluctuations occur upstream, the 4th attention head will predict the accurate time of the fluctuations reaching the downstream chamber according to the physical position encoding, and will guide the reinforcement learning machine to make compensatory actions in advance.

[0063] The single Q-table reinforcement learning machine in this embodiment adopts a multi-level Q-table cascade structure to address the state space explosion problem under large flow conditions. The system divides 1024 logical states into 1 coarse adjustment Q-table and 4 fine adjustment Q-table slices. The coarse adjustment Q-table is responsible for large step state migration at the initial stage of process switching, quickly positioning the pressure to the target range; once the pressure enters the 5% deviation range of the target value, the system will automatically switch to the corresponding fine adjustment Q-table slice. The fine adjustment Q-table slice performs higher resolution state division for specific pressure intervals. This hierarchical control strategy combined with online updating of the Bellman equation not only ensures the robustness of the system under large flow disturbances, but also further improves the steady-state accuracy.

[0064] Specifically, the Bellman equation update logic in this embodiment introduces an action consistency constraint. Its update rule is as follows: In the above formula, a new term is added. This term represents the action variability penalty, where is the smoothing coefficient, is the absolute value of the difference between the current action and the action at the last time. This improvement aims to suppress the high-frequency oscillation phenomenon of the actuator commonly seen in large flow control. By adding this action consistency penalty outside the reward function, the reinforcement learning machine will automatically find a control sequence that makes the valve action smoother while pursuing the pressure target. This not only prolongs the service life of mechanical components, but also reduces fluid disturbances caused by violent valve action.

[0065] Please refer to the attached Figure 5 The closed-loop execution and monitoring feedback module in this embodiment adds a precision voltage control unit for the piezoelectric ceramic actuator. The piezoelectric ceramic actuator has extremely fast response speed, but has serious hysteresis phenomenon. The anomaly diagnosis unit pre-corrects the displacement instructions output by the reinforcement learning machine by establishing a hysteresis inverse model. At the same time, the anomaly diagnosis unit also monitors the waveform distortion of the driving current in real time, and calculates the distortion rate to determine whether the piezoelectric ceramic has micro-cracks or performance degradation. If the distortion rate exceeds 15%, the system will switch to a degraded operation mode, limiting the action exploration step size in the Q-table to ensure safety.

[0066] In addition, the physical constraint guiding mechanism of the embodiment also contains energy conservation verification. The system calculates the total internal energy change rate of the gas entering the chamber in real time. When a certain action instruction is predicted to cause the energy change rate to violate the first law of thermodynamics or appear unreasonable temperature jump, the physical constraint layer will force the instruction to be blocked. This hard constraint based on the underlying physical law enables the analysis system to provide high reliability guarantee even in the face of extremely complex semiconductor high-pressure processes. The system has been proven through long-term operation in the industrial field that it can still recover to nanosecond-level stable control state within 200 milliseconds even under severe interference with pressure fluctuations up to 30%.

[0067] Embodiment 3: This embodiment focuses on the specific implementation of the single Q table reinforcement learning semiconductor valve analysis system with multi-head attention in the multi-process automatic switching and complex chemical component environment. In advanced semiconductor manufacturing processes, a complete process cycle often includes multiple consecutive steps, such as pre-cleaning, film deposition, purging, and etching. The gas types used in each step, pressure set points, and tolerance to fluctuations are different.

[0068] Please refer to the attached Figure 1 In order to adapt to such frequent process switching, the multi-dimensional perception front end integrates a gas component recognition unit. This unit uses infrared absorption spectroscopy or mass spectrometry analysis technology to obtain the proportion of mixed gas components entering the valve in real time. These component data are input as additional physical dimensions into the heterogeneous data preprocessing module. The preprocessing module dynamically calls different state reconstruction algorithms for different gas components. For example, for gases with high viscosity, the state transition matrix of the Kalman filter increases the weight of the damping term to more accurately simulate the inertial motion of the fluid.

[0069] Please refer to the attached Figure 3 The multi-head attention feature fusion engine optimizes the head allocation for multi-step sequence features. The first and second attention heads are assigned the task of process step recognition, which identifies the current process stage by analyzing the flow jump and component changes in the time series. The third to eighth heads dynamically adjust their attention distribution based on the identified step sequence results. In the film deposition stage, attention is focused on the minimum fluctuation of downstream pressure; while in the purging stage, attention is shifted to the pipe resonance characteristics under large flow flushing. This step sequence-aware feature fusion greatly enhances the system's adaptability to complex production environments.

[0070] Please refer to the attached Figure 2, the single Q-table reinforcement learning machine in this embodiment adopts the multi-Q-table parallel storage and fast loading technology. The system pre-stores a set of Q-table seeds for different process recipes, which are generated based on massive simulation data in an offline environment. When the gas component recognition unit detects a process switching instruction, the decision learning layer completes the saving of the current Q-table state and the loading of the new Q-table within 5 milliseconds. In order to ensure the smoothness of the switching process, the system introduces a transition state learning mechanism. In the middle window of the two step sequences, the system will simultaneously calculate the outputs of the two Q-tables and perform linear interpolation fusion according to the time weight to prevent the pressure control from appearing stepwise mutation.

[0071] The update logic of the single Q-table reinforcement learning machine in this embodiment further integrates the state entropy reduction technology. In each control period, the system evaluates the certainty of the value distribution in the current Q-table. If the Q values of multiple actions are very close for a certain state, it means that the system has decision ambiguity in this state. At this time, the system triggers the local exploration process to try different actions within the safe physical boundary, quickly widening the value gap between actions, thereby realizing self-purification of the decision logic.

[0072] The physical constraint guidance layer in this embodiment adds chemical reaction kinetics constraints. In some high-temperature processes, the flow of gas is accompanied by a change in the number of moles due to chemical reactions, which directly affects pressure balance. The physical mechanism model dynamically predicts the contribution of the reaction to the pressure by coupling the chemical reaction rate equation. When the reinforcement learning machine plans to reduce the opening to reduce the pressure, if the physical model predicts that the pressure will rise compensatorily due to the exothermic reaction, the boundary check operator will modify the opening reduction in advance, thereby achieving higher-dimensional feedforward prevention.

[0073] Please refer to the attached Figure 5 , the closed-loop execution and monitoring feedback module in embodiment 3 particularly strengthens the correlation analysis capability of the abnormal diagnosis unit. Instead of monitoring the residual error of a single valve, the unit obtains the running data of adjacent gas paths and vacuum pumps through a high-speed network, and builds a global correlation fault model. If the pressure deviation of the system is highly correlated with the speed fluctuation of the downstream pump set, the abnormal diagnosis unit will feed back this information to the reinforcement learning machine, guiding it to adjust the penalty term in the reward function, and suppressing the system-level oscillation through cooperative control. This cross-module feedback mechanism makes the single Q-table reinforcement learning machine not a standalone controller, but an intelligent node of the entire semiconductor manufacturing fluid network.

[0074] The closed-loop actuator is also equipped with a multi-stage backup drive link in this embodiment. The two signals output by the digital signal conversion unit control the main servo mechanism and the fine adjustment piezoelectric mechanism, respectively. Under normal circumstances, the total instruction output by the reinforcement learning machine is divided into coarse and fine levels. This structure perfectly matches the nonlinear quantization of the single Q-table stepping amount, making the system not only fast in response but also without overshoot when processing the rapid switching from atmospheric pressure to ultra-high vacuum. Through the playback of the successful experience of the last 5000 cycles, the system can continuously optimize the distribution ratio of coarse and fine control, achieving the optimal balance of control energy efficiency.

[0075] Embodiment 4: This embodiment details the adaptive drift compensation and preventive maintenance details of the application during long-term operation. After long-term exposure to corrosive gases and high-frequency mechanical reciprocating motion, semiconductor valves will inevitably experience mechanical wear, spring fatigue, and sensor sensitivity decline, i.e., so-called parameter drift.

[0076] Please refer to the attached Figure 1 with the attached Figure 4 The heterogeneous data preprocessing module integrates a long-term trend monitoring operator in this embodiment. This operator maintains a long-period database covering several weeks of operation time to statistically model the reference points of the pressure sensor and flow meter. If it is detected that the output value of the pressure sensor shows a slow linear shift under the same valve opening and flow rate conditions, the preprocessing module will automatically update the reference value in the normalization parameters, compensating for sensor aging online from the data source, ensuring that the data input into the multi-head attention feature fusion engine always maintains physical accuracy.

[0077] Please refer to the attached Figure 3 The multi-head attention feature fusion engine plays a role in feature decoupling in this process. As the valve seal wears out, the valve's flow characteristic curve will be distorted. The 7th and 8th attention heads targeting the action instruction and flow response lag relationship will be sensitive to this change. The multi-head attention mechanism will decouple this feature change caused by mechanical changes from the feature change caused by process gas path disturbances. This decoupled feature is sent to the single Q-table reinforcement learning machine, guiding the decision system to adapt to the hardware environment after wear without changing the core control logic by fine-tuning the mapping relationship of the action stepping amount.

[0078] Please refer to the attached Figure 2, the physical constraint guided single Q-table reinforcement learning machine introduces a virtual simulation assisted Q-value reconstruction strategy in this embodiment. When the abnormal diagnosis unit determines that the system performance has dropped beyond a certain threshold but has not yet reached the shutdown protection standard, the system starts a background simulation process. The simulation process modifies the structural parameters in the physical mechanism model using the current sensor observation data, such as adjusting the friction coefficient or the effective flow cross-sectional area. Subsequently, the system performs accelerated evolutionary training on the single Q-table in the virtual space to find the new Q-value distribution that best fits the current hardware status. After training is complete, the new Q-table is merged into the real-time control process in an incremental update manner. This approach avoids large-scale risk exploration on real devices, greatly reducing wafer loss during the tuning process.

[0079] In the design of the reward function, embodiment 4 adds a device life gain term. The calculation logic of the reward function becomes:

[0080]

[0081] wherein, is the target achievement reward, is the physical constraint violation penalty, and the newly introduced represents the potential wear and tear of the valve due to this action, which is evaluated by calculating the integral of the valve core acceleration. The coefficient is used to balance process accuracy and device life. With this improvement, the reinforcement learning machine will tend to choose action sequences that are smoother and have less mechanical impact when making decisions, thereby increasing the mean time between failures of the valve while ensuring nanosecond-level accuracy.

[0082] Please refer to the attached Figure 5 , the health status score output by the closed-loop execution and monitoring feedback module is refined into multiple dimensional health indicators in this embodiment. The digital signal conversion unit monitors the pulse-modulated current feedback to analyze the energy efficiency of the actuator; the power drive control unit records the heating curve of the motor; and the abnormal diagnosis unit calculates the autocorrelation coefficient of the physical residual. These indicators together form a high-dimensional health vector, which is sent to the manufacturing execution system through the cloud interface. Based on these in-depth analysis results, the manufacturing execution system can accurately predict maintenance needs several days or even weeks before the valve actually fails, and automatically schedule the optimal spare part replacement period in the production plan.

[0083] In summary, the present application uses the multi-head attention feature fusion engine to deeply associate and mine heterogeneous process features, and uses the physical constraint guided single Q-table reinforcement learning machine to make optimal decisions while complying with fluid physical laws. The system not only has extremely high control accuracy and convergence speed in the initial stage, but also realizes long-term stability and self-evolution ability in complex and variable semiconductor process environments through the full-closed-loop monitoring feedback and online learning mechanism.

[0084] All the embodiments run on a unified hardware platform, and the functions of the modules support each other, together forming an organic whole, effectively solving the technical problems of long-existing physical logic deficiency and low learning efficiency in the field of semiconductor valve analysis.

Claims

1. A semiconductor valve analysis system that integrates single Q-table reinforcement learning with multi-head attention, characterized in that, include: A multi-dimensional sensing front end is used to collect fluid physical parameters, valve mechanical status parameters, and environmental interference parameters in the semiconductor process chamber in real time. The heterogeneous data preprocessing module is used for high-speed sampling of the raw signals acquired by the multidimensional sensing front end; A multi-head attention feature fusion engine generates an enhanced global representation vector based on an initial state vector sequence; The physical constraint-guided single Q-table reinforcement learning machine is used to receive the enhanced global representation vector. It stores the state action value information through 1024 pre-defined discrete logical states and a single Q-table structure, and combines the embedded physical mechanism model to restrict the legality boundary of the action space. It then uses the Bellman equation as the core update function for iterative learning to output the optimal valve analysis command. The closed-loop execution and monitoring feedback module is used to convert valve analysis commands into high-resolution pulse width modulation signals through a digital signal conversion unit and send them to the power drive control unit to drive the actuator. It also captures the system state response after execution and dynamically updates the value matrix in the physical constraint-guided single Q-table reinforcement learning machine by calculating the deviation between the actual response and the expected target.

2. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 1, characterized in that, The heterogeneous data preprocessing module uses a 5th-order Butterworth low-pass filter to remove high-frequency noise components, extracts the mean, variance, and first-order rate of change of each parameter in the first 100 milliseconds of the current time using a sliding window algorithm, maps data of different dimensions to a numerical range of 0 to 1 using the minimum-maximum normalization method, and constructs an initial state vector sequence with unified time scale characteristics by combining a state estimation algorithm based on Kalman filtering.

3. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 2, characterized in that, The multi-head attention feature fusion engine uses a linear mapping layer to project the initial state vector sequence into multiple non-interfering feature spaces to generate a query matrix, a key matrix, and a value matrix. It uses eight parallel self-attention heads to calculate the correlation weights between different physical dimensions, identifies key process features that have a decisive impact on system stability, and uses a residual connection layer to sum the fused features with the initial state vector to generate an enhanced global representation vector.

4. The semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 3, characterized in that, The physical constraint-guided single-Q table reinforcement learning machine calculates the pressure change rate and flow velocity deviation that may result from the action in real time through the boundary check operator during the decision-making process. When the prediction result exceeds the safety boundary set by the physical mechanism model, the value in the single-Q table is penalized and the decision is redirected to the adjacent safe action range that conforms to the physical laws.

5. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 4, characterized in that, The multi-head attention feature fusion engine also introduces a position encoding mechanism when calculating attention weights, embedding the time series information of parameter sampling into the vector, so that the model can distinguish physical features that appear at different time points and identify pressure pulse signals with causal relationships. The multi-head attention feature fusion engine identifies the most influential parameter group under a specific process step by statistically analyzing the spatial distribution of attention, and increases the weight of pressure sensor data in the deposition process stage and the weight of flow control data in the etching process stage.

6. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 5, characterized in that, The multi-head attention feature fusion engine contains eight independent attention heads that focus on different feature dimensions; The first to third attention points are focused on capturing long-term stress fluctuation trends; The fourth to sixth attention heads are focused on capturing coupling interference between different air paths; The 7th and 8th attention heads focus on capturing the instantaneous lag between valve action commands and flow response; The global representation vector output by the multi-head attention feature fusion engine is also fed into a prediction branch to predict the chamber pressure trend within the next 50 milliseconds. The prediction result of the chamber pressure trend is then used as a feedforward compensation term and connected to the closed-loop execution and monitoring feedback module.

7. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 6, characterized in that, The physical mechanism model stores the flow rate limits of the semiconductor valve under different pressure ranges and the acceleration thresholds during the opening and closing process. It also considers the compressibility factor and adiabatic index of the gas and dynamically corrects the legal boundaries of the action space based on real-time monitored temperature and pressure parameters. The physical constraint-guided single Q-table reinforcement learning machine introduces the fluid conservation law into the physical constraint layer to automatically eliminate non-physical action commands that cause mass non-conservation or energy mutation.

8. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 7, characterized in that, The value update logic of the single Q-table reinforcement learning machine is as follows: the state space is defined as the pressure difference, flow deviation, and valve opening change at the current moment; the action space is the step increment of the valve opening. The discount factor in the core update function is set to 0.95, and the learning rate adopts a dynamic decay strategy. The initial learning rate is 0.1, and it gradually decays to 0.001 as the number of training steps increases. The reward function includes a goal achievement reward and a physical violation penalty. If the system reaches the preset pressure stability point, a positive reward is given; if a physical constraint is triggered, a negative reward is given.

9. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 8, characterized in that, The action step size in the single Q-table reinforcement learning machine undergoes nonlinear quantization processing; in the nonlinear sensitivity region where the valve opening is close to fully closed or fully open, the resolution of the action step size is set to 0.01%; in the intermediate linear region, the resolution of the action step size is set to 0.1%; when the physical constraint-guided single Q-table reinforcement learning machine detects that action commands for multiple consecutive cycles are intercepted by the physical verification gateway, it triggers a model reconstruction process, and re-finds the optimal control strategy by increasing the number of random exploration steps.

10. A semiconductor valve analysis system based on single Q-table reinforcement learning and multi-head attention as described in claim 9, characterized in that, The single Q-table reinforcement learning machine includes an offline pre-training stage and an online incremental learning stage. The offline pre-training stage initializes the single Q-table using historical process data. The online incremental learning stage fine-tunes the single Q-table based on real-time feedback during production. When performing action updates, the single Q-table reinforcement learning machine also calculates the entropy value of the current state, and reduces the exploration step size when the entropy value is lower than a preset entropy threshold determined by historical process data, and increases the exploration range when the entropy value is higher than the preset entropy threshold determined by historical process data.

Images