Constant temperature zone temperature compensation method of horizontal liquid phase epitaxial growth equipment based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based constant-temperature region temperature compensation method for horizontal liquid-phase epitaxial growth equipment. The method comprises the following steps: S1, acquiring temperature data of a plurality of measurement points in a constant-temperature region of the horizontal liquid-phase epitaxial growth equipment; s2, obtaining a maximum value, a minimum value, an average value and a standard deviation in the temperature data; comparing each temperature data with an initial set temperature to obtain a deviation value; s3, inputting the maximum value, the minimum value, the average value, the standard deviation and the deviation value in the temperature data as input quantities into a pre-constructed and trained reinforcement learning training strategy model to obtain temperature compensation values of the temperature zones; wherein a mapping relation between the input quantity and the temperature compensation value is preset in the reinforcement learning training strategy model. The method has the advantages of simple operation, rapid and accurate temperature compensation and the like.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of horizontal liquid phase epitaxial growth equipment, and in particular to a temperature compensation method for a constant temperature zone of a horizontal liquid phase epitaxial growth equipment based on reinforcement learning. Background Art

[0002] The horizontal liquid phase epitaxy process has the advantages of easy composition change, strong thickness control ability and good repeatability. It is an effective and mature method for the development and production of HgCdTe thin film materials. Horizontal liquid phase epitaxy equipment is the mainstream equipment for realizing this process method in China. Its temperature control accuracy, heating and cooling rate, and uniformity of temperature field directly affect the quality of epitaxial materials, uniformity of epitaxial wafers, repeatability, size, and equipment production capacity, and are key performance indicators of the equipment.

[0003] The working principle of the horizontal liquid phase epitaxial growth equipment, the liquid phase epitaxial growth system adopts Te-rich flux and horizontal boat pulling method. The graphite boat carrying the substrate and solvent is transported to the cantilever quartz rake fixed on the end cover of the reaction tube through the load lock and atmosphere protection glove box, and is pushed into the reaction chamber by the advance and retreat boat mechanism, and the end cover is closed. In the reaction chamber, it is first evacuated by the vacuum pump to create a background clean space, and then hydrogen is introduced through the air intake pipeline, and a clean laminar reduction atmosphere is established in coordination with the exhaust pipeline. The heat field is established through the electric heating furnace and reaches a stable state. After the graphite boat is heated thoroughly and the solvent is fully converted into a solution, it begins to cool rapidly. When the temperature drops to a slightly higher freezing point of the solution, the boat pulling mechanism drives the pull rod to move, thereby driving the pull rod and the pull rod. The slider connected to the rod moves, and the slider smoothly covers the surface of the substrate with the solvent solution, and then slowly cools down. The solution is in a critical state of solidification and evaporation, and crystallization and epitaxy are carried out. When the temperature drops to a certain temperature, the epitaxy is completed. The boat pulling mechanism drives the pull rod to move for the second time, thereby driving the slider connected to the pull rod to move. The slider smoothly leaves the surface of the substrate with the solvent solution, and then moves the electric heating furnace away from the reaction tube. The fan is started, and the air cooling is forced to cool down until it is close to room temperature, hydrogen is replaced, and the boat is withdrawn. Finally, the graphite boat is manually unloaded from the cantilever quartz rake to the load lock, and finally the graphite boat is taken out of the load lock. In the above process, from the graphite boat being pushed into the reaction tube in the above epitaxial process, the temperature control in the reaction tube runs through the entire process, and the temperature control accuracy directly determines the quality rate of the process product; at present, in order to ensure the temperature control accuracy and the temperature uniformity of the temperature field, the furnace body adopts 5 sections of independent heating constant temperature zones and independent temperature control methods.

[0004] In order to ensure the uniformity of temperature distribution in the constant temperature zone of the horizontal liquid phase epitaxial equipment, with the increase of the number of processes, the maintenance of quartz parts, internal and external thermocouple sensors, and the replacement of graphite boat tooling, it is necessary to pull the constant temperature zone manually to ensure the temperature uniformity of the constant temperature zone, and then optimize the set temperature compensation value of the five temperature zones; the operating principle of pulling the constant temperature zone compensation is as follows: adopt the temperature measuring thermocouple moving method and single-end full-course measurement; use a temperature measuring thermocouple to extend to the position flush with the top of a single internal thermocouple sensor (five temperature measuring points) (the temperature measuring thermocouple and the internal thermocouple are aligned with each other) The thermocouple sensor is in the same thermocouple sleeve). After the temperature in the constant temperature zone is stable, the working end of the temperature measuring thermocouple is moved along the axis of the reaction tube or its parallel line from the closed end of the reaction tube 250mm away from the center of the furnace tube to the open end of the reaction tube until the end of the temperature field in the constant temperature zone. The temperature measured by the movement of the temperature measuring thermocouple is subtracted from the set temperature, that is, the temperature deviation. According to the temperature deviation trend mapped by the five temperature measuring points of the inner thermocouple, the set temperature compensation values ​​of the five temperature zones are calculated, and the temperature control accuracy of the final constant temperature zone is within (450±0.5) degrees.

[0005] At present, after the horizontal liquid phase epitaxial growth equipment has run multiple processes, more Hg will be attached to the reaction tube cavity, which will affect the heat conduction, as well as the temperature measurement sensitivity and accuracy of the thermocouple. Therefore, it will directly lead to inaccurate temperature of the epitaxial process, and ultimately affect the process quality. At this time, it is also necessary for on-site equipment personnel or process personnel to re-pull the constant temperature zone, improve the uniformity of the temperature field in the constant temperature zone, and compensate the set temperature of the five temperature zones according to the temperature obtained by the temperature measuring thermocouple. It requires certain experience and understanding of the corresponding positions of the five external thermocouple temperature measurement points and the five internal thermocouple temperature measurement points in the temperature field in order to obtain the correct compensation value more quickly. This indirectly affects the online operation time and maintenance cost of the equipment.

[0006] The existing temperature compensation method has the following shortcomings:

[0007] First, the existing temperature compensation method for the constant temperature zone requires the operator to understand the position of the corresponding temperature zone of the thermocouple outside the furnace body, as well as the position and size diagram of the five temperature measurement points of the internal thermocouple sensor.

[0008] Second, the existing temperature compensation method for the constant temperature zone requires at least 5 times to control the temperature uniformity of the constant temperature zone within the technical indicators.

[0009] Third, the existing temperature compensation method for the constant temperature zone is complicated to operate and takes a long time to maintain and adjust, which is not conducive to improving the production efficiency of the equipment.

[0010] Fourth, the existing temperature compensation method for the constant temperature zone can only be applied to horizontal liquid phase epitaxial growth equipment.

[0011] Therefore, a constant temperature zone temperature compensation method is needed that can reduce the downtime of horizontal liquid phase epitaxial growth equipment. Summary of the invention

[0012] In view of the technical problems existing in the prior art, the present invention provides a temperature compensation method for a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning, which has the advantages of simple operation and fast compensation.

[0013] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0014] A method for temperature compensation in a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning comprises the following steps:

[0015] S1, obtaining temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device;

[0016] S2, obtaining the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, comparing each temperature data with the initial set temperature to obtain the deviation value;

[0017] S3. The maximum value, minimum value, average value, standard deviation and deviation value of each temperature data are input as input into a pre-built and trained reinforcement learning training strategy model to obtain the temperature compensation value of each temperature zone; wherein the reinforcement learning training strategy model presets a mapping relationship between the input and the temperature compensation value.

[0018] Preferably, in step S3, the temperature compensation value of the middle temperature zone is first obtained, and then the temperature compensation values ​​of each temperature zone are obtained one by one toward the outside in sequence.

[0019] Preferably, the process of obtaining the temperature compensation value of the intermediate temperature zone is:

[0020] According to the deviation value between each temperature data and the reference temperature, the feedback reward value Reward is established for each position point in the intermediate temperature zone;

[0021] According to the feedback reward value Reward of each position point in the middle temperature zone, the weighted reward value of the middle temperature zone close to the adjacent temperature zone is obtained, and finally the temperature compensation value of the middle temperature zone is obtained, and then the temperature setting value of the middle temperature zone is obtained.

[0022] Preferably, in step S1, the total number of measuring points of the temperature measuring thermocouples is n, the single moving distance is l, the initial set temperature of each temperature zone is k degrees, the single point stabilization time is t, the initial temperature compensation value is 0, and a position relationship diagram of each temperature measuring point on the internal thermocouple sensor corresponding to each temperature zone of the furnace body is obtained.

[0023] Preferably, n=25; l=20 mm; k=450 degrees; t=120 s.

[0024] The present invention also discloses a constant temperature zone temperature compensation system of a horizontal liquid phase epitaxial growth device based on reinforcement learning, comprising:

[0025] A temperature acquisition module, used to acquire temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device;

[0026] The temperature analysis module is used to obtain the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, each temperature data is compared with the initial set temperature to obtain the deviation value;

[0027] The temperature compensation module is used to input the maximum value, minimum value, average value, standard deviation and deviation value of each temperature data as input into a pre-built and trained reinforcement learning training strategy model to obtain the temperature compensation value of each temperature zone; wherein the reinforcement learning training strategy model presets a mapping relationship between the input and the temperature compensation value.

[0028] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0029] The present invention also discloses a computer device, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] The constant temperature zone temperature compensation method of the horizontal liquid phase epitaxial growth equipment based on reinforcement learning of the present invention can calculate the constant temperature zone temperature compensation value in a fewer number of times, thereby reducing the equipment downtime maintenance time; this constant temperature zone temperature compensation method has strong repeatability and simple implementation, and the obtained five temperature zone set temperature compensation values ​​have significant effects, reducing the intermediate process of manual calculation, low maintenance cost, strong practicality, faster acquisition of the set temperature compensation value, and improved the efficiency of pulling the constant temperature zone. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of a temperature compensation method according to an embodiment of the present invention.

[0033] Figure 2 This is a position correspondence diagram of the five external thermocouple sensors of the furnace body and the five temperature measuring points on the internal thermocouple sensor in the present invention.

[0034] Figure 3 It is a schematic diagram of the constant temperature zone of the horizontal liquid phase epitaxial growth equipment in the present invention.

[0035] Figure 4Schematic diagram of the reinforcement learning training process in the present invention.

[0036] Figure 5 The figure is a comparison diagram of the constant temperature zone before and after using the reinforcement learning training strategy in the present invention; (a) is before improvement; (b) is after improvement. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0038] like Figure 1 As shown, the constant temperature zone temperature compensation method of a horizontal liquid phase epitaxial growth device based on reinforcement learning in an embodiment of the present invention comprises the following steps:

[0039] S1, obtaining temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device;

[0040] S2, obtaining the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, comparing each temperature data with the initial set temperature to obtain the deviation value;

[0041] S3. The maximum value, minimum value, average value, standard deviation and deviation value of each temperature data are input as input into a pre-built and trained reinforcement learning training strategy model to obtain the temperature compensation value of each temperature zone; wherein the reinforcement learning training strategy model presets a mapping relationship between the input and the temperature compensation value.

[0042] The constant temperature zone temperature compensation method of the horizontal liquid phase epitaxial growth equipment based on reinforcement learning of the present invention can calculate the constant temperature zone temperature compensation value in a fewer number of times, thereby reducing the equipment downtime maintenance time; this constant temperature zone temperature compensation method has strong repeatability and simple implementation, and the obtained five temperature zone set temperature compensation values ​​have significant effects, reducing the intermediate process of manual calculation, low maintenance cost, strong practicality, faster acquisition of the set temperature compensation value, and improved the efficiency of pulling the constant temperature zone.

[0043] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] like Figure 1As shown, the constant temperature zone temperature compensation method of a horizontal liquid phase epitaxial growth device based on reinforcement learning provided by an embodiment of the present invention can be used for a semiconductor furnace tube device with five constant temperature zones. There are five temperature zones on the furnace body of the horizontal liquid phase epitaxial growth device, and they are adjacent to each other. Each temperature zone has a DC power supply to heat the furnace wire in the corresponding temperature zone in the furnace body to form a temperature field. The external temperature is a temperature value measured by an external thermocouple obtained by heating the furnace wire through a DC power supply, and the internal temperature is a temperature value measured by an internal thermocouple obtained by heat radiation from the furnace wire into the process chamber. It is divided into five sections of independent temperature control, and the length of the uniform temperature of the five temperature zones is the constant temperature zone length, and the constant temperature zone length is 500mm.

[0045] The specific steps of the above constant temperature zone temperature compensation method are as follows:

[0046] Step 1: Preset the total measuring points of the thermocouple to 25, the single moving distance to 20mm, the initial setting temperature of the five temperature zones to 450 degrees, the single point stabilization time to 120s, and the initial temperature compensation value to 0;

[0047] Step 2: Obtain the position relationship diagram of the five temperature measurement points on the internal thermocouple sensor corresponding to the five temperature zones of the furnace body, such as Figure 2-Figure 3 As shown;

[0048] Step 3: After the constant temperature zone is automatically drawn, the maximum and minimum values ​​of 25 temperature measurement data of the temperature measuring thermocouple within the constant temperature zone are obtained, and the average value and standard deviation of this set of temperature measurement data are calculated;

[0049] Step 4: According to the technical indicators of this horizontal liquid phase epitaxial growth equipment, the upper and lower limits of the measurement temperature are divided into (450±0.5) degrees, and then combined with the Q-Learning reinforcement learning training strategy model, when the temperature measurement point of a certain temperature zone exceeds the upper limit value (450+0.5) degrees, the set temperature of the corresponding temperature zone needs to be adjusted downward; when the temperature measurement point of a certain temperature zone is lower than the lower limit value (450-0.5) degrees, the set temperature of the corresponding temperature zone needs to be adjusted upward, and the compensation values ​​of the set temperatures of the five temperature zones are obtained in turn.

[0050] Take the first constant temperature zone temperature measurement data (the set temperature of the five temperature zones is 450 degrees, the temperature compensation value is 0, and the index reference temperature is 450 degrees) as an example:

[0051] Table 1 Temperature measurement data of automatic pulling constant temperature zone

[0052]

[0053] Since the core area of ​​the process is in temperature zone 3, we first analyze the changes in the measured temperature curves from temperature zone 3 to temperature zone 2 and temperature zone 4 as the single-point temperature measurement distance changes. The temperature coupling boundary position between temperature zone 3 and temperature zone 4 is the temperature measurement position 80, and the temperature coupling boundary position between temperature zone 3 and temperature zone 2 is the temperature measurement position -80. According to the deviation value between the first temperature measurement data and the reference temperature of 450 degrees, the positive feedback or negative feedback reward value Reward is established for each position point of temperature zone 2, temperature zone 3 and temperature zone 4 respectively.

[0054] Table 2 Feedback reward value weighted to obtain temperature compensation value for temperature zone 3

[0055]

[0056] Influencing factor: The current temperature measurement point is related to the step length of a certain temperature zone, and the influencing factor of the current temperature measurement point is inversely proportional to the distance from the current temperature measurement point to the temperature measurement point in a certain temperature zone. However, in the distribution of the constant temperature zones of the horizontal liquid phase epitaxial growth equipment, temperature zone 1 is located at the furnace mouth, and temperature zone 5 is located at the furnace tail. The heat dissipation effect is large, and the temperature decreases from the furnace tail to the furnace mouth. Therefore, the discount factor of the temperature measurement point position far away from temperature zone 2 and temperature zone 4 decreases with the distance, but the discount factor of the temperature measurement point position far away from temperature zone 4 towards the furnace tail is significantly smaller.

[0057] Feedback reward value: When the deviation value is higher than the reference temperature setting value, the value of the deviation value * the impact factor is negative; when the deviation value is lower than the reference temperature setting value, the value of the deviation value * the impact factor is positive.

[0058] According to the comparison between the weighted reward value of temperature zone 3 close to temperature zone 4 and the weighted reward value close to temperature zone 2, the temperature compensation value of temperature zone 3 is -2.5, and the temperature setting value of temperature zone 3 is 447.5 degrees.

[0059] Then analyze the temperature coupling change of temperature zone 4 affected by temperature zone 5 and temperature zone 3 with the single point movement position, where the temperature coupling boundary position of temperature zone 4 and temperature zone 3 is the temperature measurement position 80, and the temperature coupling boundary position of temperature zone 4 and temperature zone 5 is the temperature measurement position 220. According to the deviation value between the first temperature measurement data and the reference temperature of 450 degrees, the positive feedback or negative feedback reward value Reward is established for each position point defined in temperature zone 5, temperature zone 3 and temperature zone 4 respectively.

[0060] Table 3 Feedback reward value weighted to obtain temperature compensation value for temperature zone 4

[0061]

[0062] According to the comparison between the weighted reward value of temperature zone 4 close to temperature zone 5 and the weighted reward value close to temperature zone 3, the temperature compensation value of temperature zone 4 is -3.0, and the temperature setting value of temperature zone 4 is 447.0 degrees.

[0063] Then analyze the temperature coupling changes of temperature zone 2 affected by temperature zone 3 and temperature zone 1 with the single point movement position. The temperature coupling boundary position between temperature zone 2 and temperature zone 3 is the temperature measurement position -80, and the temperature coupling boundary position between temperature zone 2 and temperature zone 1 is the temperature measurement position -220. According to the deviation value between the first temperature measurement data and the reference temperature of 450 degrees, the positive feedback or negative feedback reward value Reward is defined for each position point in temperature zone 3, temperature zone 1 and temperature zone 2 respectively.

[0064] Table 4 Feedback reward value weighted to obtain temperature compensation value for temperature zone 2

[0065]

[0066] According to the comparison between the weighted reward value of temperature zone 4 close to temperature zone 5 and the weighted reward value close to temperature zone 3, the temperature compensation value of temperature zone 4 is -1.6, and the temperature setting value of temperature zone 2 is 448.4 degrees.

[0067] The temperature compensation values ​​of temperature zone 1 and temperature zone 5 can be obtained by analogy.

[0068] The constant temperature zone temperature compensation method of the horizontal liquid phase epitaxial growth equipment based on reinforcement learning does not depend on whether the operator understands the relative position dimension diagram of the external thermocouple of the furnace body and the position dimension diagram of the internal thermocouple sensor; the constant temperature zone temperature compensation value can be calculated in a relatively small number of times, thereby reducing the equipment downtime maintenance time; this constant temperature zone temperature compensation method has strong repeatability and is simple to implement, and the obtained five temperature zone setting temperature compensation values ​​have significant effects, reducing the intermediate process of manual calculation, and can be applied to semiconductor equipment containing five temperature zones.

[0069] Specifically, the processes in this embodiment include: HgCdTe epitaxial growth, high temperature annealing, high temperature oxidation, TEOS, POLY and other processes which require semiconductor process equipment in a constant temperature zone.

[0070] The process temperature of HgCdTe epitaxial growth equipment is 450 degrees, the pressure is 750Torr, and a constant flow of hydrogen is required to maintain a uniform temperature field in the constant temperature zone.

[0071] Furthermore, since temperature zone 3 is located in the middle position inside the graphite boat, and temperature zones 2 and 4 are respectively located on both sides of the graphite boat, in the constant temperature zone temperature compensation method, the temperature compensation value of temperature zone 3 is first obtained, and then the temperature compensation values ​​of temperature zones 2 and 4 are obtained in turn, and finally the temperature compensation values ​​of temperature zones 1 and 5 are obtained, and the initial temperature compensation value is iteratively updated to the temperature compensation value obtained by the trained strategy model.

[0072] As can be seen from the above, in this embodiment, the temperature corresponding to the 25 total measurement points measured by the temperature measuring thermocouple corresponds to the position of the set temperature of 450 degrees in the five temperature zones, and the deviation value from the set temperature of the first group of five temperature zones, combined with the Q-Learning reinforcement learning training strategy, calculates the compensation value of the set temperature of the five temperature zones. The calculation is simple and easy to implement, not limited to the total number of measurement points, does not require operators to participate in the temperature compensation of the constant temperature zone, and does not need to obtain a temperature compensation table for dynamic adjustment. It only needs to be implemented based on temperature data, which can save labor costs in the equipment maintenance process and improve equipment operation efficiency. Figure 5 Shown is a before-and-after comparison of the constant temperature zone temperature compensation method using reinforcement learning training policy model.

[0073] Specifically, the temperature control system includes: PLC, thermocouple, DC power supply, temperature controller, and host computer. Among them, the thermocouple is used to measure the temperature in the reaction chamber or furnace body; the PLC sends the host computer output signal to the electrical execution unit to control the heating power-on; the DC power supply provides the required current; the temperature controller is used to perform closed-loop feedback control based on the temperature of the furnace body thermocouple and the thermocouple in the reaction chamber; the host computer is used to send a control signal to the lower computer, and display the set temperature and the actual temperature in the reaction chamber, and feedback the temperature measurement data of the constant temperature zone.

[0074] In this embodiment, the reinforcement learning training strategy refers to a computational method for understanding and automatically processing goal-oriented learning and decision-making problems. For example, at any time step t, the agent observes the state S of the current environment. t , and obtain the corresponding reward value r t Based on these states and reward information, the agent decides how to act. The agent performs action a t Then get new feedback from the environment and obtain the state s of the next time step t+1 and reward r t+1 ; The reinforcement learning training strategy emphasizes that the agent learns through direct interaction with the environment, and the goal is to maximize the cumulative reward to obtain the optimal action strategy. The interaction process between the agent and the environment can be formalized as a Markov decision process (MDP). The Markov decision process is represented by a five-tuple<S,A,P,R,γ> To describe, where S is the state space set of the agent's environment, A is the action space set, P: S×A×S→[0,1] is the state transition function, and R: is the reward function, γ∈(0,1] is the discount factor. The initial state is given by the initial state distribution ρ 0 Determine. Usually, the strategy π is a probability distribution mapping from state to action: π: S→P(A=a|S). At each discrete time step t, the agent observes the current state s t ∈S, use the strategy to select action a t ∈A, the environment is represented by the transformation function P(st+1 |s t ,a t ) Transition to the next state s t+1 .

[0075] The reinforcement learning training strategy model of this embodiment is a value-based method. The behavior of the agent when interacting with the environment is generated based on a certain strategy π. The value function corresponding to the strategy is used to estimate the expected return of the agent at the current state or state action, including the following steps:

[0076] Value-based methods usually define the following action-value function to evaluate the quality of the strategy:

[0077]

[0078] Optimal value function Q * (s, a) gives the maximum expected return value among all strategies for each state-action pair, that is, it satisfies:

[0079] Q * (s,a)=maxπE[R t |s t =s,a t =a,π]

[0080] The strategy with the best value function is the optimal strategy. For a given MDP, although the optimal value function is unique, there may be multiple optimal strategies. The optimal action value function follows the Bellman equation:

[0081]

[0082] Q-Learning is one of the classic algorithms for learning action-value functions. It uses a Q-value table to record the action-value function, calculates the optimal value function, and then obtains the optimal strategy. The core iterative update function of the algorithm is as follows:

[0083]

[0084] Where α∈(0,1) is the learning rate, which is used to weigh the influence of recently acquired knowledge and previously known knowledge. The optimal policy for each state selects the action that maximizes the Q value.

[0085] In the Q-Learning algorithm of the reinforcement learning training strategy, the agent starts from any initial state and enters a new state by taking actions until it reaches the target state. Each time the agent completes the process of reaching the target state from the initial state is called an episode. In each episode, the agent selects an action in the current state and applies it to the environment to transfer to a new state. This process will continue until the agent reaches the target state or reaches the maximum step limit. From the above, the Q-Learning algorithm can be divided into the following steps:

[0086] Step 1: Given the parameter γ and reward matrix R

[0087] Step 2: Set Q = 0

[0088] Step 3: For each episode:

[0089] 3.1 Randomly select an initial state s

[0090] 3.2 If the optimal target state is not reached, continue to perform the following steps

[0091] Select an action a from all possible behaviors in the current state s;

[0092] Using the selected behavior a, get the next state s′;

[0093] According to the formula

[0094] Let s=s′.

[0095] According to the above steps, the Q-Learning algorithm can be used to achieve automatic optimization and find the optimal solution to achieve the required objective function.

[0096] Since the optimization goal is to obtain the minimum TempBias value, that is, the smaller the TempBias, the larger the value function Reward. The present invention uses the following formula to calculate Reward. The final result can be calculated according to the number of states and actions. All actions that the behavior agent can perform in Q-learning are listed as all states.

[0097] Reward formula is: Reward = 100000 / TempBias; Table 2 Q-Learning state / action value table

[0098]

[0099] Specifically, in the Q-Learning reinforcement learning training strategy model, such as Figure 4As shown, the initial state of the agent is S 0 , the agent will randomly choose an action (assuming it is A 1 ). After the action is completed, the temperature control system uses the changed variables to calculate the corresponding process parameters, returns to the training strategy model, substitutes the parameters into the objective function to calculate TempBias. The reward mechanism gives the agent a positive or negative feedback reward value Reward by comparing the changes in TempBias, and then enters the next iteration.

[0100] After completing the action, enter state S 1 , the agent will randomly select an action again (assuming it is A 2 ), this training strategy model gives the agent new feedback based on the reward mechanism's judgment of TempBias. The current position of the agent and the Q value of the current action can be obtained through the code.

[0101] The present invention aims at the fact that the uniformity of the temperature field in the constant temperature zone of the horizontal liquid phase epitaxial growth equipment directly affects the substrate coating effect in the epitaxial process stage. The constant temperature zone temperature compensation method has strong repeatability and is simple to implement. It only needs to set 5 input values ​​(the upper and lower limits of the temperature uniformity of the constant temperature zone, the initial temperature compensation value, the single moving distance, and the total number of measurement points) according to the existing constant temperature zone data combined with the temperature compensation method to obtain the set temperature compensation values ​​of the five temperature zones. The effect is significant and does not depend on whether the operator is experienced or not. The intermediate process of manual calculation is reduced, the maintenance cost is low, the practicability is strong, the compensation value of the set temperature is obtained more quickly, the efficiency of pulling the constant temperature zone is improved, and the maintenance time of equipment downtime is reduced.

[0102] The constant temperature zone temperature compensation method of the present invention is not limited to horizontal liquid phase epitaxial growth equipment, but can also be used for other semiconductor equipment with five temperature zones and different processes, such as diffusion, epitaxy, TEOS, and Poly process equipment.

[0103] The present invention measures the actual temperature of the constant temperature zone in equal intervals in sequence by single-point movement and single-point stabilization time. Based on the first set of data that the initial set temperature of the five temperature zones is 450 degrees and the initial compensation value is 0, and the reinforcement learning training strategy model, the set temperature compensation values ​​of the five temperature zones are accurately obtained. This constant temperature zone temperature compensation method does not depend on the operator's familiarity with the equipment, saving the manpower cost consumed by temperature compensation and the downtime cost caused by equipment maintenance. The core measurement area of ​​this constant temperature zone is inside the graphite boat, the position closest to the epitaxial growth, and can sensitively monitor the temperature changes in this process.

[0104] The present invention also discloses a constant temperature zone temperature compensation system of a horizontal liquid phase epitaxial growth device based on reinforcement learning, comprising:

[0105] A temperature acquisition module, used to acquire temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device;

[0106] The temperature analysis module is used to obtain the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, each temperature data is compared with the initial set temperature to obtain the deviation value;

[0107] The temperature compensation module is used to input the maximum value, minimum value, average value, standard deviation and deviation value of each temperature data as input into a pre-built and trained reinforcement learning training strategy model to obtain the temperature compensation value of each temperature zone; wherein the reinforcement learning training strategy model presets a mapping relationship between the input and the temperature compensation value.

[0108] The present invention further discloses a computer-readable storage medium having a computer program stored thereon, and the computer program executes the steps of the above method when being executed by a processor. The present invention also discloses a computer device, including a memory and a processor connected to each other, and the memory having a computer program stored thereon, and the computer program executes the steps of the above method when being executed by the processor. The constant temperature zone temperature compensation system, medium and device of the present invention correspond to the above constant temperature zone temperature compensation method, and also have the advantages described in the above constant temperature zone temperature compensation method.

[0109] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0110] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A temperature compensation method for a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning, characterized in that: Includes steps: S1, obtaining temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device; S2, obtaining the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, comparing each temperature data with the initial set temperature to obtain the deviation value; S3, inputting the maximum value, minimum value, average value, standard deviation and deviation value of each temperature data into a pre-built and trained reinforcement learning training strategy model as input, and obtaining the temperature compensation value of each temperature zone; The reinforcement learning training strategy model presets a mapping relationship between the input quantity and the temperature compensation value.

2. The method for temperature compensation in a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning according to claim 1, characterized in that: In step S3, the temperature compensation value of the middle temperature zone is first obtained, and then the temperature compensation values ​​of each temperature zone are obtained one by one in sequence toward the outside.

3. The method for temperature compensation in a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning according to claim 2, characterized in that: The process of obtaining the temperature compensation value of the intermediate temperature zone is: According to the deviation value between each temperature data and the reference temperature, the feedback reward value Reward is established for each position point in the intermediate temperature zone; According to the feedback reward value Reward of each position point in the middle temperature zone, the weighted reward value of the middle temperature zone close to the adjacent temperature zone is obtained, and finally the temperature compensation value of the middle temperature zone is obtained, and then the temperature setting value of the middle temperature zone is obtained.

4. The constant temperature zone temperature compensation method of horizontal liquid phase epitaxial growth equipment based on reinforcement learning according to claim 1, 2 or 3, characterized in that: In step S1, the total number of measuring points of the temperature measuring thermocouple is n, the single moving distance is l, the initial setting temperature of each temperature zone is k degrees, the single point stabilization time is t, the initial temperature compensation value is 0, and the position relationship diagram of each temperature measuring point on the internal thermocouple sensor corresponding to each temperature zone of the furnace body is obtained.

5. The method for temperature compensation in a constant temperature zone of a horizontal liquid phase epitaxial growth device based on reinforcement learning according to claim 4, characterized in that: Wherein n=25; l=20mm; k=450 degrees; t=120s.

6. A constant temperature zone temperature compensation system for horizontal liquid phase epitaxial growth equipment based on reinforcement learning, characterized in that: include: A temperature acquisition module, used to acquire temperature data of multiple measurement points in a constant temperature zone of a horizontal liquid phase epitaxial growth device; The temperature analysis module is used to obtain the maximum value, minimum value, average value and standard deviation of each temperature data; at the same time, each temperature data is compared with the initial set temperature to obtain the deviation value; The temperature compensation module is used to input the maximum value, minimum value, average value, standard deviation and deviation value of each temperature data as input into the pre-built and trained reinforcement learning training strategy model to obtain the temperature compensation value of each temperature zone; The reinforcement learning training strategy model presets a mapping relationship between the input quantity and the temperature compensation value.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 5.

8. A computer device comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 5.

Images