Aluminum nitride template process parameter optimization method and device and readable storage medium

Abstract

The invention provides an optimization method and device for technological parameters of an aluminum nitride template and a readable storage medium, and relates to the technical field of semiconductors. The method for optimizing the technological parameters of the aluminum nitride template based on deep reinforcement learning comprises the following steps: setting one or more technological parameters of the aluminum nitride template; defining a state space and an action space based on technological parameters for preparing the aluminum nitride template; constructing a neural network value function Q (S, A) = theta (S, A); at the end of each time step, based on the value of a state action pair calculated by a neural network value function Q (S, A), an epsilon-greedy strategy is adopted to select and implement actions so as to change corresponding process parameters; the reward function is used for calculating a reward value for implementing the action; and in a preset duration, when each time step is finished, actions are implemented to change process parameters so as to grow the aluminum nitride template, and the neural network theta is trained according to the value function and the reward function, so that a new neural network value function Q (S, A) is formed.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor technology, and in particular to a method, a device and a readable storage medium for optimizing process parameters of an aluminum nitride template. Background Art

[0002] Currently, aluminum nitride (AlN) templates are usually prepared using MOCVD (Metal-Organic Chemical Vapor Deposition) technology, which can achieve high-quality and high-speed thin film growth.

[0003] In the related art, when preparing aluminum nitride templates by MOCVD technology, the process parameters of the aluminum nitride templates are controlled by empirical data to optimize the growth quality of the aluminum nitride templates. However, empirical data relies on manual experience and subjective judgment, so that the setting of process parameters (such as temperature, flow rate, pressure, etc.) depends on historical experience or trial and error, resulting in low process efficiency for optimizing the process parameters of the aluminum nitride templates.

[0004] At the same time, when preparing aluminum nitride templates through MOCVD technology, preset fixed parameters are usually used for thin film preparation, lacking an effective real-time feedback mechanism. During the growth of aluminum nitride templates, it is impossible to flexibly adjust the growth process based on real-time monitoring data (such as reflectivity curves, temperature, gas flow, etc.). This makes the aluminum nitride template prone to fluctuations during the growth process, and the uniformity and crystal quality may gradually deteriorate over time.

[0005] At the same time, in the growth process of aluminum nitride templates, traditional optimization methods usually only adjust a single parameter, lacking systematic consideration of the interaction of multiple parameters, making the optimization efficiency low in high-dimensional parameter space and difficult to avoid local optimal solutions. Especially in large-scale epitaxial growth, traditional optimization methods are difficult to fully integrate the influence of multiple factors such as temperature, gas flow, and pressure, and cannot achieve global optimization, thus affecting the quality and production efficiency of the film. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0007] To this end, the first aspect of the present invention proposes a method for optimizing aluminum nitride template process parameters based on deep reinforcement learning.

[0008] A second aspect of the present invention provides a device for processing aluminum nitride template process parameters based on deep reinforcement learning.

[0009] A third aspect of the present invention provides a readable storage medium.

[0010] In view of this, the first aspect of the present invention provides an optimization method for aluminum nitride template process parameters based on deep reinforcement learning, including: setting one or more process parameters of the aluminum nitride template; defining a state space and an action space based on the process parameters for preparing the aluminum nitride template; constructing a neural network value function Q(S, A)=θ(S, A), wherein Q(S, A) is the value of the state-action pair, θ is the neural network, S is the state space parameter, and A is the action to be evaluated, wherein the input layer of the neural network takes the state space as input, the output layer of the neural network corresponds to the value of each action in the action space, the weight parameter set of the neural network is W, and the bias The parameter set is b; at the end of each time step, based on the value of the state-action pair calculated by the neural network value function Q(S, A), an ε-greedy strategy is used to select and implement actions to change corresponding process parameters; a reward function is constructed to calculate the reward value corresponding to the action implemented at the end of the n-1th time step at the end of the nth time step, and the reward function is related to the reflectivity curve of the aluminum nitride template growth, and n≥2; within a predetermined time length, at the end of each time step, actions are implemented to change the process parameters to grow the aluminum nitride template, and the neural network θ is trained according to the value function and the reward function to form a new neural network value function Q(S, A).

[0011] The optimization method of aluminum nitride template process parameters based on deep reinforcement learning proposed by the present invention, within a predetermined time, during the growth process of the aluminum nitride template, the neural network θ is trained using the value function and the reward function, and a new neural network value function Q(S, A) can be obtained. At the end of each time step, according to the value of the state-action pair calculated by the new neural network value function Q(S, A), the ε-greedy strategy is used to select and implement actions at the end of each time step to change the process parameters of the aluminum nitride template, and then the growth process of the aluminum nitride template is controlled, so that the aluminum nitride template grows more uniformly and more stably. At the same time, the present invention replaces the method of optimization based on artificial experience with the optimization method of aluminum nitride template process parameters based on deep reinforcement learning, realizes the automatic optimization of aluminum nitride template process parameters, improves the efficiency of the process parameter optimization process of the aluminum nitride template, and then improves the overall stability and production efficiency of the aluminum nitride template.

[0012] By training the neural network θ, it can respond to changes in monitoring data in real time during the growth of aluminum nitride templates, ensuring process flexibility. Compared with traditional fixed parameter settings, the neural network θ can optimize the film quality in real time during the growth process, thereby growing a uniform and stable aluminum nitride template.

[0013] At the same time, by training the neural network θ, the optimization method of the process parameters of the aluminum nitride template based on deep reinforcement learning has the ability of global optimization. This method can simultaneously consider the interaction of complex factors in the process parameters in the multi-dimensional parameter space, thereby realizing the global optimization of the large-size aluminum nitride epitaxial growth process. This method avoids the limitations of the local optimal solution and ensures that the optimization process is more comprehensive and efficient.

[0014] The time step is the interval between two adjacent actions.

[0015] Specifically, a neural network is used to construct a value function, which can express the precise correspondence between the growth state of the aluminum nitride template and the output action value. Specifically, the growth state parameters of the aluminum nitride template are input into the input layer of the neural network, and the values ​​of each action in the action space are obtained in the output layer of the neural network. Among them, the neural network is also provided with a weight parameter set W and a bias parameter set b. During the training process of the neural network, the weight parameter set W and the bias parameter set b will be continuously adjusted, with the aim of making the output of the neural network as consistent as possible with the expected results. Among them, the process parameters of the aluminum nitride template are temperature, pressure, gas flow, etc. The state space is selected from the above parameters to express the growth state of the aluminum nitride template; the action space is used to increase the process parameters, keep the process parameters unchanged or reduce the process parameters, such as increasing or decreasing the temperature by 1°C to adjust the state space parameters.

[0016] Specifically, the reward function is used to calculate the reward value corresponding to the action performed at the end of the previous time step at the end of each time step. For example, at the end of the nth time step, the reward value corresponding to the action performed at the end of the n-1th time step is calculated, the n-1th time step is the previous time step of the nth time step, and n is greater than or equal to 2. In which, in-situ monitoring probes are respectively set at the center of the aluminum nitride template epitaxial wafer and at the position close to the edge of the aluminum nitride template epitaxial wafer to monitor and obtain the reflectivity curve in real time. The reward function is related to the reflectivity curve of the aluminum nitride template growth. The change of the reflectivity curve will change the change of the reward function, and then change the reward value of the corresponding implementation action.

[0017] In addition, the method for optimizing the process parameters of the aluminum nitride template based on deep reinforcement learning in the above technical solution provided by the present invention may also have the following additional technical features:

[0018] In some technical solutions of the present invention, optionally, the neural network is a fully connected layer neural network.

[0019] In this technical solution, the neural network is a fully connected layer neural network, which has the characteristics of strong feature extraction ability, excellent fitting ability, and good model expression ability. It enables the constructed value function to more accurately express the correspondence between the state and the output action value, and provides an effective decision-making basis for the deep reinforcement learning process.

[0020] In some technical solutions of the present invention, optionally, at the end of n-1 time steps, action A is performed. n-1 The reward function is defined as: R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ), where R n-1 is the reward value, α and β are weight coefficients used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 )’s contribution to the reward, and n ≥ 2; σc n represents the standard deviation of multiple first sampling values ​​on the in-situ monitoring reflectivity curve of the center point of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the nth time step; σc n-1 represents the standard deviation of multiple first sampling values ​​on the in-situ monitoring reflectivity curve of the center point of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the n-1th time step; σe n represents the standard deviation of multiple second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the nth time step; σe n-1 represents the standard deviation of multiple second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the n-1th time step; U n is the root mean square. At the end of the nth time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n , U n-1 is the root mean square. At the end of the n-1th time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n-1 .

[0021] In this technical solution, action A is performed at the end of n-1 time steps. n-1 The reward function is defined as: R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ), where R n-1 is the reward value, α and β are weight coefficients used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 )'s contribution to the reward, where α and β are empirical data, determined by the effect of past aluminum nitride template growth. n +σe n -σc n-1 -σe n-1 ) can reflect the influence of the action in the n-1th time step on the standard deviation of the reflectivity curve of the aluminum nitride template growth, that is, the influence of the reflectivity oscillation intensity, which is used to evaluate the crystal quality. The larger the standard deviation, the stronger the oscillation, the higher the quality of the thin film crystal, and the larger the reward value of the n-1th time step. At the same time, (U n -U n-1 ) can reflect the impact of the action in the n-1th time step on the difference between the reflectivity curves at the center and near the edge, (U n -U n-1 ) is larger, indicating that the difference between the reflectivity curves at the center and near the edge is larger, the uniformity of the aluminum nitride template is worse, and the reward value of the n-1th time step is lower. The reward function takes into account the in-situ monitoring reflectivity curves at the center and near the edge, while taking into account the growth quality and uniformity of the aluminum nitride template, and comprehensively improves the process technology of the aluminum nitride template from multiple dimensions.

[0022] For example, let the current time be the end time of the nth time step. In the time window with the current time as the end point, the in-situ monitoring reflectivity curve of the epitaxial wafer center point is sampled to obtain a group of sampling points C1, C2, C3, ..., and calculate the standard deviation σc of all sampling points n The in-situ monitoring reflectivity curve near the edge of the epitaxial wafer is sampled to obtain a set of sampling points E1, E2, E3, ..., and the standard deviation σe of all sampling points is calculated nThe sampling rate and time window of the epitaxial wafer center point and the point close to the edge are the same, the sampling rate is 10ms to 10s, and the time window is 1min to 30min. The time window is the time period for sampling the in-situ monitoring reflectivity curve of the center point of the aluminum nitride template epitaxial wafer and the point close to the edge of the aluminum nitride template epitaxial wafer in each time step.

[0023] Assume that the current time is the end time of the nth time step. For the sampled C1, C2, C3, ... and E1, E2, E3, ..., calculate C1-E1, C2-E2, C3-E3, ... to obtain a series. The root mean square of the series is calculated as U n .

[0024] Based on the above calculation parameters, the reward function for the action taken at the end of the n-1th time step is defined as:

[0025] R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ). R can be obtained through the reward function formula n-1 , using R n-1 Evaluate the reward for taking the action at the end of n-1 time steps.

[0026] Exemplarily, at the beginning of AlN template growth, which is the first time step, within the time window ending at the end of the first time step, the in-situ monitoring reflectivity curves at the center and near the edge of the epitaxial wafer are sampled. At the end of the first time step, the state is S1, and the corresponding implementation of the first action is A1. When in the second time step, within the time window ending at the end of the second time step, the in-situ monitoring reflectivity curves at the center and near the edge of the epitaxial wafer are sampled. At the end of the second time step, the state is S2, and the corresponding implementation of the second action is A2. When in the n-1th time step, within the time window ending at the end of the n-1th time step, the in-situ monitoring reflectivity curves at the center and near the edge of the epitaxial wafer are sampled. At the end of the n-1th time step, the state is S n-1 , the corresponding implementation of the n-1th action is A n-1 At the nth time step, within the time window ending at the end of the nth time step, the in-situ monitoring reflectivity curves at the center and near the edge of the epitaxial wafer are sampled. At the end of the nth time step, the state is S n , the corresponding action is A n .

[0027] In some technical solutions of the present invention, optionally, according to the neural network value function and the reward function, training the neural network θ includes: establishing a second neural network θ' that is identical to the neural network θ; updating the Q value by using the time difference according to the following formula, Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ;θ);θ'), where Q target (S n-1 , A n-1 ) is the updated value of the corresponding state-action pair at the end of the n-1th time step, R n-1 is the reward value for taking the action at the end of the n-1th time step, γQ(S n ,maxQ(S n , A n ; θ);θ') represents the use of neural network θ to select action A according to the greedy strategy n , and use the second neural network θ' to calculate (S n , A n ) is the Q value of the state-action pair, γ is the attenuation factor, S n-1 represents the state parameters of the state space at the end of the n-1th time step, S n represents the state parameters of the state space at the end of the nth time step, A n-1 represents the action performed at the end of the n-1th time step, A n represents the action implemented at the end of the nth time step; the obtained Q target (S n-1 , A n-1 ) is brought into the neural network θ as the target value, and the weight parameter set W and the bias parameter set b are updated by the back-propagation gradient descent method to update the neural network θ; and at predetermined intervals, the updated neural network θ is copied to the second neural network θ' to update the second neural network θ'; and the above process is repeated within a training period.

[0028] In this technical solution, according to the neural network value function and the reward function, training the neural network θ includes establishing a second neural network θ' that is identical to the neural network θ, so that the training of the neural network θ is more stable. Specifically, by Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ; θ);θ') calculation, we can get Qtarget (S n-1 , A n-1 ), Q target (S n-1 , A n-1 ) is brought into the neural network θ as the target value, and the weight parameter set W and the bias parameter set b are updated by the back-propagation gradient descent method to update the neural network θ. At predetermined intervals, the updated neural network θ is copied to the second neural network θ', and the second neural network θ' is updated, so as to smoothly update the Q value and train the neural network θ, thereby improving the stability of the neural network θ. Within a training period, the above process is repeated, the Q value is continuously updated, and the neural network θ is continuously trained, so that the neural network θ can respond to changes in monitoring data in real time during the growth of the aluminum nitride template, ensuring the flexibility of the aluminum nitride template process. Compared with the traditional fixed parameter setting, the neural network θ can optimize the film quality in real time during the growth process, so that the aluminum nitride template maintains uniformity and stability throughout the growth stage.

[0029] Furthermore, the duration of each round of neural network θ training is a fixed-time epitaxial process. If training is required again, the substrate must be replaced and the growth process must be restarted.

[0030] The substrate for growing the aluminum nitride template may be sapphire, silicon (Si), silicon carbide (SiC), etc.

[0031] For example, let the current time be the end time of the nth time step, and the current state be S n , the neural network θ selects action A according to the greedy strategy n , using the second neural network to calculate (S n , A n ) The Q value of the state-action pair is obtained by Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ;θ);θ') formula can be used to obtain Q target (S n-1 , A n-1 ), the obtained Q target (S n-1 , A n-1) is brought into the neural network θ, and the weight parameter set W and bias parameter set b in the neural network θ are updated. Every certain period of time, the updated weight parameter set W and bias parameter set b are copied to the second neural network, thereby stabilizing the Q value update and improving the stability of the neural network θ. Within a training period, the above process is repeated, the Q value is continuously updated, and the neural network θ is continuously trained, so that the neural network θ can respond to changes in monitoring data in real time during the growth of the aluminum nitride template, ensuring the flexibility of the process. Compared with the traditional fixed parameter setting, the neural network θ can optimize the film quality in real time during the growth process, so that a uniform and stable aluminum nitride template can be grown.

[0032] At the same time, by continuously training the neural network θ, the neural network θ replaces the manual experience in process optimization, avoiding the inefficiency and uncertainty of the traditional trial and error method. Through interactive feedback with the environment, the optimization method of process parameters can be accurately learned to achieve efficient automatic optimization. Without manual intervention, the overall stability and production efficiency of the aluminum nitride template are significantly improved.

[0033] At the same time, by continuously training the neural network θ, the optimization method of the process parameters of the aluminum nitride template based on deep reinforcement learning has the ability of global optimization. This method can simultaneously consider the interaction of complex factors in the process parameters in the multi-dimensional parameter space, thereby realizing the global optimization of the large-size aluminum nitride epitaxial growth process. This method avoids the limitations of the local optimal solution and ensures that the optimization process is more comprehensive and efficient.

[0034] In some technical solutions of the present invention, optionally, the method for optimizing the process parameters of the aluminum nitride template based on deep reinforcement learning also includes: growing the aluminum nitride template based on the new neural network θ, characterizing the grown aluminum nitride template, and verifying the new neural network θ.

[0035] In this technical solution, the optimization method of aluminum nitride template process parameters based on deep reinforcement learning also includes using the trained neural network θ to grow the aluminum nitride template to verify the effectiveness of the neural network θ. During the growth of the aluminum nitride template, the neural network θ will accurately give the growth parameter optimization action at each time step, and the growth process will end when the predetermined time is reached. Finally, the prepared aluminum nitride template is characterized, and the optimization effect of the new neural network θ in the aluminum nitride template growth process parameters is verified by evaluating the quality and uniformity of the crystal.

[0036] In some technical solutions of the present invention, optionally, the state space parameters include at least multiple of the following parameters: temperature, pressure, gas flow, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, and growth time.

[0037] In this technical solution, the state space parameters at least include temperature, pressure, gas flow, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, growth time and other parameters. The state space can fully express the growth state of the aluminum nitride template, and then the neural network θ has more refined data support, which improves the accuracy of the aluminum nitride template growth process control.

[0038] Among them, the initial values ​​of process parameters such as temperature, pressure, hydrogen (H2) carrier gas flow, trimethylaluminum (TMAl) flow, ammonia (NH3) flow, and epitaxial wafer rotation speed in the state space should be based on the empirical values ​​of the optimal process parameters for aluminum nitride template growth. By selecting these empirical values ​​as the initial state, the neural network θ can be helped to converge to the appropriate control strategy faster, avoiding learning from scratch and improving the efficiency of neural network θ training.

[0039] At the same time, the in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, etc. in the state space are obtained by real-time monitoring of the in-situ monitoring system of metal organic chemical vapor deposition. In-situ monitoring probes are set at the center and near the edge of the epitaxial wafer to monitor the reflectivity curves of different positions of the aluminum nitride template in real time. The monitoring wavelength can be selected from ultraviolet (200nm to 400nm), visible light (400nm to 700nm), and infrared (above 700nm).

[0040] In some technical solutions of the present invention, optionally, the action space includes at least a plurality of the following actions: increasing, maintaining unchanged or decreasing the following growth process parameters: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed.

[0041] In this technical solution, the action space includes at least a plurality of the following actions: increasing, keeping unchanged or decreasing the following growth process parameters: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed. The action space includes the key process parameters for the growth of aluminum nitride templates. The neural network θ adjusts the process parameters to adapt to the dynamic requirements of the growth conditions, thereby making it easier to grow aluminum nitride templates with high crystal quality and good uniformity.

[0042] Exemplarily, during the growth of the aluminum nitride template, an action is performed at the end of each time step, and the time step can be set to 30s-30min. The temperature increases or decreases by 1°C to 5°C each time. The pressure increases or decreases by 1Torr to 5Torr each time. The H2 carrier gas flow rate increases or decreases by 100sccm to 1000sccm each time. The TMAl flow rate increases or decreases by 5sccm to 50sccm each time. The NH3 flow rate increases or decreases by 100sccm to 500sccm each time. The epitaxial wafer rotation speed increases or decreases by 2r / min to 20r / min each time.

[0043] In some technical solutions of the present invention, optionally, the neural network includes an input layer, one or more hidden layers and an output layer. The input layer contains state parameters defined in the state space, and the number of multiple state parameters in the state space is the same as the number of multiple first neurons in the input layer; the one or more hidden layers are used to extract state parameter features, and the hidden layers use a ReLU (rectified linear unit) activation function; the output layer outputs the value of the action defined in the action space, the number of multiple second neurons in the output layer is the same as the number of actions in the action space, one second neuron represents the value of one action, and the output layer uses a linear activation function.

[0044] In this technical solution, through the input layer, one or more hidden layers and output layer of the neural network, during the growth process of the aluminum nitride template, the state of the aluminum nitride template process parameters and the output action value are accurately corresponded. Among them, the neural network is also provided with a weight parameter set W and a bias parameter set b. During the training process of the neural network, the weight parameter set W and the bias parameter set b are continuously adjusted, with the purpose of making the output of the neural network as consistent as possible with the expected result.

[0045] Specifically, the state parameters defined in the state space are input into the input layer, and the number of the multiple state parameters of the state space is the same as the number of the multiple first neurons of the input layer, so that the multiple state parameters of the state space correspond to the multiple first neurons of the input layer.

[0046] Specifically, one or more hidden layers are used to extract features of state parameters, and the hidden layers use a ReLU activation function to increase nonlinear expression capabilities.

[0047] Specifically, the output layer outputs the value of the action defined in the action space, the number of the multiple second neurons in the output layer is the same as the number of actions in the action space, one second neuron represents the value of one action, and the output layer adopts a linear activation function to adapt to the value prediction requirements.

[0048] The second aspect of the present invention provides a processing device for aluminum nitride template process parameters based on deep reinforcement learning, and the processing device includes a setting unit, a definition unit, a first construction unit, a parameter change unit, a second construction unit and a training unit. Among them, the setting unit is used to set one or more process parameters of the aluminum nitride template; the definition unit is used to define the state space and the action space based on the process parameters for preparing the aluminum nitride template; the first construction unit is used to construct a neural network value function Q(S, A) = θ(S, A), wherein Q(S, A) is the value of the state-action pair, θ is the neural network, S is the state space parameter, and A is the action to be evaluated, wherein the input layer of the neural network takes the state space as input, the output layer of the neural network corresponds to the value of each action in the action space, the weight parameter set of the neural network is W, and the bias parameter set is b; the parameter change unit is used to conclude at each time step When the state-action pair is bound, the ε-greedy strategy is used to select and implement actions to change the corresponding process parameters based on the value of the state-action pair calculated by the neural network value function Q(S, A); the second construction unit is used to construct a reward function, and at the end of the nth time step, the reward value corresponding to the action implemented at the end of the n-1th time step is calculated, and the reward function is related to the reflectivity curve of the aluminum nitride template growth, and n≥2; the training unit is used to implement actions to change the process parameters at the end of each time step within a predetermined time length to grow the aluminum nitride template, and train the neural network θ according to the value function and the reward function to form a new neural network value function Q(S, A).

[0049] In the technical scheme, the processing device of the aluminum nitride template process parameters based on deep reinforcement learning includes a setting unit, a definition unit, a first construction unit, a parameter change unit, a second construction unit and a training unit. The present invention uses a value function and a reward function to train the neural network θ, and a new neural network value function Q(S, A) can be obtained. At the end of each time step, according to the value of the state-action pair calculated by the new neural network value function Q(S, A), an ε-greedy strategy is adopted to select and implement actions at the end of each time step to change the process parameters of the aluminum nitride template, thereby controlling the growth process of the aluminum nitride template, so that the aluminum nitride template grows more uniformly and more stably. At the same time, the present invention replaces the method of process optimization based on artificial experience with the optimization method of the aluminum nitride template process parameters based on deep reinforcement learning, realizes the automatic optimization of the aluminum nitride template process parameters, improves the efficiency of the process parameter optimization process of the aluminum nitride template, and thus improves the overall stability and production efficiency of the aluminum nitride template.

[0050] The third aspect of the present invention provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of a method for optimizing aluminum nitride template process parameters based on deep reinforcement learning as in any of the above-mentioned technical solutions.

[0051] Since the present invention provides a readable storage medium having a program or instruction stored thereon, and the program or instruction, when executed by a processor, implements the steps of the method for optimizing the process parameters of an aluminum nitride template based on deep reinforcement learning as in any of the above-mentioned technical solutions, the readable storage medium has all the beneficial effects of the steps of the method for optimizing the process parameters of an aluminum nitride template based on deep reinforcement learning as in any of the above-mentioned technical solutions, which will not be repeated here.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] Figure 1 One of the flowcharts of a method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of a neural network according to an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of time steps, time windows, states, and moments of implementing actions according to an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a device for processing aluminum nitride template process parameters based on deep reinforcement learning according to an embodiment of the present invention;

[0058] Figure 5 This is a second flowchart of a method for optimizing process parameters of an aluminum nitride template based on deep reinforcement learning according to an embodiment of the present invention;

[0059] Figure 6 This is a third flowchart of a method for optimizing process parameters of an aluminum nitride template based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0062] Refer to the following Figures 1 to 6 A method for optimizing aluminum nitride template process parameters based on deep reinforcement learning, a device for processing aluminum nitride template process parameters based on deep reinforcement learning, and a readable storage medium are described according to some embodiments of the present invention.

[0063] Aspects of the present invention provide a method for optimizing aluminum nitride template process parameters based on deep reinforcement learning, such as Figure 1 As shown, including:

[0064] Step 102: setting one or more process parameters of the aluminum nitride template;

[0065] Step 104: defining a state space and an action space based on process parameters for preparing the aluminum nitride template;

[0066] Step 106: Construct a neural network value function Q(S, A)=θ(S, A), where Q(S, A) is the value of the state-action pair, θ is the neural network, S is the state space parameter, and A is the action to be evaluated. The input layer of the neural network takes the state space as input, and the output layer of the neural network corresponds to the value of each action in the action space. The weight parameter set of the neural network is W, and the bias parameter set is b.

[0067] Step 108: At the end of each time step, based on the value of the state-action pair calculated by the neural network value function Q(S, A), an ε-greedy strategy is used to select and implement actions to change the corresponding process parameters;

[0068] Step 110: constructing a reward function for calculating, at the end of the nth time step, a reward value corresponding to the action performed at the end of the n-1th time step, wherein the reward function is related to a reflectivity curve of the aluminum nitride template growth, and n≥2;

[0069] Step 112: Within a predetermined time period, at the end of each time step, actions are taken to change process parameters to grow an aluminum nitride template, and the neural network θ is trained according to the value function and the reward function to form a new neural network value function Q(S, A).

[0070] In this embodiment, the optimization method of aluminum nitride template process parameters based on deep reinforcement learning proposed by the present invention, within a predetermined time, during the growth process of the aluminum nitride template, the value function and the reward function are used to train the neural network θ, and a new neural network value function Q(S, A) can be obtained. At the end of each time step, according to the value of the state-action pair calculated by the new neural network value function Q(S, A), the ε-greedy strategy is selected and the action is implemented at the end of each time step to change the process parameters of the aluminum nitride template, thereby controlling the growth process of the aluminum nitride template, so that the aluminum nitride template grows more uniformly and more stably. At the same time, the present invention replaces the process optimization method based on artificial experience with the aluminum nitride template process parameter optimization method based on deep reinforcement learning, realizes the automatic optimization of the aluminum nitride template process parameters, improves the efficiency of the process parameter optimization process of the aluminum nitride template, and thus improves the overall stability and production efficiency of the aluminum nitride template. By training the neural network θ, the neural network θ can respond to the changes in the monitoring data in real time during the growth of the aluminum nitride template, ensuring the flexibility of the process. Compared with traditional fixed parameter settings, the neural network θ can optimize the film quality in real time during the growth process, thereby growing a uniform and stable aluminum nitride template.

[0071] At the same time, by training the neural network θ, the optimization method of the process parameters of the aluminum nitride template based on deep reinforcement learning has the ability of global optimization. This method can simultaneously consider the interaction of complex factors in the process parameters in the multi-dimensional parameter space, thereby realizing the global optimization of the large-size aluminum nitride epitaxial growth process. This method avoids the limitations of the local optimal solution and ensures that the optimization process is more comprehensive and efficient.

[0072] The time step is the interval between two adjacent actions.

[0073] Specifically, a neural network is used to construct a value function, which can express the precise correspondence between the growth state of the aluminum nitride template and the output action value. Specifically, the growth state parameters of the aluminum nitride template are input into the input layer of the neural network, and the value of each action in the action space can be obtained in the output layer of the neural network. Among them, the neural network is also provided with a weight parameter set W and a bias parameter set b. During the training process of the neural network, the weight parameter set W and the bias parameter set b will be continuously adjusted, with the aim of making the output of the neural network as consistent as possible with the expected results. Among them, the process parameters of the aluminum nitride template are temperature, pressure, gas flow, etc. The state space is selected from the above parameters to express the growth state of the aluminum nitride template; the action space is used to increase the process parameters, keep the process parameters unchanged or reduce the process parameters, such as increasing or decreasing the temperature by 1°C to adjust the state space parameters.

[0074] Specifically, the reward function is used to calculate the reward value corresponding to the action performed at the end of the previous time step at the end of each time step. For example, at the end of the nth time step, the reward value corresponding to the action performed at the end of the n-1th time step is calculated, the n-1th time step is the previous time step of the nth time step, and n is greater than or equal to 2. In which, in-situ monitoring probes are respectively set at the center of the aluminum nitride template epitaxial wafer and near the edge of the aluminum nitride template epitaxial wafer to monitor and obtain the reflectivity curve in real time. The reward function is related to the reflectivity curve of the aluminum nitride template growth. The change of the reflectivity curve will change the change of the reward function, and then change the reward value of the corresponding implementation action.

[0075] In some embodiments of the present invention, optionally, the neural network is a fully connected layer neural network.

[0076] In this embodiment, the neural network is a fully connected layer neural network, which has the characteristics of strong feature extraction ability, excellent fitting ability, and good model expression ability, so that the constructed value function can more accurately express the correspondence between the state and the output action value, providing an effective decision-making basis for the deep reinforcement learning process.

[0077] In some embodiments of the present invention, optionally, at the end of n-1 time steps, action A is performed. n-1 The reward function is defined as: R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ), where R n-1 is the reward value, α and β are weight coefficients used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 )’s contribution to the reward, and n ≥ 2; σc n represents the standard deviation of multiple first sampling values ​​on the in-situ monitoring reflectivity curve of the center point of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the nth time step; σc n-1 represents the standard deviation of multiple first sampling values ​​on the in-situ monitoring reflectivity curve at the center point of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the n-1th time step; σe n represents the standard deviation of multiple second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the nth time step; σen-1 represents the standard deviation of multiple second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the n-1th time step; U n is the root mean square. At the end of the nth time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n , U n-1 is the root mean square. At the end of the n-1th time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n-1 .

[0078] In this embodiment, action A is performed at the end of n-1 time steps. n-1 The reward function is defined as: R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ), where R n-1 is the reward value, α and β are weight coefficients used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 )'s contribution to the reward, where α and β are empirical data, determined by the effect of past aluminum nitride template growth. n +σe n -σc n-1 -σe n-1 ) can reflect the influence of the action in the n-1th time step on the standard deviation of the reflectivity curve of the aluminum nitride template growth, that is, the influence of the reflectivity oscillation intensity, which is used to evaluate the crystal quality. The larger the standard deviation, the stronger the oscillation, the higher the quality of the thin film crystal, and the larger the reward value of the n-1th time step. At the same time, (U n -U n-1 ) can reflect the impact of the action in the n-1th time step on the difference between the reflectivity curves at the center and near the edge, (U n -U n-1) is larger, indicating that the difference between the reflectivity curves at the center and near the edge is larger, the uniformity of the aluminum nitride template is worse, and the reward value of the n-1th time step is lower. The reward function takes into account the in-situ monitoring reflectivity curves at the center and near the edge, while taking into account the growth quality and uniformity of the aluminum nitride template, and comprehensively improves the process technology of the aluminum nitride template from multiple dimensions.

[0079] For example, let the current time be the nth time step. In the time window with the current time as the end point, the in-situ monitoring reflectivity curve of the epitaxial wafer center point is sampled to obtain a group of sampling points C1, C2, C3, ..., and calculate the standard deviation σc of all sampling points n The in-situ monitoring reflectivity curve near the edge of the epitaxial wafer is sampled to obtain a set of sampling points E1, E2, E3, ..., and the standard deviation σe of all sampling points is calculated n The sampling rate and time window of the epitaxial wafer center point and the point close to the edge are the same, the sampling rate is 10ms to 10s, and the time window is 1min to 30min. The time window is the time period for sampling the in-situ monitoring reflectivity curve of the center point of the aluminum nitride template epitaxial wafer and the point close to the edge of the aluminum nitride template epitaxial wafer in each time step.

[0080] Assume that the current time step is the nth time step. For the sampled C1, C2, C3, ... and E1, E2, E3, ..., calculate C1-E1, C2-E2, C3-E3, ... to obtain a series. The root mean square of the series is calculated as U n .

[0081] According to the above calculation parameters, the reward function of the n-1th time action is defined as:

[0082] R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ). R can be obtained through the reward function formula n-1 , using R n-1 Evaluate the reward for taking the action at the end of n-1 time steps.

[0083] For example, Figure 3As shown, at the beginning of the growth of the aluminum nitride template, which is the first time step, within the time window with the end of the first time step as the end point, the in-situ monitoring reflectivity curves of the center point and near the edge of the epitaxial wafer are sampled. At the end of the first time step, the state is S1, and the corresponding implementation of the first action is A1. When in the second time step, within the time window with the end of the second time step as the end point, the in-situ monitoring reflectivity curves of the center point and near the edge of the epitaxial wafer are sampled. At the end of the second time step, the state is S2, and the corresponding implementation of the second action is A2. When in the n-1th time step, within the time window with the end of the n-1th time step as the end point, the in-situ monitoring reflectivity curves of the center point and near the edge of the epitaxial wafer are sampled. At the end of the n-1th time step, the state is S n-1 , the corresponding implementation of the n-1th action is A n-1 At the nth time step, within the time window ending at the end of the nth time step, the in-situ monitoring reflectivity curves at the center and near the edge of the epitaxial wafer are sampled. At the end of the nth time step, the state is S n , the corresponding action is A n .

[0084] In some embodiments of the present invention, optionally, training the neural network θ according to the neural network value function and the reward function includes: establishing a second neural network θ' identical to the neural network θ; updating the Q value by using the time difference according to the following formula, Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ;θ);θ'), where Q target (S n-1 , A n-1 ) is the updated value of the corresponding state-action pair at the end of the n-1th time step, R n-1 is the reward value for taking the action at the end of the n-1th time step, γQ(S n ,maxQ(S n , A n ; θ);θ') represents the use of neural network θ to select action A according to the greedy strategy n , and use the second neural network θ' to calculate (S n , A n ) is the Q value of the state-action pair, γ is the attenuation factor, S n-1 represents the state parameters of the state space at the end of the n-1th time step, S n represents the state parameters of the state space at the end of the nth time step, A n-1represents the action performed at the end of the n-1th time step, A n represents the action implemented at the end of the nth time step; the obtained Q target (S n-1 , A n-1 ) is brought into the neural network θ as the target value, and the weight parameter set W and the bias parameter set b are updated by the back-propagation gradient descent method to update the neural network θ; and at predetermined intervals, the updated neural network θ is copied to the second neural network θ' to update the second neural network θ'; and the above process is repeated within a training period.

[0085] In this embodiment, according to the neural network value function and the reward function, training the neural network θ includes establishing a second neural network θ' that is identical to the neural network θ, so that the training of the neural network θ is more stable. Specifically, by Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ; θ);θ') calculation, we can get Q target (S n-1 , A n-1 ), Q target (S n-1 , A n-1 ) is brought into the neural network θ as the target value, and the weight parameter set W and the bias parameter set b are updated by the back-propagation gradient descent method to update the neural network θ. At predetermined intervals, the updated neural network θ is copied to the second neural network θ', and the second neural network θ' is updated, so as to smoothly update the Q value and train the neural network θ, thereby improving the stability of the neural network θ. Within a training period, the above process is repeated, the Q value is continuously updated, and the neural network θ is continuously trained, so that the neural network θ can respond to changes in monitoring data in real time during the growth of the aluminum nitride template, ensuring the flexibility of the aluminum nitride template process. Compared with the traditional fixed parameter setting, the neural network θ can optimize the film quality in real time during the growth process, so that the aluminum nitride template maintains uniformity and stability throughout the growth stage.

[0086] Furthermore, the duration of each round of neural network θ training is a fixed-time epitaxial process. If training is required again, the substrate must be replaced and the growth process must be restarted.

[0087] The substrate for growing the aluminum nitride template may be sapphire, silicon (Si), silicon carbide (SiC), etc.

[0088] For example, let the current time be the end time of the nth time step, and the neural network θ selects action A according to the greedy strategy n , using the second neural network to calculate (S n , A n ) The Q value of the state-action pair is obtained by Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n , A n ;θ);θ') formula can be used to obtain Q target (S n-1 , A n-1 ), the obtained Q target (S n-1 , A n-1 ) is brought into the neural network θ, the weight parameter set W and the bias parameter set b in the neural network θ are updated, and the updated weight parameter set W and the bias parameter set b are copied to the second neural network at regular intervals, thereby stabilizing the Q value update and improving the stability of the neural network θ. Within a training period, the above process is repeated, the Q value is continuously updated, and the neural network θ is continuously trained, so that the neural network θ can respond to changes in monitoring data in real time during the growth of the aluminum nitride template, ensuring the flexibility of the process. Compared with the traditional fixed parameter setting, the neural network θ can optimize the film quality in real time during the growth process, so that a uniform and stable aluminum nitride template can be grown.

[0089] At the same time, by continuously training the neural network θ, the neural network θ replaces the manual experience in process optimization, avoiding the inefficiency and uncertainty of the traditional trial and error method. Through interactive feedback with the environment, the optimization method of process parameters can be accurately learned to achieve efficient automatic optimization. Without manual intervention, the overall stability and production efficiency of the aluminum nitride template are significantly improved.

[0090] At the same time, by continuously training the neural network θ, the optimization method of the process parameters of the aluminum nitride template based on deep reinforcement learning has the ability of global optimization. This method can simultaneously consider the interaction of complex factors in the process parameters in the multi-dimensional parameter space, thereby realizing the global optimization of the large-size aluminum nitride epitaxial growth process. This method avoids the limitations of the local optimal solution and ensures that the optimization process is more comprehensive and efficient.

[0091] In some technical solutions of the present invention, optionally, the method for optimizing the process parameters of the aluminum nitride template based on deep reinforcement learning also includes: growing the aluminum nitride template based on the new neural network θ, characterizing the grown aluminum nitride template, and verifying the new neural network θ.

[0092] In this technical solution, the optimization method of aluminum nitride template process parameters based on deep reinforcement learning also includes using the trained neural network θ to grow the aluminum nitride template to verify the effectiveness of the neural network θ. During the growth of the aluminum nitride template, the neural network θ will accurately give the growth parameter optimization action at each time step, and the growth process will end when the predetermined time is reached. Finally, the prepared aluminum nitride template is characterized, and the optimization effect of the new neural network θ in the aluminum nitride template growth process parameters is verified by evaluating the quality and uniformity of the crystal.

[0093] In some embodiments of the present invention, optionally, the state space parameters include at least a plurality of the following parameters: temperature, pressure, gas flow rate, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, and growth time.

[0094] In this embodiment, the state space is a multidimensional variable, and the state space parameters include at least a plurality of the following parameters: temperature, pressure, gas flow, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, growth time, etc. The multidimensional variable state space can fully express the growth state of the aluminum nitride template, thereby enabling the neural network θ to have more refined data support, thereby improving the accuracy of the aluminum nitride template growth process control.

[0095] Among them, the initial values ​​of process parameters such as temperature, pressure, hydrogen (H2) carrier gas flow, trimethylaluminum (TMAl) flow, ammonia (NH3) flow, and epitaxial wafer rotation speed in the state space should be based on the empirical values ​​of the optimal process parameters for aluminum nitride template growth. By selecting these empirical values ​​as the initial state, the neural network θ can be helped to converge to the appropriate control strategy faster, avoiding learning from scratch and improving the efficiency of neural network θ training.

[0096] At the same time, the in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, etc. in the state space are obtained by real-time monitoring of the in-situ monitoring system of metal organic chemical vapor deposition. In-situ monitoring probes are set at the center and near the edge of the epitaxial wafer to monitor the reflectivity curves of different positions of the aluminum nitride template in real time. The monitoring wavelength can be selected from ultraviolet (200nm to 400nm), visible light (400nm to 700nm), and infrared (above 700nm).

[0097] In some embodiments of the present invention, optionally, the action space includes at least a plurality of the following actions: increasing, maintaining unchanged or decreasing the following growth process parameters: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed.

[0098] In this embodiment, the action space includes at least a plurality of the following actions: increasing, maintaining unchanged or decreasing the following growth process parameters: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed. The action space includes the key process parameters for the growth of the aluminum nitride template. The neural network θ interacts with the growth state of the aluminum nitride template through these actions, and adjusts the various process parameters in real time and in multiple dimensions to adapt to the dynamic requirements of the growth conditions, thereby making it easier to grow an aluminum nitride template with high crystal quality and good uniformity.

[0099] Exemplarily, during the growth of the aluminum nitride template, an action is performed every other time step, and the time step can be set to 30s-30min. The temperature increases or decreases by 1°C to 5°C each time. The pressure increases or decreases by 1Torr to 5Torr each time. The H2 carrier gas flow rate increases or decreases by 100sccm to 1000sccm each time. The TMAl flow rate increases or decreases by 5sccm to 50sccm each time. The NH3 flow rate increases or decreases by 100sccm to 500sccm each time. The epitaxial wafer speed increases or decreases by 2r / min to 20r / min each time.

[0100] In some embodiments of the present invention, optionally, Figure 2 As shown, the neural network includes an input layer, one or more hidden layers and an output layer. The input layer contains state parameters defined in the state space, and the number of multiple state parameters in the state space is the same as the number of multiple first neurons in the input layer; the one or more hidden layers are used to extract state parameter features, and the hidden layers use the ReLU activation function; the output layer outputs the value of the action defined in the action space, and the number of multiple second neurons in the output layer is the same as the number of actions in the action space, one second neuron represents the value of one action, and the output layer uses a linear activation function.

[0101] In this embodiment, through the input layer, one or more hidden layers and the output layer of the neural network, during the growth process of the aluminum nitride template, the state of the aluminum nitride template process parameters and the output action value are accurately corresponded. Among them, the neural network is also provided with a weight parameter set W and a bias parameter set b. During the training process of the neural network, the weight parameter set W and the bias parameter set b are continuously adjusted, with the purpose of making the output of the neural network as consistent as possible with the expected result.

[0102] Specifically, the state parameters defined in the state space are input into the input layer, and the number of the multiple state parameters of the state space is the same as the number of the multiple first neurons of the input layer, so that the multiple state parameters of the state space correspond to the multiple first neurons of the input layer.

[0103] Specifically, one or more hidden layers are used to extract features of state parameters, and the hidden layers use a ReLU activation function to increase nonlinear expression capabilities.

[0104] Specifically, the output layer outputs the value of the action defined in the action space, the number of the multiple second neurons in the output layer is the same as the number of actions in the action space, one second neuron represents the value of one action, and the output layer adopts a linear activation function to adapt to the value prediction requirements.

[0105] The second aspect of the present invention provides a processing device 200 for aluminum nitride template process parameters based on deep reinforcement learning, such as Figure 4 As shown, the processing device 200 includes a setting unit 202, a definition unit 204, a first construction unit 206, a parameter change unit 208, a second construction unit 210 and a training unit 212. The setting unit 202 is used to set one or more process parameters of the aluminum nitride template; the definition unit 204 is used to define the state space and the action space based on the process parameters for preparing the aluminum nitride template; the first construction unit 206 is used to construct a neural network value function Q(S, A) = θ(S, A), wherein Q(S, A) is the value of the state-action pair, θ is the neural network, S is the state space parameter, and A is the action to be evaluated, wherein the input layer of the neural network takes the state space as input, the output layer of the neural network corresponds to the value of each action in the action space, the weight parameter set of the neural network is W, and the bias parameter set is b; the parameter change unit 208 is used to calculate the value of the neural network at each time. At the end of the interval, based on the value of the state-action pair calculated by the neural network value function Q(S, A), an ε-greedy strategy is used to select and implement actions to change the corresponding process parameters; the second construction unit 210 is used to construct a reward function, and at the end of the nth time step, the reward value corresponding to the action implemented at the end of the n-1th time step is calculated, and the reward function is related to the reflectivity curve of the aluminum nitride template growth, and n≥2; the training unit 212 is used to implement actions to change the process parameters at the end of each time step within a predetermined time length to grow the aluminum nitride template, and train the neural network θ according to the value function and the reward function to form a new neural network value function Q(S, A).

[0106] In this embodiment, the processing device 200 of the aluminum nitride template process parameters based on deep reinforcement learning includes a setting unit 202, a definition unit 204, a first construction unit 206, a parameter change unit 208, a second construction unit 210 and a training unit 212. The present invention uses a value function and a reward function to train the neural network θ, and a new neural network value function Q(S, A) can be obtained. At the end of each time step, according to the value of the state-action pair calculated by the new neural network value function Q(S, A), an ε-greedy strategy is adopted to select and implement an action at the end of each time step to change the process parameters of the aluminum nitride template, thereby controlling the growth process of the aluminum nitride template, so that the aluminum nitride template grows more uniformly and more stably. At the same time, the present invention replaces the process optimization method based on artificial experience with the aluminum nitride template process parameter optimization method based on deep reinforcement learning, realizes the automatic optimization of the aluminum nitride template process parameters, improves the efficiency of the process parameter optimization process of the aluminum nitride template, and thus improves the overall stability and production efficiency of the aluminum nitride template.

[0107] A third aspect of the present invention provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for optimizing the process parameters of the aluminum nitride template based on deep reinforcement learning.

[0108] Since the present invention provides a readable storage medium having a program or instruction stored thereon, and the program or instruction, when executed by a processor, implements the steps of the method for optimizing the process parameters of an aluminum nitride template based on deep reinforcement learning as in any of the above-mentioned technical solutions, the readable storage medium has all the beneficial effects of the steps of the method for optimizing the process parameters of an aluminum nitride template based on deep reinforcement learning as in any of the above-mentioned technical solutions, which will not be repeated here.

[0109] In some embodiments of the present invention, optionally, Figure 5 As shown, the aluminum nitride template process parameter optimization method based on deep reinforcement learning includes:

[0110] Step 302: defining a state space and an action space according to the technical characteristics of the aluminum nitride template process optimization;

[0111] Step 304: constructing a neural network value function and a reward function according to the technical characteristics of the aluminum nitride template process optimization;

[0112] Step 306: growing an aluminum nitride template and training a deep reinforcement learning model;

[0113] Step 308: Use the trained deep reinforcement learning model to grow an aluminum nitride template to verify the effectiveness of the model.

[0114] In this embodiment, the state space and action space are defined according to the technical characteristics of the aluminum nitride template process optimization, that is, the process parameters of the aluminum nitride template. Then, according to the technical characteristics of the aluminum nitride template process optimization, a neural network value function and a reward function are constructed. During the growth process of the aluminum nitride template, the deep reinforcement learning model is trained through the neural network value function and the reward function, so that the deep reinforcement learning model is updated. The trained deep reinforcement learning model is used to grow the aluminum nitride template to verify the effectiveness of the model. During the growth process of the aluminum nitride template, the deep reinforcement learning model will accurately give the growth parameter optimization action at each time step, and the growth process will end when the predetermined duration is reached. Finally, the prepared aluminum nitride template is characterized, the quality and uniformity of the crystal are evaluated, and the optimization effect of the deep reinforcement learning model in the aluminum nitride template growth process parameters is verified.

[0115] Among them, the temperature in the process parameters: by adjusting the temperature of the reaction chamber, the growth rate and crystal quality of the aluminum nitride template are controlled. High temperature usually helps to improve the quality of aluminum nitride crystals, but it may also cause parasitic reactions and stress problems.

[0116] Gas flow: mainly includes hydrogen carrier gas flow and precursor gas (such as TMAl and NH3 flow). The adjustment of gas flow directly affects the deposition rate, thickness uniformity and stress of the aluminum nitride template.

[0117] Pressure: The gas pressure in the reaction chamber affects the growth rate and crystal quality of the aluminum nitride template. Lower pressure helps improve the crystal quality of the film, but too low pressure may destroy the uniformity of the aluminum nitride template.

[0118] Reflectivity monitoring: The growth process of the thin film is usually monitored by the reflectivity curve. The change of reflectivity can provide information about the surface quality, thickness and uniformity of the film to help adjust the growth process parameters.

[0119] By continuously adjusting the process parameters of the aluminum nitride template, the aluminum nitride template remains uniform and stable throughout the growth stage, thereby improving the quality of the aluminum nitride template. Even for the preparation of large-sized aluminum nitride templates, the deep reinforcement learning method of the present invention can simultaneously consider the interaction of complex factors such as temperature, gas flow, and pressure in a multidimensional parameter space, thereby achieving global optimization of the large-sized aluminum nitride epitaxial growth process. This method avoids the limitations of local optimal solutions and ensures that the optimization process is more comprehensive and efficient.

[0120] In some embodiments of the present invention, optionally, Figure 6 As shown, the aluminum nitride template process parameter optimization method based on deep reinforcement learning includes:

[0121] Step 402: Initialize a deep reinforcement learning environment for aluminum nitride template growth;

[0122] Step 404: define the state space;

[0123] Step 406: define the action space;

[0124] Step 408: constructing a value function;

[0125] Step 410: construct a reward function;

[0126] Step 412: deep reinforcement learning model training;

[0127] Step 414: Model validity verification.

[0128] Specifically, when initializing the deep reinforcement learning environment for the growth of aluminum nitride templates, prepare MOCVD (metal organic chemical vapor deposition) equipment with controllable process parameter adjustment functions such as temperature, pressure, and H2 flow. An in-situ reflectivity monitoring system is installed in the MOCVD equipment to detect the reflectivity curves of the center and edge of the epitaxial wafer in real time. The substrate for growing the aluminum nitride template can be sapphire, Si, SiC, etc.

[0129] The state space is defined as a multidimensional variable, and the following parameters can be selected: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, growth time, etc. The multidimensional variable state space includes the key processes and growth states of the aluminum nitride template, so that the model has more refined data support when making decisions, and improves the control accuracy.

[0130] The initial values ​​of process parameters such as temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, and epitaxial wafer rotation speed in the state space should be based on the empirical values ​​of the optimal process parameters for aluminum nitride template growth. By selecting these empirical values ​​as the initial state, the model can converge to the appropriate control strategy faster, avoid learning from scratch, and improve training efficiency.

[0131] In-situ monitoring of reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, etc. in state space is obtained by real-time monitoring of the MOCVD in-situ monitoring system. An in-situ monitoring probe is set at the center and near the edge of the epitaxial wafer to monitor the reflectivity curve of different positions of the aluminum nitride template in real time. The monitoring wavelength can be selected from ultraviolet (200-400nm), visible light (400-700nm), and infrared (above 700nm).

[0132] Define the action space and choose from the following actions: increase, keep unchanged or decrease the following growth process parameters: temperature, pressure, H2 carrier gas flow, TMAl flow, NH3 flow, epitaxial wafer rotation speed, etc.

[0133] During the growth process, an action is performed every other time step, and the time step can be set to 30s-30min.

[0134] The temperature increases or decreases by 1-5°C each time; the pressure increases or decreases by 1-5 Torr each time; the H2 carrier gas flow increases or decreases by 100-1000 sccm each time; the TMAl flow increases or decreases by 5-50 sccm each time; the NH3 flow increases or decreases by 100-500 sccm each time; the epitaxial wafer speed increases or decreases by 2-20 r / min each time;

[0135] The action space contains the key process parameters for the growth of aluminum nitride templates. The deep reinforcement learning model interacts with the growth state of the aluminum nitride template through these actions, and adjusts various process parameters in real time and multi-dimensionally to adapt to the dynamic needs of growth conditions, making it easier to grow aluminum nitride templates with high crystal quality and good uniformity.

[0136] The value function is constructed specifically by using a fully connected layer neural network. The input layer of the neural network takes the key growth process of aluminum nitride and the state parameters of the growth process (i.e., the state space) as input, and the number of neurons in the input layer is equal to the number of parameters selected in the state space.

[0137] The network may include one or more hidden layers for extracting state features. The activation function of the hidden layer may adopt the ReLU function to increase the nonlinear expression capability.

[0138] The output layer of the network corresponds to the value of each action in the action space. The number of neurons in the output layer is equal to the number of selected actions, and each output neuron represents the value of a specific action. The output layer can use a linear activation function to meet the needs of value prediction.

[0139] The weight parameter set of the neural network is W, and the bias parameter set is b.

[0140] The value function can be expressed as: Q(S, A) = θ(S, A),

[0141] Where Q(S, A) is the value of the state-action pair, θ is the neural network, S is the state space parameter, and A is the action to be evaluated.

[0142] The value function constructed with a neural network can express the precise correspondence between the state and the output action value, providing an effective decision-making basis for the deep reinforcement learning process.

[0143] Construct a reward function. At the end of each time step, use the ε-greedy strategy to select and implement actions based on the value function, and calculate the reward value at that moment to evaluate the value of the state-action pair in the previous time step. The reward value is calculated as follows:

[0144] Assume that the current time is the end time of the nth time step. In the time window with the current time as the end point, sample the in-situ monitoring reflectivity curve of the epitaxial wafer center point to obtain a set of sampling points C1, C2, C3, ..., and calculate the standard deviation σc of all sampling points n The in-situ monitoring reflectivity curve near the edge of the epitaxial wafer is sampled to obtain a set of sampling points E1, E2, E3, ..., and the standard deviation σe of all sampling points is calculated n The sampling rate and time window of the epitaxial wafer center point and the point close to the edge are the same, the sampling rate is 10ms-10s, and the time window is 1min-30min.

[0145] Assume that the current time is the end time of the nth time step. For the sampled C1, C2, C3, ... and E1, E2, E3, ..., calculate C1-E1, C2-E2, C3-E3, ... to obtain a series. The root mean square of the series is calculated as U n .

[0146] According to the above calculation parameters, the reward function of the n-1 time step action is defined as:

[0147] R n-1 =α·(σc n +σe n -σc n-1 -σe n-1 )-β·(U n -U n-1 ),

[0148] Where (σc n +σe n -σc n-1 -σe n-1 ) can reflect the impact of the action in the previous time step on the standard deviation of the reflectivity curve of the AlN template growth, that is, the impact of the reflectivity oscillation intensity, which is used to evaluate the crystal quality. The larger the standard deviation, the stronger the oscillation, the higher the quality of the thin film crystal, and the greater the reward value of the action in the previous time step.

[0149] U n -U n-1 It can reflect the impact of the action in the previous time step on the difference between the reflectivity curves at the center and near the edge, U n -U n-1The larger it is, the greater the difference between the reflectivity curves at the center and near the edge, the worse the uniformity of the AlN template, and the lower the reward value at the previous time step.

[0150] α, β are weight coefficients used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 ) to the reward. This reward function starts with the in-situ monitoring reflectivity curves at the center and near the edge, while taking into account the growth quality and uniformity of the AlN template, which can comprehensively improve the process technology of aluminum nitride templates from multiple dimensions.

[0151] In order to enhance the stability of model training, two identical neural network value functions θ and θ' are established, where θ is called the action selection network and θ' is called the Q value calculation network. The Q value is updated using temporal difference (TD): Q target (S n-1 , A n-1 )=R n-1 +γQ(S n ,maxQ(S n ,A n ; θ);θ'),

[0152] Where Q target (S n-1 , A n-1 ) is the updated value of the previous state-action pair, R n-1 is the state-action pair reward calculated by the reward function, γQ(S n ,maxQ(S n ,A n ; θ);θ') represents the use of θ network to select action A according to the greedy strategy n , and use the θ' network to calculate S n ,A n The Q value of the state-action pair, γ is the decay factor.

[0153] The obtained Q target (S n-1 , A n-1 ) is brought into the θ network, and the network parameters W and b are updated by the back-propagation gradient descent method. At regular intervals, the parameters of the θ action selection network are copied to the θ'Q value calculation network, thereby stabilizing the Q value update and improving the stability of the model.

[0154] The duration of each round of training is a fixed-time epitaxial process. If training is required again, the substrate must be replaced and the growth process restarted.

[0155] The model effectiveness verification is specifically to use the trained neural network θ to grow aluminum nitride templates. Each time step in the growth process will accurately give the growth parameter optimization action, and the growth process will end when the predetermined time is reached. Finally, the prepared aluminum nitride template is characterized, the quality and uniformity of the crystal are evaluated, and the optimization effect of the deep reinforcement learning model in the aluminum nitride template growth process parameters is verified.

[0156] In the claims, specification and drawings of the present invention, the term "multiple" refers to two or more than two. Unless otherwise clearly defined, the orientation or positional relationship indicated by the terms "upper" and "lower" is based on the orientation or positional relationship shown in the drawings, which is only for the purpose of more conveniently describing the present invention and making the description process simpler, rather than indicating or implying that the device or element referred to must have the specific orientation described, be constructed and operated in a specific orientation, so these descriptions cannot be understood as limiting the present invention; the terms "connect", "install", "fix" and the like should be understood in a broad sense. For example, "connection" can be a fixed connection between multiple objects, or a detachable connection between multiple objects, or an integral connection; it can be a direct connection between multiple objects, or an indirect connection between multiple objects through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood based on the specific circumstances of the above data.

[0157] In the claims, specification and drawings of the present invention, the description of the terms "one embodiment", "some embodiments", "specific embodiments" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In the claims, specification and drawings of the present invention, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing aluminum nitride template process parameters based on deep reinforcement learning, characterized in that: include: Setting one or more process parameters of the aluminum nitride template; Based on the process parameters for preparing the aluminum nitride template, defining a state space and an action space; Construct the neural network value function Q(S, A) = θ(S, A), Wherein, Q(S, A) is the value of the state-action pair, θ is a neural network, S is a state space parameter, A is an action to be evaluated, wherein the input layer of the neural network takes the state space as input, the output layer of the neural network corresponds to the value of each action in the action space, the weight parameter set of the neural network is W, and the bias parameter set is b; At the end of each time step, based on the value of the state-action pair calculated by the neural network value function Q(S, A), an ε-greedy strategy is used to select and implement actions to change the corresponding process parameters; Constructing a reward function for calculating, at the end of the nth time step, a reward value corresponding to the action performed at the end of the n-1th time step, wherein the reward function is related to the reflectivity curve of the aluminum nitride template growth, and n is an integer and n≥2; Within a predetermined time period, at the end of each time step, actions are taken to change process parameters to grow the aluminum nitride template, and the neural network θ is trained according to the value function and the reward function to form a new neural network value function Q(S, A).

2. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: The neural network is a fully connected layer neural network.

3. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: At the end of n-1 time steps, perform action A n-1 The reward function is defined as: R n-1 =α·(σc n +you n -σc n-1 -you n-1 )-β·(U n -U n-1 ), Among them, R n-1 is the reward value, α and β are weight coefficients, used to balance (σc n +σe n -σc n-1 -σe n-1 ) and (U n -U n-1 )’s contribution to the reward, where n is an integer and n≥2; σc n represents the standard deviation of a plurality of first sampling values ​​on the in-situ monitoring reflectivity curve of the center point of the aluminum nitride epitaxial wafer within the time window with this moment as the end point at the end of the nth time step; σc n-1 represents the standard deviation of a plurality of first sampling values ​​on the in-situ monitoring reflectivity curve of the center point of the aluminum nitride epitaxial wafer within the time window with this moment as the end point at the end of the n-1th time step; σe n represents the standard deviation of a plurality of second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the nth time step; σe n-1 represents the standard deviation of a plurality of second sampling values ​​on the in-situ monitoring reflectivity curve at the edge of the aluminum nitride epitaxial wafer within the time window ending at this moment at the end of the n-1th time step; U n is the root mean square. At the end of the nth time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n , U n-1 is the root mean square. At the end of the n-1th time step, within the time window with this moment as the end point, calculate the difference between the first sampling value and the second sampling value at the same sampling moment to form a series. Calculate the root mean square of the series to get U n-1 .

4. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 3, characterized in that: Training the neural network θ according to the neural network value function and the reward function includes: Establishing a second neural network θ' which is identical to the neural network θ; The Q value is updated using the timing difference through the following formula: Q target (S n-1 ,A n-1 )=R n-1 +γQ(S n ,maxQ(S n ,A n ;θ);θ’), Where Q target (S n-1 , A n-1 ) is the updated value of the corresponding state-action pair at the end of the n-1th time step, R n-1 is the reward value for taking the action at the end of the n-1th time step, γQ(S n ,maxQ(S n , A n ; θ);θ') represents the use of neural network θ to select action A according to the greedy strategy n , and use the second neural network θ' to calculate (S n , A n ) is the Q value of the state-action pair, γ is the attenuation factor, S n-1 represents the state parameters of the state space at the end of the n-1th time step, S n represents the state parameters of the state space at the end of the nth time step, A n-1 represents the action performed at the end of the n-1th time step, A n represents the action implemented at the end of the nth time step; The obtained Q target (S n-1 , A n-1 ) is brought into the neural network θ as the target value, and the weight parameter set W and the bias parameter set b are updated by the back-propagation gradient descent method to achieve the update of the neural network θ; and At predetermined intervals, copy the updated neural network θ' to the second neural network θ' to update the second neural network θ'; and Repeat the above process within one training period.

5. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 4, characterized in that: Also includes: The substrate is replaced and the growth of the aluminum nitride template is restarted, restarting for another training period.

6. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: Also includes: The aluminum nitride template is grown based on the new neural network θ, the grown aluminum nitride template is characterized, and the new neural network θ is verified.

7. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: The state space parameters include at least a plurality of the following parameters: Temperature, pressure, gas flow, epitaxial wafer rotation speed, in-situ monitoring reflectivity data, aluminum nitride epitaxial layer thickness, aluminum nitride epitaxial layer stress, growth rate, aluminum nitride surface roughness, and growth time.

8. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: The action space includes at least a plurality of the following actions: Increase, keep unchanged or decrease the following growth process parameters: temperature, pressure, H2 carrier gas flow rate, TMAl flow rate, NH3 flow rate, epitaxial wafer rotation speed.

9. The method for optimizing aluminum nitride template process parameters based on deep reinforcement learning according to claim 1, characterized in that: The neural network comprises: An input layer, wherein the input includes state parameters defined in the state space, and the number of the plurality of state parameters in the state space is the same as the number of the plurality of first neurons in the input layer; One or more hidden layers, used to extract state parameter features, wherein the hidden layer uses a ReLU activation function; The output layer outputs the value of the action defined in the action space. The number of the plurality of second neurons in the output layer is the same as the number of actions in the action space. One second neuron represents the value of one action. The output layer adopts a linear activation function.

10. A processing device for aluminum nitride template process parameters based on deep reinforcement learning, characterized in that: include: A setting unit, used to set one or more process parameters of the aluminum nitride template; A definition unit, used for defining a state space and an action space based on process parameters for preparing the aluminum nitride template; A first construction unit is used to construct a neural network value function Q(S, A)=θ(S, A), wherein Q(S, A) is the value of a state-action pair, θ is a neural network, S is a state space parameter, and A is an action to be evaluated, wherein an input layer of the neural network takes the state space as input, an output layer of the neural network corresponds to the value of each action in the action space, a weight parameter set of the neural network is W, and a bias parameter set is b; A parameter changing unit, for selecting and implementing actions using an ε-greedy strategy at the end of each time step based on the value of the state-action pair calculated by the neural network value function Q(S, A) to change the corresponding process parameters; A second construction unit is used to construct a reward function, and at the end of the nth time step, calculate a reward value corresponding to the action performed at the end of the n-1th time step, wherein the reward function is related to the reflectivity curve of the aluminum nitride template growth, and n is an integer and n≥2; A training unit is used to implement actions to change process parameters at the end of each time step within a predetermined time period to grow the aluminum nitride template, and train the neural network θ according to the value function and the reward function to form a new Q(S, A) neural network value function.

11. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or the instruction is executed by the processor, the steps of the method for optimizing the process parameters of the aluminum nitride template based on deep reinforcement learning as described in any one of claims 1 to 9 are implemented.

Images