Compact model parameter extraction method based on DQN algorithm

Abstract

The invention discloses a compact model parameter extraction method based on a DQN algorithm, and the method comprises the steps: firstly collecting a current-voltage characteristic curve and a capacitance-voltage characteristic curve of a device, and determining a BSIM model parameter needing to be extracted; secondly, simulating the initial BSIM model by using Hspice, regarding parameter extraction of the BSIM model as a Markov decision problem, combining with a deep Q network, constructing and training a NoisyNet-DQN model, updating network parameters through a gradient descent method, and improving the precision of the NoisyNet-DQN model through continuous iteration; and finally, verifying the effectiveness of the model through a parameter result after Hspice simulation training is completed. According to the compact model parameter extraction method, the accuracy of automatic parameter extraction is effectively improved, and the problems of high dependence on experience, low efficiency and the like in the traditional manual extraction process are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit computer-aided design, and in particular to a compact model parameter extraction method based on a DQN algorithm. Background Art

[0002] A compact model is a simplified mathematical model used to describe the behavior of semiconductor devices. It is widely used in integrated circuit simulation. It is based on physical analytical equations and empirical knowledge to improve simulation efficiency and reduce computational complexity. SPICE model (Simulation Program with Integrated Circuit Emphasis) is the most important compact model in integrated circuit design, among which BSIM (Berkeley Short-channel IGFET Model) is usually used to simulate the charge and current behavior of devices such as MOSFET. In order to make the model accurately match the characteristics of a specific device, it must go through the parameter extraction step, that is, the process of calibrating the model parameters with a set of measurement data. By adjusting the model parameters, the simulation results are consistent with the actual measurement data to achieve a good fitting effect. This process directly affects the accuracy and reliability of the simulation, thereby determining the overall effect of the circuit simulation.

[0003] Traditional parameter extraction relies on manual adjustment of model parameters. Engineers manually adjust parameters to fit measured data. This process is time-consuming, labor-intensive, inefficient, and highly dependent on experience. With the development of semiconductor technology, non-ideal effects in devices are becoming more and more complex. Traditional manual adjustment methods are difficult to meet the needs of the rapid development of new devices, especially in the design of integrated circuits with high complexity and high precision requirements.

[0004] In recent years, deep learning has been used to automate the parameter extraction process, improving the extraction accuracy and efficiency through automated processes. However, deep learning methods require a large amount of high-quality data, and the "black box" nature of the model makes the extracted parameters lack a clear physical explanation, increasing the uncertainty of the application. In addition, existing research is mainly based on simulation data rather than actual measurement data, which may lead to a gap between the model fitting results and the actual situation. Applying deep learning models to industrial standards and integrating them into existing circuit simulation tools still faces great challenges.

[0005] Deep Q-Network (DQN) is an algorithm in deep reinforcement learning, combining Q-Learning and deep learning. The core idea of ​​DQN is to use deep neural networks to approximate the value function (Q value) of state-action pairs, so that the optimal strategy can be effectively learned in an environment with a very large state space. As a powerful reinforcement learning algorithm, it has shown great potential in tasks that require intelligent decision-making and automatic control. With the continuous improvement of the algorithm, the DQN algorithm can be applied to optimize model parameters.

[0006] Chinese invention patent application CN118114744A discloses a method for automatically extracting high-dimensional parameters of device models based on reinforcement learning. The method collects the target electrical characteristic curve of the BSIM model and builds a simulation environment in combination with process design parameters. Then, the device compact model (DCM) parameter extraction problem is described as a Markov decision process using an advanced reinforcement learning algorithm. High-dimensional device model parameter extraction is performed in a continuous parameter search space. The device model knowledge is combined in the reinforcement learning algorithm framework to further improve the output accuracy of the device model, and the electrical characteristic curve is fitted through simulation. This method can realize the automatic extraction of high-dimensional parameters of the device model, but the reinforcement learning algorithm described in the invention directly uses the proximal policy optimization algorithm (PPO) and does not optimize and improve the algorithm. The convergence effect is poor in actual use. Summary of the invention

[0007] The problem to be solved by the present invention is to provide a compact model parameter extraction method based on the DQN algorithm, which is used to improve the accuracy of automated parameter extraction, solve the problems of high dependence on experience and low efficiency in the traditional manual extraction process, and also improve the defects of deep learning models in the physical sense.

[0008] The present invention adopts the following technical solution: a compact model parameter extraction method based on the DQN algorithm, the parameter extraction of the compact model includes the following steps:

[0009] Step 1: Collect the current-voltage characteristic and capacitance-voltage characteristic curves of the device, construct a data set, and preprocess the data set;

[0010] Step 2: Determine the BSIM model parameters that need to be extracted and set a reasonable value range for the parameters;

[0011] Step 3: Use Hspice to simulate the initial BSIM model and output the current-voltage curve and capacitance-voltage curve;

[0012] Step 4: The parameter extraction of the BSIM model is regarded as a Markov decision problem and combined with a deep Q network: the agent selects action a according to the current state s with the ε-greedy strategy, executes action a in the environment to update the BSIM model parameters, calculates the difference between the curves before and after the update based on the interaction with the environment, obtains the reward r through the reward function, and the agent changes to the next state s'. The reward discount factor γ is introduced to discount the long-term reward, and the four-tuple data sampled from the environment each time is stored in the experience replay pool;

[0013] Step 5: Based on the experience replay pool data, build and train the NoisyNet-DQN model: use the NoisyNets Q value network to calculate the predicted Q value under the current state s, and calculate the target Q value through the target network, calculate the loss function between the predicted Q value and the target Q value, and update the parameters of the NoisyNets Q value network through the gradient descent method; update the parameters of the target network through the parameters of the NoisyNets Q value network at regular intervals, and improve the accuracy of the NoisyNet-DQN model through continuous iteration;

[0014] The effectiveness of the NoisyNet-DQN model is verified by the parameter results after Hspice simulation training. The parameters extracted from the simulation are used to describe the electrical characteristics of MOSFET, help predict the performance of the circuit, and perform circuit simulation and optimization.

[0015] Preferably, in step 1, the device is a MOS device, the MOS device is simulated by TCAD, the initial BSIM model of the MOS device is extracted, the current-voltage and capacitance-voltage data are collected by a numerical analysis method, the curve range and step size are set, and the actual measurement data of the current-voltage and capacitance-voltage of the MOS device are combined to form a data set, and the data of the data set is normalized before training.

[0016] Preferably, in step 2, based on the influence of different parameters on the simulation results, the BSIM model parameters are determined, including: acde, moin, noff and voffcv, which are used to fit the actual measurement data with the Hspice simulation data, and corresponding upper and lower limits are set for the parameters.

[0017] Preferably, in step 4, the state space is the BSIM model parameters to be extracted determined in step 2, and the action is selected by the ε greedy strategy, and each parameter action includes three types: increase, decrease and unchanged, and the environment is the result of Hspice simulation of the BSIM model;

[0018] The ε-greedy strategy is:

[0019]

[0020] Where ω is the Q network parameter, ε is the random exploration probability, and a = argmax a Q ω (s,a), is the action with the largest Q value, which is randomly explored with a probability of ε; adding noise in the execution strategy link makes ε decay over time, and the probability of exploration will continue to decrease.

[0021] Furthermore, in step 4, the reward function is defined according to the difference between the actual curve and the simulation curve, as shown below:

[0022]

[0023]

[0024] The difference between the target curve and the fitted curve is measured by the root mean square error. new_rms represents the new root mean square error value calculated after taking the action, old_rms is the old root mean square error value after the previous action is executed, and Tar i Indicates the point value on the target curve, Rrf i Represents the point value on the training curve.

[0025] Preferably, the four-tuple data includes: state, action, reward, and next state; the experience replay pool adopts a dual experience pool, the main experience pool is used to store regular data, and the secondary experience pool is used to store important data; in the training stage, regular data is selected from the main experience pool, and part of the important data is selected from the secondary experience pool to balance the ratio of regular data to important data.

[0026] Preferably, in step 5, a NoisyNet-DQN model is constructed, NoisyNets is a noise function, noise is introduced into the weights and biases of the network, and the noise is parameterized to obtain a noise injection model of the weights, which is expressed as follows:

[0027] θ=μ+Σβ

[0028] Where θ represents the network parameter after adding noise, μ represents the learnable mean parameter, Σ represents the learnable noise scale, ζ = (μ, Σ), β represents the zero-mean noise variable, and ⊙ represents element-wise multiplication;

[0029] Gaussian noise is added to the parameter space to transform the linear layer parameters of the deep Q network into a noise network. By adding noise to the linear layer parameters of the deep Q network, a change in weight can affect the strategy across multiple time steps, and the NoisyNets Q value network is obtained. The linear layer is defined as follows:

[0030] y=(μ ω +σ ω β ω)x+μ b +σ b ⊙β b

[0031] Among them, μ ω and μ b is the average of the weights and biases, σ ω and σ b is the noise amplitude of the weights and biases, β ω and β b is independent Gaussian noise.

[0032] Furthermore, when there is enough data in the experience replay pool, some data are randomly sampled to train the NoisyNet-DQN model. Before the training starts, the NoisyNets Q value network and the target network are initialized with random parameters. The experience replay pool is initialized and hyperparameters are defined, including: experience replay pool capacity, random exploration probability ε, and reward discount factor γ. The target network is the NoisyNets Q value network updated at fixed step sizes, so that the target network is more stable than the training network.

[0033] Preferably, the loss function formula is as follows:

[0034] Loss(ζ)=E[Loss(ω)];

[0035] Loss(ω)=E (s,a,r,s')～D [r+γmax a' Q ω' (s',a')-Q ω (s,a)) 2 ];

[0036] Loss(ζ)=E[E (s,a,r,s')～D [r+γmax a' Q θ' (s',a';ζ')-Q θ (s,a;ζ)] 2 ];

[0037] Among them, Loss(ω) is the loss function of the deep Q network, ω is the parameter of the deep Q network, ω' is the parameter of the target network, Loss(ζ) is the loss function of the NoisyNet-DQN network after adding noise, E and E (s,a,r,s')～D They represent the expectation of the whole and the expectation of the parameters obtained from the experience replay pool respectively;

[0038] θ represents the network parameters of the NoisyNets Q-value network, θ' represents the network parameters of the target network after adding noise, ζ is the learnable noise parameter in the network after adding noise, D is the sample distribution in the experience replay pool, Q ω(s,a) and Q θ (s, a; ζ) are the predicted Q values ​​before and after adding noise, max a' Q ω' (s',a') and max a' Q θ' (s', a'; ζ') are the target Q values ​​for taking the optimal action before and after adding the noise network, r+γmax a' Q θ' (s', a'; ζ') is the time difference result calculated by the target Q value. The update goal is to make Q θ (s,a;ζ) approximates r+γmax a' Q θ' (s',a';ζ').

[0039] Furthermore, the parameters of the NoisyNets Q value network are updated by the gradient descent method. There is a set of learnable parameter vectors ζ in the network parameters θ after noise addition. During the training process, it is optimized by gradient descent, and the gradient is calculated by Monte Carlo approximation. The formula is as follows:

[0040]

[0041] in, Indicates the gradient of the loss function Loss It means to find the gradient of the loss function Loss under the parameters μ and Σ; the core of the Monte Carlo approximation calculation is to determine the approximate value function and use gradient descent to minimize the loss function to continuously debug the network weights.

[0042] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0043] The compact model parameter extraction method based on the DQN algorithm in the present invention effectively improves the accuracy of automated parameter extraction, solves the problems of high reliance on experience and low efficiency in the traditional manual extraction process, and also improves the defects of the deep learning model that are unclear in the physical sense. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a compact model parameter extraction method based on the DQN algorithm of the present invention;

[0045] Figure 2 This is a graph showing the training reward value results of an embodiment of the present invention;

[0046] Figure 3 This is the training parameter fitting result of the embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the application is further elaborated in detail below in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments of other researchers in the field on this embodiment belong to the protection scope of the present invention. At the same time, for the step numbering in the embodiment of the present invention, it is only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] The present invention provides a compact model parameter extraction method based on the DQN algorithm, such as Figure 1 As shown, the compact model parameter extraction includes the following steps:

[0049] Step 1: Collect the current-voltage characteristic and capacitance-voltage characteristic curves of the device, construct a data set, and preprocess the data set;

[0050] Step 2: Determine the BSIM model parameters that need to be extracted and set a reasonable value range for the parameters;

[0051] Step 3: Use Hspice to simulate the initial BSIM model and output the current-voltage curve and capacitance-voltage curve;

[0052] Step 4: The parameter extraction of the BSIM model is regarded as a Markov decision problem and combined with a deep Q network: the agent selects action a according to the current state s with the ε-greedy strategy, executes action a in the environment to update the BSIM model parameters, calculates the difference between the curves before and after the update based on the interaction with the environment, obtains the reward r through the reward function, and the agent changes to the next state s'. The reward discount factor γ is introduced to discount the long-term reward, and the four-tuple data sampled from the environment each time is stored in the experience replay pool;

[0053] Step 5: Based on the experience replay pool data, build and train the NoisyNet-DQN model: use the NoisyNets Q value network to calculate the predicted Q value under the current state s, and calculate the target Q value through the target network, calculate the loss function between the predicted Q value and the target Q value, and update the parameters of the NoisyNets Q value network through the gradient descent method; update the parameters of the target network through the parameters of the NoisyNets Q value network at regular intervals, and improve the accuracy of the NoisyNet-DQN model through continuous iteration;

[0054] Step 6: Verify the effectiveness of the NoisyNet-DQN model through the parameter results after Hspice simulation training. Use the parameters extracted from the simulation to describe the electrical characteristics of the MOSFET, helping engineers accurately predict the performance of the circuit and perform circuit simulation and optimization.

[0055] In one embodiment of the present invention, a compact model parameter extraction method based on the DQN algorithm is as follows:

[0056] Step 1: Simulate the SOI MOS device through TCAD, manually extract the initial BSIM4 baseline model, and collect current-voltage (IV) and capacitance-voltage (CV) data using numerical analysis methods.

[0057] The current-voltage curve range is set to 0-1.5V with a step size of 0.0375V; the capacitance-voltage curve range is set to 0-2V with a step size of 0.05V; in addition, the actual measurement data of IV and CV together constitute the original data set.

[0058] Perform data preprocessing and normalize the original data set before training, otherwise it may cause gradient explosion.

[0059] Step 2: Based on the experience of manual extraction and considering that different parameters have different degrees of influence on the simulation results, this embodiment selects parameters in the BSIM4 model, including: acde, moin, noff and voffcv, and sets corresponding upper and lower limits for these parameters.

[0060] These parameters are charge and capacitance parameters in the BSIM official manual and are used to fit actual measurement data with Hspice simulation data.

[0061] Step 4: The state space is the BSIM model parameters selected above. The action is selected using the ε greedy strategy. Each parameter action includes increase, decrease and unchanged. The environment is the result of Hspice simulation of the BSIM model. The reward is defined according to the difference between the actual curve and the simulation curve.

[0062] In this embodiment, the ε greedy strategy is:

[0063]

[0064] Where ω is the Q network parameter, a = argmax a Q ω (s,a) is the action with the largest Q value, which is randomly explored with a probability of ε. This is equivalent to adding a certain amount of noise in the execution of the strategy, so that the model has a certain exploration ability and can make ε decay over time, then the probability of exploration will continue to decrease;

[0065] The reward function of this embodiment is:

[0066]

[0067]

[0068] Among them, the difference between the target curve and the fitting curve is measured by the root mean square error (rms), new_rms represents the new rms value calculated after taking the action, old_rms represents the old rms value after the previous action is performed, Tari represents the point value on the target curve, and Rrfi represents the point value on the training curve.

[0069] In particular, in this embodiment, the experience replay pool adopts a dual experience pool. This method optimizes the data selection mechanism by setting two experience pools, balancing the ratio of regular data and important data; the main experience pool is mainly used to store most of the training data, and the secondary experience pool is specifically used to store important and rare data, which can often provide higher rewards or help the model learn quickly in model training; in the training stage, data is mainly selected from the main experience pool, and a small amount of data is selected from the secondary experience pool, which increases the probability of important data being selected, thereby better maintaining its reward and increasing the average cumulative reward.

[0070] The result of the experience replay pool training reward value in this embodiment is as follows Figure 2 shown.

[0071] Step 5: Before training begins, you need to initialize the NoisyNets Q value network and target network with random parameters, initialize the experience replay pool and define hyperparameters: experience replay pool capacity, random exploration probability ε, reward discount factor γ, etc.

[0072] The noise function NoisyNets is a neural network that drives exploration by introducing noise into the weights and biases of the network. Parameterizing the noise can yield a noise injection model for the weights:

[0073] θ=μ+∑⊙β

[0074] Among them, θ represents the network parameters after noise addition, μ is the learnable mean parameter, Σ is the learnable noise scale, ζ = (μ, Σ), β is the zero-mean noise variable, and ⊙ represents element-wise multiplication.

[0075] The NoisyNets Q-value network refers to adding Gaussian noise in the parameter space. In this embodiment, the linear layer of the value function Q network is parameterized as a noise network. By adding noise to the deep Q network parameters, a change in a weight can affect the strategy across multiple time steps, thereby obtaining a NoisyNets Q-value network, thereby stimulating complex state-dependent behavior changes and improving exploration efficiency.

[0076] The linear layer definition:

[0077] y=(μ ω +σ ω ⊙β ω )x+μ b +σ b ⊙β b

[0078] Among them, μ ω and μ b is the average of the weights and biases, σ ω and σ b is the noise amplitude of the weights and biases, β ω and β b is independent Gaussian noise;

[0079] In this embodiment, the loss function formula is as follows:

[0080] Loss(ζ)=E[Loss(ω)];

[0081] Loss(ω)=E (s,a,r,s')～D [r+γmax a' Q ω' (s',a')-Q ω (s,a)) 2 ];

[0082] Loss(ζ)=E[E (s,a,r,s')～D [r+γmax a' Q θ' (s',a';ζ')-Q θ (s,a;ζ)] 2 ];

[0083] Among them, Loss(ω) is the loss function of DQN, ω is the parameter of the neural network, ω' is the parameter of the target network, Loss(ζ) is the loss function of Noisy-DQN after adding noise, θ represents the network parameter after adding noise, ζ is the learnable noise parameter in the neural network, D is the sample distribution in the experience replay pool, Q θ (s,a;ζ) is the predicted Q value, r+γmax a' Q θ'(s', a'; ζ') is the time difference (TD) result calculated from the target Q value. The final update goal of the algorithm is to make Q θ (s,a;ζ) approximates r+γmax a' Q θ' (s',a';ζ').

[0084] Furthermore, there is a set of learnable parameter vectors ζ in the noise vector θ, which can be optimized by gradient descent during training, where the gradient is calculated by Monte Carlo approximation:

[0085]

[0086] in, Indicates the gradient of the loss function Loss It means to find the gradient of the loss function Loss under the parameters μ and Σ.

[0087] Therefore, the core of Monte Carlo approximate calculation is to determine the approximate value function and use gradient descent to minimize the loss function to continuously debug the network weights.

[0088] The training parameter fitting results of this embodiment are as follows: Figure 3 As shown: the left is the output current-voltage IV curve, and the right is the output capacitance-voltage CV curve.

[0089] It can be seen that the compact model parameter extraction method of the present invention effectively improves the accuracy of automated parameter extraction, solves the problems of high dependence on experience and low efficiency in the traditional manual extraction process, and also improves the defects of deep learning models that are unclear in the physical sense.

[0090] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A compact model parameter extraction method based on DQN algorithm, characterized in that: Parameter extraction of the compact model consists of the following steps: Step 1: Collect the current-voltage characteristic and capacitance-voltage characteristic curves of the device, construct a data set, and preprocess the data set; Step 2: Determine the BSIM model parameters that need to be extracted and set a reasonable value range for the parameters; Step 3: Use Hspice to simulate the initial BSIM model and output the current-voltage curve and capacitance-voltage curve; Step 4: The parameter extraction of the BSIM model is regarded as a Markov decision problem and combined with a deep Q network: the agent selects action a according to the current state s with the ε-greedy strategy, executes action a in the environment to update the BSIM model parameters, calculates the difference between the curves before and after the update based on the interaction with the environment, obtains the reward r through the reward function, and the agent changes to the next state s'. The reward discount factor γ is introduced to discount the long-term reward, and the four-tuple data sampled from the environment each time is stored in the experience replay pool; Step 5: Based on the experience replay pool data, build and train the NoisyNet-DQN model: use the NoisyNets Q value network to calculate the predicted Q value under the current state s, and calculate the target Q value through the target network, calculate the loss function between the predicted Q value and the target Q value, and update the parameters of the NoisyNets Q value network through the gradient descent method; update the parameters of the target network through the parameters of the NoisyNets Q value network at regular intervals, and improve the accuracy of the NoisyNet-DQN model through continuous iteration; Step 6: Verify the effectiveness of the NoisyNet-DQN model through the parameter results after Hspice simulation training. Use the parameters extracted from the simulation to describe the electrical characteristics of the MOSFET, help predict the performance of the circuit, and perform circuit simulation and optimization.

2. The compact model parameter extraction method based on the DQN algorithm according to claim 1, characterized in that: In step 1, the device is a MOS device, and the MOS device is simulated by TCAD, the initial BSIM model of the MOS device is extracted, the current-voltage and capacitance-voltage data are collected by numerical analysis method, the curve range and step size are set, and the actual measurement data of the current-voltage and capacitance-voltage of the MOS device are combined to form a data set, and the data of the data set is normalized before training.

3. The compact model parameter extraction method based on the DQN algorithm according to claim 2 is characterized in that: In step 2, based on the impact of different parameters on the simulation results, the BSIM model parameters are determined, including acde, moin, noff, and voffcv, which are used to fit the actual measurement data with the Hspice simulation data, and the corresponding upper and lower limits are set for the parameters.

4. The compact model parameter extraction method based on the DQN algorithm according to claim 2 is characterized in that: In step 4, the state space is the BSIM model parameters to be extracted determined in step 2, and the action is selected by the ε greedy strategy. Each parameter action includes three types: increase, decrease, and unchanged. The environment is the result of Hspice simulation of the BSIM model. The ε-greedy strategy is: Where ω is the Q network parameter, ε is the random exploration probability, and a = argmax a Q ω (s,a), is the action with the largest Q value, which is randomly explored with a probability of ε; adding noise in the execution strategy link makes ε decay over time, and the probability of exploration will continue to decrease.

5. The compact model parameter extraction method based on the DQN algorithm according to claim 4 is characterized in that: In step 4, the reward function is defined according to the difference between the actual curve and the simulation curve, which is expressed as follows: The difference between the target curve and the fitted curve is measured by the root mean square error. new_rms represents the new root mean square error value calculated after taking the action, old_rms is the old root mean square error value after the previous action is executed, and Tar i Indicates the point value on the target curve, Rrf i Represents the point value on the training curve.

6. The compact model parameter extraction method based on the DQN algorithm according to claim 4 is characterized in that: The four-tuple data includes: state, action, reward, and next state; the experience replay pool adopts a dual experience pool, the main experience pool is used to store regular data, and the secondary experience pool is used to store important data; in the training stage, regular data is selected from the main experience pool, and some important data is selected from the secondary experience pool to balance the ratio of regular data to important data.

7. The compact model parameter extraction method based on the DQN algorithm according to claim 6 is characterized in that: In step 5, the NoisyNet-DQN model is constructed. NoisyNets is a noise function that introduces noise into the weights and biases of the network. The noise is parameterized to obtain a noise injection model of the weights, which is expressed as follows: θ=μ+∑⊙β Where θ represents the network parameter after adding noise, μ represents the learnable mean parameter, Σ represents the learnable noise scale, ζ = (μ, Σ), β represents the zero-mean noise variable, and ⊙ represents element-wise multiplication; Gaussian noise is added to the parameter space to transform the linear layer parameters of the deep Q network into a noise network. By adding noise to the linear layer parameters of the deep Q network, a change in weight can affect the strategy across multiple time steps, and the NoisyNets Q value network is obtained. The linear layer is defined as follows: y=(μ ω +s ω ⊙β ω )x+μ b +s b ⊙β b Among them, μ ω and μ b is the average of the weights and biases, σ ω and σ b is the noise amplitude of the weights and biases, β ω and β b is independent Gaussian noise.

8. The compact model parameter extraction method based on the DQN algorithm according to claim 7 is characterized in that: In step 5, when there is enough data in the experience replay pool, some data are randomly sampled to train the NoisyNet-DQN model. Before the training starts, the NoisyNets Q value network and the target network are initialized with random parameters. The experience replay pool is initialized and hyperparameters are defined, including: experience replay pool capacity, random exploration probability ε, and reward discount factor γ. The target network is the NoisyNets Q value network updated at fixed step sizes, so that the target network is more stable than the training network.

9. The compact model parameter extraction method based on the DQN algorithm according to claim 7, characterized in that: The loss function formula is as follows: Loss(ζ)=E[Loss(ω)]; Loss(ω)=E (s,a,r,s')～D [r+γmax a' Q ω' (s',a')-Q ω (s,a)) 2 ]; Loss(ζ)=E[E (s,a,r,s')～D [r+γmax a' Q θ' (s',a';ζ')-Q θ (s,a;ζ)] 2 ]; Among them, Loss(ω) is the loss function of the deep Q network, ω is the parameter of the deep Q network, ω' is the parameter of the target network, Loss(ζ) is the loss function of the NoisyNet-DQN network after adding noise, E and E (s,a,r,s')～D They represent the expectation of the whole and the expectation of the parameters obtained from the experience replay pool respectively; θ represents the network parameters of the NoisyNets Q-value network, θ' represents the network parameters of the target network after adding noise, ζ is the learnable noise parameter in the network after adding noise, D is the sample distribution in the experience replay pool, Q ω (s,a) and Q θ (s, a; ζ) are the predicted Q values ​​before and after adding noise, max a' Q ω' (s',a') and max a' Q θ' (s', a'; ζ') are the target Q values ​​for taking the optimal action before and after adding the noise network, r+γmax a' Q θ' (s', a'; ζ') is the time difference result calculated by the target Q value. The update goal is to make Q θ (s,a;ζ) approximates r+γmax a' Q θ' (s',a';ζ').

10. The compact model parameter extraction method based on the DQN algorithm according to claim 7, characterized in that: The parameters of the NoisyNets Q value network are updated by the gradient descent method. There is a set of learnable parameter vectors ζ in the network parameters θ after noise addition. During the training process, it is optimized by gradient descent and the gradient is calculated by Monte Carlo approximation. The formula is as follows: in, Indicates the gradient of the loss function Loss It means to find the gradient of the loss function Loss under the parameters μ and Σ; the core of the Monte Carlo approximation calculation is to determine the approximate value function and use gradient descent to minimize the loss function to continuously debug the network weights.

Images