Dynamic buffer area size adjusting method based on deep reinforcement learning

Abstract

The invention discloses a dynamic buffer size adjusting method based on deep reinforcement learning. The method comprises the following steps: constructing a Duelling DQN model and an experience playback pool, extracting features of a network state through two full connection layers in sequence, carrying out attention weight distribution through a self-attention mechanism, and introducing noise disturbance through a noise layer; an action advantage network is adopted to evaluate actions capable of being adopted by the intelligent agent, a state value network is adopted to evaluate a network state, a Q value of the network is obtained, the actions are executed, and a network environment sample is formed and stored in an experience playback pool based on a feedback reward and the network state at the next moment; extracting a network environment sample to train a Dueling DQN model; according to the method designed by the invention, the size of the buffer area is dynamically adjusted to reduce jitter and delay under different flow loads and network states through a reinforcement learning algorithm, so that the transmission performance of the network is effectively improved, and the service quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of information engineering, and in particular to a dynamic buffer size adjustment method based on deep reinforcement learning. Background Art

[0002] With the development of network technology, traffic management in complex network environments has become increasingly important. Traditional fixed buffer management strategies cannot flexibly cope with the dynamic changes of network traffic, especially in scenarios with high traffic fluctuations, where buffer overflow, network delay, and packet loss become significant problems. In response to these problems, dynamic buffer management strategies have emerged, aiming to adjust the buffer size in real time according to the network status, thereby improving the network's transmission performance and reducing delay and packet loss.

[0003] With the rapid growth of the scale and capacity of the Internet, the traditional bandwidth-delay product (BDP) rule is no longer applicable to the future Internet. At the same time, projects such as the Global Environment for Network Innovation (GENI) and the Future Internet Design (FIND) have promoted the innovation of Internet technology, bringing challenges and opportunities to the buffer management of next-generation routers. The research community has shown increasing interest in the buffer size issue, and a large number of studies have emerged in the past few years. However, these studies are based on certain assumptions about Internet traffic, which may be limited in application in other traffic models and show inconsistent or even contradictory results. For example, existing fixed buffer management methods or rule-based dynamic management methods often have difficulty coping with complex network changes, especially when faced with burst traffic or high bandwidth requirements, and traditional methods have weak adaptability.

[0004] With the development of deep learning (DL) in recent years, deep reinforcement learning (DRL), which is a combination of deep learning and reinforcement learning (RL), has provided new ideas for buffer capacity management. In particular, DQN (Deep QNetwork) has demonstrated its superior performance in complex decision-making scenarios by combining Q-learning with neural networks. However, the use of DQN alone has shortcomings in some complex scenarios. For example, its performance may be limited when facing a continuous state space. In order to further optimize the performance of buffer management, Dueling DQN proposes the idea of ​​modeling state value and action advantage separately. This approach reduces the impact of insignificant actions in most states, while enabling better decisions in critical states.

[0005] Based on this, the present invention constructs a router buffer size dynamic adjustment model through Dueling DQN, and introduces a self-attention mechanism to enable it to make more accurate decisions in a dynamically changing network environment. At the same time, the introduction of a noise layer avoids falling into a local optimal solution and accelerates the convergence process of the algorithm, aiming to dynamically adjust the buffer size to cope with traffic changes in complex networks. Summary of the invention

[0006] Purpose of the invention: In view of the problem that the buffer management strategy in the current network is difficult to adapt to the complex and dynamically changing network traffic, the present invention proposes a dynamic buffer size adjustment method based on deep reinforcement learning. The method aims to dynamically adjust the buffer size to reduce jitter and delay under different traffic loads and network conditions through reinforcement learning algorithms, thereby effectively improving the transmission performance of the network and improving the quality of service (QOS).

[0007] To achieve the above functions, the present invention designs a dynamic buffer size adjustment method based on deep reinforcement learning. For a network with an agent and a buffer, the following steps S1 to S7 are performed to complete the automatic adjustment of the buffer size for the network: Step S1: Initialize the initial value and upper and lower bounds of the buffer; Step S2: Construct a Dueling DQN model, including network state, action and reward function, as well as a state value network and an action advantage network; the network state consists of the characteristics of the buffer, the action is defined as the agent's adjustment strategy for the buffer, and the reward function gives rewards based on the agent's action feedback; the state value network and the action advantage network evaluate the network state and action respectively; Step S3: Construct an experience replay pool to store network environment samples, including the current network state, action, reward, the next network state, and experience data on whether the process is over; Step S4: for the current network state, extract features through two fully connected layers in sequence, assign attention weights to the features extracted by the two fully connected layers through the self-attention mechanism, combine the attention weights with the extracted features, and then introduce noise disturbance through the noise layer; Step S5: Use the action advantage network of the Dueling DQN model to evaluate the actions that the agent can take, use the state value network to evaluate the network state at the current moment, obtain the Q value of the network at the current moment, select and execute actions based on the Q value, and construct a network environment sample based on the current network state, action, reward, next moment network state, and experience data of whether the process is over, and store it in the experience replay pool; Step S6: extract network environment samples from the experience replay pool, use mean square error as the loss function, train the Dueling DQN model through the back propagation algorithm, and obtain the trained Dueling DQN model after iterative training; Step S7: Apply the trained Dueling DQN model to complete automatic adjustment of the network buffer size.

[0008] Beneficial effects: Compared with the prior art, the advantages of the present invention include: The present invention designs a dynamic buffer size adjustment method based on deep reinforcement learning, uses the DuelingDQN architecture to build an intelligent body, and effectively achieves a dual reduction in jitter and delay in the network by introducing a noise layer and a self-attention mechanism, combined with intelligent dynamic adjustment of the buffer capacity, especially in scenarios with high traffic loads. Compared with the traditional static buffer management method, the present invention can more flexibly adapt to changes in network traffic, automatically adjust the buffer size, and improve the overall stability of network transmission. By reducing the average transmission delay and jitter, the present invention not only directly optimizes the network performance, but also indirectly improves the reliability of data transmission, especially in data transmission scenarios requiring high reliability and low latency, ensuring the continuous and stable operation of the network. In addition, the present invention avoids the waste of resources by dynamically managing the use of buffer resources, so that the resource utilization efficiency of the network is improved. The present invention is applicable to a variety of complex dynamic network environments, especially to high-performance network scenarios requiring low latency and low jitter, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart of a method for dynamic buffer size adjustment based on deep reinforcement learning provided according to an embodiment of the present invention; Figure 2 is a framework diagram of automatic adjustment of network buffer size provided according to an embodiment of the present invention; Figure 3 is a neural network structure diagram of the Dueling DQN model provided according to an embodiment of the present invention; Figure 4 is a training flow chart of the Dueling DQN model provided according to an embodiment of the present invention; Figure 5 is a structural diagram of a dumbbell topology provided according to an embodiment of the present invention; Figure 6 is a comparison diagram of jitter results provided according to an embodiment of the present invention; Figure 7 is a result comparison diagram of the delay provided according to an embodiment of the present invention; Figure 8is an average throughput comparison diagram provided according to an embodiment of the present invention; Fig. 9 4 is a comparison chart of packet loss rates provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0011] A dynamic buffer size adjustment method based on deep reinforcement learning provided by an embodiment of the present invention refers to Figure 1 For a network with an agent and a buffer, the following steps S1 to S7 are performed to complete the automatic adjustment of the buffer size for the network; the framework diagram of the automatic adjustment of the buffer size is referred to Figure 2 : Step S1: Initialize the initial value and upper and lower bounds of the buffer; Step S2: Construct a Dueling DQN model, including network state, action and reward function, as well as a state value network and an action advantage network; the network state consists of the characteristics of the buffer, the action is defined as the agent's adjustment strategy for the buffer, and the reward function gives rewards based on the agent's action feedback; the state value network and the action advantage network evaluate the network state and action respectively; The neural network structure diagram of the Dueling DQN model is as follows Figure 3 As shown; the network state, action, and reward function in step S2 are as follows: The network status is defined as follows: ; In the formula, for The network status at any time, Indicates the current buffer size, defined as the actual amount of cache allocated to the router; Indicates the current dequeue rate, which measures the speed at which data packets flow out of the buffer to the link, affecting the timeliness of data transmission; Indicates the current queue delay, which is the average waiting time of packets in the buffer queue. Indicates the number of queued packets in the current queue, which is used to measure the utilization of the current buffer; The action space is defined as follows: ; In the formula, represents the action space, represents the action, which is defined as the action space of the agent as the adjustment strategy for the buffer zone, ,in Indicates increasing the buffer size. Indicates reducing the buffer size, Indicates keeping the buffer size unchanged; Design a reward function based on changes in network performance. The reward value will take into account packet loss rate, latency, and throughput. When the buffer adjustment effectively reduces the packet loss rate or reduces latency, a positive reward will be given; otherwise, a negative reward will be given. The reward feedback guides the agent to continuously optimize the strategy. Define the reward function as follows: ; In the formula, represents the reward function, It is a queue delay reward. is the packet loss rate reward, is the throughput reward; , , They are the weights of queuing delay reward, packet loss rate reward, and throughput reward. Each weight satisfies , , .

[0012] Step S3: Construct an experience replay pool to save network environment samples, including the current network state, action, reward, next network state, and experience data on whether the process has ended. The role of the experience replay pool is to store past experience so that the agent can randomly sample from it for training, thereby improving the stability of training and avoiding overfitting.

[0013] The capacity of the experience replay pool in step S3 is set to N In this embodiment N =5000, when the capacity reaches the upper limit, the oldest network environment samples will be removed; The storage format of the experience replay pool is as follows: ; In the formula, Indicates a sample network environment. express The network status at any time, Indicates action, Indicates reward, Indicates the network status at the next moment. Experience data indicating whether the round is over.

[0014] Step S4: for the current network state, extract features through two fully connected layers in sequence, assign attention weights to the features extracted by the two fully connected layers through the self-attention mechanism, combine the attention weights with the extracted features, and then introduce noise disturbance through the noise layer; In step S4, the self-attention mechanism is used to improve the feature modeling ability, and the noise layer is used to enhance the exploration ability and training stability. The specific steps are as follows: Step S4.1: Input the current network state into the first fully connected layer, assuming that the input network state is , the first fully connected layer maps the input network state to the first hidden layer, and obtains the output of the first hidden layer , and processed using the ReLU activation function, the formula is as follows: ; In the formula, Represents the weight matrix of the first hidden layer, which is used to transform the input network state Perform a linear transformation, The bias vector of the first hidden layer is added to the result of the linear transformation to increase the expressiveness of the model. Its dimension is consistent with the number of neurons in the first hidden layer. Step S4.2: The output of the first hidden layer Input the second fully connected layer to get the output of the second hidden layer : ; In the formula, Represents the weight matrix connecting the first hidden layer to the second hidden layer, which is used to convert the output of the first hidden layer Perform a linear transformation, The bias vector of the second hidden layer is added to the result of the linear transformation to increase the expressiveness of the model; Step S4.3: The output of the second hidden layer is processed by the self-attention layer Dynamic weight allocation is performed. The self-attention mechanism can capture the interdependence of different features in the input network state. The query vector is first calculated , key vector Sum value vector , the calculation formula is as follows: ; In the formula, , , is a trainable weight matrix; Then, using the query vector and key vector The inner product of is used to calculate the correlation between the network state at each historical moment and the network state at the current moment to obtain the weight. The weight calculation formula is as follows: ; In the formula, is the attention weight, is the query vector, is the key matrix, is the dimension of the key vector; the softmax operation ensures that the weight values ​​are normalized so that the sum of the weights of the network state at all historical moments is 1; Step S4.4: Combine the attention weights and input features, and the weighted output is: ; In the formula, is the attention weight, is a value vector, is the weighted output; Step S4.5: The output of the self-attention layer is passed to the noise layer. The noise layer enhances the model's exploration ability by introducing noise disturbance, so that the model can not only rely on historical data for training, but also randomly explore different actions. The calculation formula of the noise layer is as follows: ; In the formula, is the output of the self-attention layer, is the noise floor, is a random perturbation sampled from a noise distribution.

[0015] Step S5: Use the action advantage network of the Dueling DQN model to evaluate the actions that the agent can take, use the state value network to evaluate the network state at the current moment, obtain the Q value of the network at the current moment, select and execute actions based on the Q value, and construct a network environment sample based on the current network state, action, reward, next moment network state, and experience data of whether the process is over, and store it in the experience replay pool; The specific steps of step S5 are as follows: Step S5.1: Evaluate the action using the advantage function of the action advantage network and the value function of the state value network of the Dueling DQN model. The advantage function is used to measure the advantages and disadvantages of an action relative to other actions, while the value function evaluates the value of the current network state. The value function is as follows: ; in, Indicates network status to its corresponding value, Network status the value of The advantage function is as follows: ; in, Indicates network status and actions to its corresponding advantage value, For Action The advantage value of Step S5.2: Calculate the Q value as follows: ; in, In network status Take Action The Q value, Indicates the number of actions, For Action The advantage value of Network status The value of represents all possible actions in the action space, Represents the advantage value of all possible actions in the action space; Step S5.3: Select actions using the Boltzmann Softmax strategy , the Boltzmann Softmax strategy calculates the selection probability of each action through the Q value, so that the action with a higher Q value is more likely to be selected. The calculation formula for the probability of selecting an action is: ; in, is the temperature parameter used to control the balance between exploration and exploitation, Represents all possible actions in the action space; Indicates the current network status All possible actions The expected future cumulative return of all possible actions Q value; Step S5.4: Execute action , and get rewards for feedback And the network status at the next moment , and store the network environment samples in the experience replay pool.

[0016] Step S6: extract network environment samples from the experience replay pool, use the mean square error (MSE) as the loss function, train the Dueling DQN model through the back propagation algorithm, and obtain the trained Dueling DQN model after iterative training; Reference Figure 4 , the specific steps of step S6 are as follows: Step S6.1: Randomly extract a batch of network environment samples from the experience replay pool and calculate the target Q value. The formula is: ; In the formula, represents the target network Q value, Indicates immediate reward, represents the discount factor, represents the advantage value of the action, A value representing the state of the network, is the normalization operation of the action, represents the target network parameters, Indicates the number of actions; is the target network estimated in the network state Next action The advantage function value of represents all possible actions in the action space, i.e., increasing, decreasing, and maintaining the buffer size; Step S6.2: Update the main network parameters by minimizing the mean square error as the loss function. The loss function is as follows: ; in, is the loss function, is the main network parameter; represents the expectation of a mini-batch of data sampled from the experience replay pool, that is, the average loss of each sample, represents the target network Q value, is the Q value of the main network; Step S6.3: Use the soft update strategy to update the network parameters. The soft update strategy can more effectively maintain the balance between the target network and the current network, making the training more stable and gradual, and avoiding large parameter jumps in the model. The formula is as follows: ; in, is the weight of soft update, represents the target network parameters, is the main network parameter; Step S6.4: The Dueling DQN model is trained through multiple iterations to obtain a trained Dueling DQN model, so that the model can eventually accurately predict and dynamically adjust the network buffer size, effectively reducing jitter and latency.

[0017] After training is complete, the Dueling DQN model receives the current network status and outputs the optimal buffer adjustment action in real time. When network traffic changes, the model can quickly adjust the buffer size to avoid jitter and excessive latency.

[0018] Step S7: Apply the trained Dueling DQN model to complete automatic adjustment of the network buffer size.

[0019] The following is an application example for verifying a dynamic buffer size adjustment method based on deep reinforcement learning designed by the present invention: This embodiment adopts a dumbbell topology to simulate the dynamic cache management performance under a high load environment. The structure of the dumbbell topology is as follows: Figure 5 As shown in the figure, there are 50 source nodes on the left and 50 target nodes on the right. All source nodes are connected to the target nodes through a central bottleneck link. The bandwidth of the bottleneck link is set to 10kbps and the delay is 5ms. The initial cache capacity is 5 packets, the upper limit is 20 packets, and the lower limit is set to 2 packets.

[0020] In order to test the processing capability of the system and the effect of algorithm optimization, this embodiment uses a service flow based on the ON / OFF traffic model to simulate three typical network services: fax, video, and voice. In the sending rules of the service flow, 1 to 50 send data to nodes 51 to 100 respectively. The simulation duration is 40 seconds. The voice service starts to be sent during the period of 1-20 seconds, and the video service starts to be sent during the period of 10 to 30 seconds. The fax service starts to be sent during the period of 20-40 seconds.

[0021] The experimental results show that compared with the static cache management algorithm BDP and the dynamic cache capacity management algorithm DQN, the jitter of the present invention under high load is reduced by 40.85% and 15% respectively compared with the static cache management algorithm BDP and the dynamic cache capacity management algorithm DQN. Figure 6 As can be seen, the present invention shows higher stability. In terms of latency, the present invention reduces latency by 20.95% and 10.76% respectively compared with the BDP algorithm and the DQN algorithm when the transmission rate continues to increase. Figure 7 As shown in the figure, it effectively meets the low latency requirement and significantly improves the transmission efficiency of latency-sensitive services. Figure 8 As shown in the figure, the throughputs of the three algorithms are 225.555, 225.342, and 225.542 respectively. The throughput of the present invention is slightly higher than that of the DQN algorithm and slightly lower than that of the BDP algorithm. This is because the cache size is set smaller than that of the BDP algorithm, so the present invention can obtain a more objective throughput under the condition of a lower buffer capacity. Fig. 9 As shown, their values ​​are: 0.87558, 0.875708, 0.879057 respectively. It can be seen that the packet loss rate of the present invention is slightly higher than the other two. This is because the present algorithm makes a trade-off between the packet loss rate and the buffer capacity. Although the packet loss rate is slightly higher than the other two algorithms, this result is acceptable.

[0022] In summary, the technical effects achieved by the method designed in the present invention include: 1. Improve the adaptive adjustment capability of the buffer in the network and reduce the average delay jitter and average queuing delay; 2. Dynamically optimize buffer allocation in complex network environments to adapt to traffic fluctuations; 3. Utilize the learning ability of Dueling DQN model to reduce the need for manual adjustment of network parameters and improve overall network efficiency; 4. Reduce the cost of buffer zone deployment.

[0023] The present invention aims to solve the deficiencies in the existing buffer capacity management methods and provide a solution with intelligent adjustment capability to meet the performance requirements in future highly dynamic network environments.

[0024] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A dynamic buffer size adjustment method based on deep reinforcement learning, characterized in that: For a network with an agent and a buffer, the following steps S1 to S7 are performed to complete automatic adjustment of the buffer size for the network: Step S1: Initialize the initial value and upper and lower bounds of the buffer; Step S2: Construct the Dueling DQN model, including network state, action and reward function, as well as state value network and action advantage network; The network state consists of the characteristics of the buffer, the action is defined as the agent's adjustment strategy for the buffer, and the reward function feedbacks the reward according to the agent's action; the state value network and the action advantage network evaluate the network state and action respectively; Step S3: Construct an experience replay pool to store network environment samples, including the current network state, action, reward, the next network state, and experience data on whether the process is over; Step S4: for the current network state, extract features through two fully connected layers in sequence, assign attention weights to the features extracted by the two fully connected layers through the self-attention mechanism, combine the attention weights with the extracted features, and then introduce noise disturbance through the noise layer; Step S5: Use the action advantage network of the Dueling DQN model to evaluate the actions that the agent can take, use the state value network to evaluate the network state at the current moment, obtain the Q value of the network at the current moment, select and execute actions based on the Q value, and construct a network environment sample based on the current network state, action, reward, next moment network state, and experience data of whether the process is over, and store it in the experience replay pool; Step S6: extract network environment samples from the experience replay pool, use mean square error as the loss function, train the Dueling DQN model through the back propagation algorithm, and obtain the trained Dueling DQN model after iterative training; Step S7: Apply the trained Dueling DQN model to complete automatic adjustment of the network buffer size.

2. According to the method of dynamic buffer size adjustment based on deep reinforcement learning in claim 1, it is characterized in that: The network state, action, and reward function in step S2 are as follows: The network status is defined as follows: ； In the formula, for The network status at any time, Indicates the current buffer size, Indicates the current dequeue rate, Indicates the current queue delay, Indicates the number of queued packets in the current queue; The action space is defined as follows: ； In the formula, represents the action space, Indicates action, ,in Indicates increasing the buffer size. Indicates reducing the buffer size, Indicates keeping the buffer size unchanged; The reward function is defined as follows: ； In the formula, represents the reward function, It is a queue delay reward. is the packet loss rate reward, is the throughput reward; , , They are the weights of queue delay reward, packet loss rate reward, and throughput reward. Each weight satisfies , , .

3. The method for dynamic buffer size adjustment based on deep reinforcement learning according to claim 1, characterized in that: The capacity of the experience replay pool in step S3 is set to N , which is stored in the following format: ； In the formula, Indicates a sample network environment. express The network status at any time, Indicates action, Indicates reward, Indicates the network status at the next moment. Experience data indicating whether the round is over.

4. The method for dynamic buffer size adjustment based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step S4 are as follows: Step S4.1: Input the current network state into the first fully connected layer, assuming that the input network state is , the first fully connected layer maps the input network state to the first hidden layer, and obtains the output of the first hidden layer , and processed using the ReLU activation function, the formula is as follows: ； In the formula, represents the weight matrix of the first hidden layer, represents the bias vector of the first hidden layer; Step S4.2: The output of the first hidden layer Input the second fully connected layer to get the output of the second hidden layer : ； In the formula, represents the weight matrix connecting the first hidden layer to the second hidden layer, represents the bias vector of the second hidden layer; Step S4.3: The output of the second hidden layer is processed by the self-attention layer To perform dynamic weight allocation, first calculate the query vector , key vector Sum value vector , the calculation formula is as follows: ； In the formula, , , is a trainable weight matrix; Then, using the query vector and key vector The inner product of is used to calculate the correlation between the network state at each historical moment and the network state at the current moment to obtain the weight. The weight calculation formula is as follows: ； In the formula, is the attention weight, is the query vector, is the key matrix, is the dimension of the key vector; Step S4.4: Combine the attention weights and input features, and the weighted output is: ； In the formula, is the attention weight, is a value vector, is the weighted output; Step S4.5: Pass the output of the self-attention layer to the noise layer. The calculation formula of the noise layer is as follows: ； In the formula, is the output of the self-attention layer, is the noise floor, is a random perturbation sampled from a noise distribution.

5. The method for dynamic buffer size adjustment based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step S5.1: Evaluate the action using the advantage function of the action advantage network and the value function of the state value network of the Dueling DQN model. The advantage function is used to measure the advantages and disadvantages of an action relative to other actions, while the value function evaluates the value of the current network state. The value function is as follows: ； in, Indicates network status to its corresponding value, Network status the value of The advantage function is as follows: ； in, Indicates network status and actions to its corresponding advantage value, For Action The advantage value of Step S5.2: Calculate the Q value as follows: ； in, In network status Take Action The Q value, Indicates the number of actions, For Action The advantage value of Network status The value of represents all possible actions in the action space, Represents the advantage value of all possible actions in the action space; Step S5.3: Select actions using the Boltzmann Softmax strategy , the calculation formula for the probability of selecting an action is: ； in, is the temperature parameter, represents all possible actions in the action space, Indicates the current network status All possible actions Q value; Step S5.4: Execute action , and get rewards for feedback And the network status at the next moment , and store the network environment samples in the experience replay pool.

6. The method for dynamic buffer size adjustment based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step S6 are as follows: Step S6.1: Randomly extract a batch of network environment samples from the experience replay pool and calculate the target Q value. The formula is: ； In the formula, represents the target network Q value, Indicates immediate reward, represents the discount factor, represents the advantage value of the action, A value representing the state of the network, is the normalization operation of the action, represents the target network parameters, Indicates the number of actions; is the target network estimated in the network state Next action The advantage function value of Represents all possible actions in the action space; Step S6.2: Update the main network parameters by minimizing the mean square error as the loss function. The loss function is as follows: ； in, is the loss function, is the main network parameter; It means to average the loss of each sample. represents the target network Q value, is the Q value of the main network; Step S6.3: Update network parameters using the soft update strategy, the formula is as follows: ； in, is the weight of soft update, represents the target network parameters, is the main network parameter; Step S6.4: Train the Dueling DQN model through multiple iterations to obtain a trained Dueling DQN model.

Images