Two-stage resource optimization method for energy-saving semantic communication

Abstract

The invention relates to a two-stage resource optimization method for energy-saving semantic communication, which comprises the following steps of: constructing a resource optimization problem which comprises a channel allocation transmission model, a data queue model and a battery and energy model and is oriented to energy-saving semantic communication, and maximizing semantic energy efficiency of a system in a long time as an optimization target; channel allocation and the number of semantic symbols of a single word when different users extract semantics are obtained based on a multi-dimensional discrete near-end strategy optimization algorithm, and then two-stage deep learning of power allocation of different channels is obtained through a depth deterministic strategy gradient algorithm, so that resource optimization configuration of a semantic communication system for sustainable power supply is realized. According to the method, through variable conversion and multi-dimensional discrete action spaces, the dimensionality output by the neural network when multiple discrete action spaces are processed at the same time is remarkably reduced, the convergence performance is improved, and the method is suitable for various scenes formed by wireless semantic communication systems containing complex mixed optimization variables.

Description

Technical Field

[0001] The present invention relates to a technology in the field of deep learning, specifically a two-stage resource optimization method for energy-saving semantic communication. Background Art

[0002] Existing semantic communication is task-based, understanding first and then transmitting. It uses semantic information for communication through selective feature extraction, compression and transmission, and has higher transmission efficiency and lower energy consumption. However, the implementation of semantic communication systems also faces many challenges. How to balance resources such as communication, computing, energy consumption and storage has become an important issue. Although a larger semantic knowledge base can transmit and restore information more accurately, it will also occupy a large amount of storage resources; semantic communication systems based on artificial intelligence (AI) will incur high computing costs and energy consumption during training and deployment. Therefore, coordinating different resources, optimizing semantic communication systems, and breaking through the limits of traditional communications have become necessary ideas and new research areas to meet the needs of sustainable development of future communications. Summary of the invention

[0003] The present invention aims to solve the reinforcement learning problem in mixed action space that cannot be solved by existing technologies; and the poor processing effect on multidimensional action space composed of multiple single-dimensional discrete actions. A two-stage resource optimization method for energy-saving semantic communication is proposed. Through variable conversion and multidimensional discrete action space, the dimension of neural network output is significantly reduced when processing multiple discrete action spaces at the same time and the convergence performance is improved. The method is suitable for scenarios consisting of various wireless semantic communication systems containing complex mixed optimization variables.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a two-stage resource optimization method for energy-saving semantic communication. By constructing a resource optimization problem for energy-saving semantic communication including a channel allocation transmission model, a data queue model, and a battery and energy model, maximizing the semantic energy efficiency of the system over a long period of time is taken as an optimization goal. Channel allocation and the number of single-word semantic symbols when different users extract semantics are obtained by a multi-dimensional discrete proximal policy optimization algorithm. Then, a two-stage deep learning of power allocation of different channels is obtained by a deep deterministic policy gradient algorithm, thereby realizing resource optimization configuration for a semantic communication system with sustainable power supply.

[0006] The two-stage deep learning mentioned above refers to: setting the reward functions of the two stages respectively, initializing the training environment, and obtaining the initial state from the environment. In the first stage, the multi-dimensional discrete proximal policy optimization (MPPO) algorithm is used to obtain the channel allocation and the number of symbols in a single word; in the second stage, the deep deterministic policy gradient optimization (DDPG) algorithm is used to obtain the power allocation size based on the channel allocation and the number of symbols. As input to the environment, the battery level, the queue length stored in the data buffer, and the MPPO and DDPG network parameters are updated until the reward function converges. Technical Effects

[0007] The present invention combines a multi-dimensional discrete action space with a proximal policy optimization (PPO) algorithm, and according to the characteristics of the discrete action space, proposes a variable form conversion to solve the optimization of discrete variables; through a two-stage reinforcement learning scheme, the optimization problem of discrete variables is solved in the first stage, and then according to the results of the first stage, the optimization problem of continuous variables is solved in the second stage by training a single channel once instead of training all channels once, while improving the convergence performance; thirdly, for the semantic communication system under wireless conditions, the communication performance is combined with the system energy consumption through semantic energy efficiency, and the proposed semantic communication system can be effectively optimized. Compared with the prior art, the present invention significantly reduces the dimension of the neural network output, solves the problem of the difficulty of reinforcement learning convergence when multiple high-dimensional discrete action spaces are combined, and can effectively measure the relationship between the energy consumption and performance of the semantic communication system under wireless conditions, and at the same time, establishes an energy-saving semantic communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of the present invention;

[0009] Figure 2 is a scene diagram of an embodiment;

[0010] Figure 3 Schematic diagram of input and output of multi-dimensional discrete action space;

[0011] Figure 4 It is the MPPO-DDPG flow chart;

[0012] Figure 5 It is a schematic diagram of the effect of the embodiment. DETAILED DESCRIPTION

[0013] like Figure 2As shown, this is the application scenario of this embodiment, that is, a wireless communication system for energy-saving semantic communication including a base station and N = {1, 2, ..., N} devices, each device is equipped with a data buffer and a rechargeable battery, and a deep learning-based semantic communication system (DeepSC) module is installed as a semantic communication model. The data buffer on the device is used to temporarily store unsent data and follows the first-in-first-out principle, but its storage size is limited. The rechargeable battery collects renewable energy (such as wind energy, solar energy, etc.) from the external environment through an energy harvesting device, and uses it to provide energy for semantic extraction and transmission to the device. Compared with obtaining energy directly from the traditional power grid, the use of renewable energy not only improves the economy of the system, but also further realizes an energy-saving semantic communication system. The total bandwidth between the device and the base station is evenly distributed to M = {1, 2, ..., M} channels, and the channel conditions change over time.

[0014] Assuming that the duration of each time slot is unit time, the numerical values ​​of "power" and "energy" are equal here and the concepts can be used interchangeably.

[0015] like Figure 1 As shown, this embodiment relates to a two-stage resource optimization method for energy-saving semantic communication, including:

[0016] Step 1: Construct a channel allocation transmission model, which specifically includes: a channel can only be allocated to one device at the same time, and a device can select multiple channels at the same time. And when all channels are time-varying channels, the channel fading coefficient changes with time t, so the signal-to-noise ratio of the mth channel connected to device n at time t is Semantic transmission rate of the mth channel of device n at time t in: Assign a variable to the channel associated with the user, when the mth channel is assigned to device n at time t otherwise is the transmission power of device n on channel m at time t, is the channel fading coefficient of the mth channel connected to device n at time t, N0 is the noise power spectral density, W is the bandwidth of a single channel, L is the average number of words in a single sentence, is the average number of semantic symbols in a single word, the semantic unit (sut) measures the size of semantic information, let I be the average amount of semantic information to be transmitted by a single sentence, is the semantic similarity, which is used to measure the semantic fidelity, that is, the semantic information distance between the original text and the restored text. Signal-to-noise ratio and the number of symbols per word is a function of , which depends on the channel state and the structure of the semantic communication system, and W is equal to the symbol rate of the passband transmission. In addition, since I and L are only related to the information source to be transmitted, they can be regarded as constants.

[0017] Step 2: construct a data queue model, specifically: the relationship between the amount of data temporarily stored in the data buffer of device n and the time (t+1) satisfies: in: is the random arrival rate, which satisfies the uniform distribution. The service rate of the semantic information transmitted on the mth channel of device n at time t is Q max It is the maximum queue length that the data buffer can temporarily store and satisfies the first-in-first-out (FIFO) principle for transmitted data. is the total information service rate of device n at time t.

[0018] Step 3: Construct a battery and energy model, specifically: the battery charge of device n at time (t+1) Semantic Energy Efficiency Among them: The energy obtained by device n at time t to satisfy uniform distribution is Energy consumed The remaining power is The maximum capacity of the rechargeable battery in each device is E max , the energy consumed by device n at time t should not exceed the available energy of the battery at that time The semantic encoding power of device n at time t is κ n is the energy conversion coefficient related only to the chip structure, the clock frequency of the GPU of device n at time t G is the average number of floating-point operations required for a single sentence, and z is the number of floating-point operations per GPU cycle.

[0019] The energy consumed by the device n at time t includes two parts: the energy required by the semantic encoder to extract semantic information before data transmission and the energy consumed during the data transmission process.

[0020] Step 4: Based on the model established in steps 1-3, construct a resource optimization problem for energy-saving semantic communication, that is, maximize the semantic energy efficiency of the system under the constraints of the number of devices and channel connections, the arrival and departure of data packets and the size of the data buffer, the battery charging and discharging and capacity level, and the semantic fidelity of the system. Specifically: Where: min is the semantic similarity threshold to ensure the accuracy of the system.

[0021] Preferably, in order to solve the problem of channel allocation variables associated with users and channels in the channel allocation transmission model as the number of devices and channels increases, The number of will increase rapidly, which will lead to the problem of increased training time and convergence difficulty of the deep reinforcement learning (DRL) algorithm. Modified to: n=1,2,...,N,m=1,2,...,M means the nth device selects the mth channel at time t, where N is the number of devices and M is the number of channels. Therefore, the relationship between the channel allocation variables before and after the conversion is n=1,2,...,N,m=1,2,...,M; through conversion, new channel allocation variables and the number of symbols in a single word It can be regarded as a multi-dimensional discrete action composed of multiple single-dimensional discrete actions. In order to solve the multi-discrete action space, the multi-dimensional discrete action is usually directly converted into discrete actions (CMD) and invalid actions (RA) are removed according to the actual problem. However, this often significantly increases the dimension of the neural network output. Therefore, the multi-dimensional discrete action space is introduced into the first stage of the DRL algorithm, and multiple heads are used to process the multi-dimensional discrete actions, such as Figure 3 As shown in Figure 1, each head is linked to a discrete action space. After the conversion of channel allocation variables and the introduction of multi-dimensional discrete action space, the output dimension of the neural network in the DRL algorithm is significantly reduced from the original (N M ×C N ) is reduced to (NM+CN), and the convergence ability is also improved.

[0022] Step 5: Figure 4 As shown, the resource optimization problem for energy-saving semantic communication is solved by multi-dimensional discrete proximal policy optimization-deep deterministic policy gradient algorithm (MPPO-DDPG), including: using two stages of two DRL frameworks to adapt to the mixed action space including discrete and continuous actions, where: in the first stage, the multi-dimensional discrete proximal policy optimization algorithm (MPPO) is used to generate discrete actions, namely channel allocation and the number of semantic symbols, and in the second stage, the deep deterministic policy gradient algorithm (DDPG) is used to obtain continuous transmission power, and finally the target actor and critic network are soft-updated.

[0023] The triplet of the DRL framework in this embodiment is {(S i ,A i ,r i ),i=1,2}, where: S i ,A i ,r i They are the state, action and reward of the Markov decision process (MDP) in the first or second stage, including:

[0024] The triples in the first stage of the multi-dimensional discrete proximal policy optimization algorithm (MPPO) include: At time t, the state of MPPO is Actions are discrete elements Rewards for device n Where: η n is the penalty for data overflow or the total energy used exceeds the sum of the battery power and the collected energy or the semantic similarity does not meet the threshold; the instant reward of MPPO is

[0025] The first stage is as follows: at each time t, the MPPO agent deployed at the base station observes the current state Output the probabilities of all actions and randomly select actions based on the probability distribution Like PPO, MPPO also adopts the actor-critic framework, where the actor network is used to obtain the strategy and the critic network is used to minimize the loss between the estimated state value and the actual observed return.

[0026] The actor network is updated by maximizing the objective function, specifically: Where: θ1 and φ1 are the parameters of the actor and critic networks of MPPO respectively. The probability ratio between the new and old strategies of MPPO is is the action probability of the current strategy, is the action probability of the previous strategy, m is the number of output heads, and the advantage function at time t is the state value function, with discount factor δ∈[0,1]. The function clip(·) clips β t (θ1) keeps it between (1-∈) and (1+∈), where: ∈ is the clipping factor.

[0027] The critic network is updated by minimizing the loss function, specifically:

[0028] Preferably, the updating is performed by setting a replay buffer to store data groups such as the state, action, reward strategy and next step action of each step.

[0029] Preferably, each training update of the DRL network uses the MPPO replay buffer The data group in each step is The data set in the replay buffer is divided into several small batches through importance sampling to train and update the MPPO network.

[0030] The triples in the second stage of the Deep Deterministic Policy Gradient (DDPG) algorithm include: The state of DDPG is in: Actions with sequential elements in: According to the instant reward of MPPO, the instant reward of DDPG is in:

[0031] The second stage is as follows: the DDPG agent observes the current state at each time t And output action The actor network selects and evaluates actions according to the strategy μ, and the critic network approximates the target value function.

[0032] The critic network is updated by minimizing the loss function, specifically: in: For the critic network, For the actor network, is the target critic network, is the target actor network, They are the model parameters of four deep neural networks (DNNs), The discount factor ρ∈[0,1], N' is the batch size.

[0033] The actor network is updated by maximizing the policy objective function, specifically: Its update direction is

[0034] The soft update includes: Where: σ is the learning rate.

[0035] Preferably, the replay buffer z2 of DDPG is set to store the data group The DDPG network is trained by obtaining a set small batch of data sets through random sampling.

[0036] After specific practical experiments, N = 5 devices equipped with DeepSC semantic communication components are evenly placed in a circular area with a radius of 400m, and the base station is located in the center. The number of channels M = 10, the bandwidth of a single channel is W = 200kHz, and the threshold of sentence similarity ξ min =0.9. The energy harvesting satisfies the uniform distribution in the range [900,1000]mJ, and the arrival rate satisfies the uniform distribution in the range [1000,4000] sentences / second. The maximum capacity of the battery is E max =2000mJ, the initial level is The data buffer size configured on the device is Q max =50,000 sentences.

[0037] like Figure 4 As shown, two-stage deep reinforcement learning is performed based on the above scenario, including:

[0038] Step i) Initialize DRL parameters: Initialize the parameters θ1 of the MPPO actor network, φ1 of the critic network, and the parameters of the DDPG actor network respectively. Parameters of the critic network Parameters of the target actor network Parameters of the target critic network Set the replay buffer size of MPPO and DDPG respectively and The batch sizes are z1 and z2 respectively.

[0039] Step ii) Initialize the environment: Set the reward function, initialize the environment according to the optimization problem P1, and obtain the initial state from the environment

[0040] Step iii) Solving for discrete parameters: At each time slot t, MPPO is used to solve for discrete parameters, namely, channel allocation and the number of symbols in a single word when each device extracts semantics.

[0041] Step iv) Solving the continuous parameters: Based on the obtained discrete parameters, DDPG is used to solve the continuous parameters, that is, the power allocation problem of each channel for each user, in each time slot t.

[0042] Step v) Update the environment and store data: Take the actions of the two stages as input to the environment, update the environment, and obtain the reward, penalty, and state at the next moment. Put the state, action, reward, probability of strategy at the current moment, and state at the next moment of the first stage as a data group into the replay buffer of MPPO, and put the state, action, reward, and state at the current moment as a data group into the replay buffer of DDPG.

[0043] Step vi) Update the DRL network: When the data in the replay buffer of MPPO and DDPG is updated to a certain amount, update the MPPO and DDPG network parameters.

[0044] Step vii) Convergence judgment: If the reward function has converged, stop training and update the network, and save the parameters of the actor and critic networks. Otherwise, repeat steps iii) to vi) until convergence.

[0045] Since both MPPO and DDPG algorithms are based on the actor-critic structure, they only differ in the number of actors or critic networks used. The computational complexity based on the actor-critic structure is in: is the number of neurons in the jth layer of the actor network, is the number of neurons in the jth layer of the critic network. According to the complexity calculation rules, in: are the number of neurons in the jth layer of the actor and critic networks in MPPO and DDPG, respectively.

[0046] like Figure 5 As shown, the relationship between the training benefits and training rounds under different algorithms of the present invention is shown. The energy-saving wireless semantic communication system is modeled through software simulation, and the system resources are configured through the algorithm multi-dimensional discrete proximal policy optimization-deep deterministic policy gradient algorithm (MPPO-DDPG) in the present invention, and compared with multi-dimensional discrete proximal policy optimization-dual-delay deep deterministic policy gradient (MPPO-TD3), multi-dimensional discrete proximal policy optimization-deep deterministic policy gradient (entire) algorithm (MPPO-DDPG (Entirety)) considering single-slot joint training of all channel power configurations, and random algorithms that do not consider optimizing channel allocation, transmission power and number of symbols. It can be seen that by configuring resources for the energy-saving wireless semantic communication system through MPPO-DDPG, higher benefits can be obtained in a shorter time, that is, higher energy efficiency.

[0047] Compared with the existing technology, this method solves the difficulties of traditional reinforcement learning in processing large-scale multi-dimensional discrete action space and mixed action space. It solves the actual situation that traditional resource methods cannot be applied due to the particularity of semantic communication, improves the communication performance of the semantic communication system under unit energy consumption, and realizes resource allocation for energy-saving semantic communication.

[0048] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A two-stage resource optimization method for energy-saving semantic communication, characterized in that: By constructing a resource optimization problem for energy-saving semantic communication that includes a channel allocation transmission model, a data queue model, and a battery and energy model, maximizing the semantic energy efficiency of the system over a long period of time is taken as the optimization goal. The channel allocation and the number of single-word semantic symbols when different users extract semantics are obtained through a multi-dimensional discrete proximal policy optimization algorithm. Then, a two-stage deep learning of power allocation for different channels is obtained through a deep deterministic policy gradient algorithm, thereby achieving resource optimization configuration for a semantic communication system with sustainable power supply.

2. The two-stage resource optimization method for energy-saving semantic communication according to claim 1 is characterized by: The two-stage deep learning mentioned above refers to: setting the reward functions of the two stages respectively, initializing the training environment, and obtaining the initial state from the environment. In the first stage, the channel allocation and the number of symbols in a single word are obtained by the multi-dimensional discrete proximal policy optimization (MPPO) algorithm; In the second stage, the power allocation size is obtained based on the channel allocation and the number of symbols through the deep deterministic policy gradient optimization (DDPG) algorithm. Finally, the actions of the two stages are combined. As input to the environment, the battery level, the queue length stored in the data buffer, and the MPPO and DDPG network parameters are updated until the reward function converges.

3. The two-stage resource optimization method for energy-saving semantic communication according to claim 1 is characterized in that: include: Step 1: Construct a channel allocation transmission model, which specifically includes: a channel can only be allocated to one device at the same time, and a device can select multiple channels at the same time. And when all channels are time-varying channels, the channel fading coefficient changes with time t, so the signal-to-noise ratio of the mth channel connected to device n at time t is Semantic transmission rate of the mth channel of device n at time t in: Assign a variable to the channel associated with the user, when the mth channel is assigned to device n at time t otherwise is the transmission power of device n on channel m at time t, is the channel fading coefficient of the mth channel connected to device n at time t, N0 is the noise power spectral density, W is the bandwidth of a single channel, L is the average number of words in a single sentence, is the average number of semantic symbols in a single word, the semantic unit (sut) measures the size of semantic information, let I be the average amount of semantic information to be transmitted by a single sentence, is the semantic similarity, which is used to measure the semantic fidelity, that is, the semantic information distance between the original text and the restored text. Signal-to-noise ratio and the number of symbols per word A function of which depends on the channel state and the structure of the semantic communication system, W is equal to the symbol rate of the passband transmission; Step 2: construct a data queue model, specifically: the relationship between the amount of data temporarily stored in the data buffer of device n and the time (t+1) satisfies: in: is the random arrival rate, which satisfies the uniform distribution. The service rate of the semantic information transmitted on the mth channel of device n at time t is Q max The maximum queue length that the data buffer can temporarily store and that satisfies the first-in-first-out (FIFO) principle for transmitted data. is the total information service rate of device n at time t; Step 3: Construct a battery and energy model, specifically: the battery charge of device n at time (t+1) Semantic Energy Efficiency Among them: The energy obtained by device n at time t to satisfy uniform distribution is Energy consumed The remaining power is The maximum capacity of the rechargeable battery in each device is E max , the energy consumed by device n at time t should not exceed the available energy of the battery at that time The semantic encoding power of device n at time t is κ n is the energy conversion coefficient related only to the chip structure, the clock frequency of the GPU of device n at time t G is the average number of floating-point operations required for a single sentence, and z is the number of floating-point operations per GPU cycle; Step 4: Based on the model established in steps 1-3, construct a resource optimization problem for energy-saving semantic communication, that is, maximize the semantic energy efficiency of the system under the constraints of the number of devices and channel connections, the arrival and departure of data packets and the size of the data buffer, the battery charging and discharging and capacity level, and the semantic fidelity of the system. Specifically, P1: Where: min is the semantic similarity threshold to ensure the accuracy of the system; Step 5. Solve the resource optimization problem for energy-saving semantic communication through multi-dimensional discrete proximal policy optimization-deep deterministic policy gradient algorithm (MPPO-DDPG), including: using two deep reinforcement learning (DRL) frameworks in two stages to adapt to the mixed action space including discrete and continuous actions, where: in the first stage, the multi-dimensional discrete proximal policy optimization algorithm (MPPO) is used to generate discrete actions, namely channel allocation and the number of semantic symbols, and in the second stage, the deep deterministic policy gradient algorithm (DDPG) is used to obtain continuous transmission power, and finally the target actor and critic network are soft-updated.

4. The two-stage resource optimization method for energy-saving semantic communication according to claim 2 or 3, characterized in that: The multidimensional discrete action space is introduced into the first stage of the DRL algorithm. Each head is linked to a discrete action space. After the conversion of channel allocation variables and the introduction of multidimensional discrete action space, the output dimension of the neural network in the DRL algorithm is significantly reduced from the original (N M ×C N ) is reduced to (NM+CN), and the convergence ability is also improved.

5. The two-stage resource optimization method for energy-saving semantic communication according to claim 2 or 3, characterized in that: The triplet of the deep reinforcement learning (DRL) framework is {(S i ,A i ,r i ),i=1,2}, where: S i ,A i ,r i They are the state, action, and reward of the MDP in the first or second stage, including: The triples in the first stage of the multi-dimensional discrete proximal policy optimization algorithm (MPPO) include: At time t, the state of MPPO is Actions are discrete elements Rewards for device n Where: η n is the penalty for data overflow or the total energy used exceeds the sum of the battery power and the collected energy or the semantic similarity does not meet the threshold; the instant reward of MPPO is The triples in the second stage of the Deep Deterministic Policy Gradient (DDPG) algorithm include: The state of DDPG is in: Actions with sequential elements in: According to the instant reward of MPPO, the instant reward of DDPG is in:

6. The two-stage resource optimization method for energy-saving semantic communication according to claim 5 is characterized by: The first stage is as follows: at each time t, the MPPO agent deployed at the base station observes the current state Output the probabilities of all actions and randomly select actions based on the probability distribution MPPO adopts an actor-critic framework, where the actor network is used to obtain the strategy and the critic network is used to minimize the loss between the estimated state value and the actual observed return; The actor network is updated by maximizing the objective function, specifically: Among them: θ1 and φ1 are the parameters of the actor and critic networks of MPPO respectively, and the probability ratio between the new and old strategies of MPPO is is the action probability of the current strategy, is the action probability of the previous strategy, m is the number of output heads, and the advantage function at time t is the state value function, the discount factor δ∈[0,1], and the function clip(·) clips β t (θ1) keeps it between (1-∈) and (1+∈), where: ∈ is the clipping factor; The critic network is updated by minimizing the loss function, specifically:

7. The two-stage resource optimization method for energy-saving semantic communication according to claim 6 is characterized by: The update is performed by setting a replay buffer to store data groups such as the state, action, reward strategy, and next step action of each step; Each training update of the DRL network uses the MPPO replay buffer The data group in each step is The data set in the replay buffer is divided into several small batches through importance sampling to train and update the MPPO network.

8. The two-stage resource optimization method for energy-saving semantic communication according to claim 5 is characterized in that the second stage is specifically: the DDPG agent observes the current state at each time t And output action The actor network selects and evaluates actions according to the strategy μ, and the critic network approximates the target value function. The critic network is updated by minimizing the loss function, specifically: in: For the critic network, For the actor network, is the target critic network, is the target actor network, are the model parameters of four deep neural networks (DNNs), Discount factor ρ∈[0,1], N' is the batch size; The actor network is updated by maximizing the policy objective function, specifically: Its update direction is 9. The two-stage resource optimization method for energy-saving semantic communication according to claim 8, characterized in that: The soft update includes: Where: σ is the learning rate; Setting up DDPG's replay buffer To save data sets The DDPG network is trained by obtaining a set small batch of data sets through random sampling.

Images