A multi-relay node joint optimization method and device based on deep reinforcement learning

Abstract

The present invention discloses a multi-relay node joint optimization method and device based on deep reinforcement learning. In this method, when a task node gradually moves away from the command center while executing a task, the command center intelligent agent determines whether to send or retract relay nodes based on the status it collects, maintaining an appropriate number of nodes in the relay link. At the same time, the relay node intelligent agent outputs the next acceleration based on the status it collects to optimize the link status. The two cooperate with each other to optimize the relay link, so that the relay link can obtain higher link quality and lower operating energy consumption while maintaining smooth operation, while maintaining a low number of nodes. During the training process, the intelligent agent converges quickly and can complete the relay optimization task more flexibly and effectively through methods such as an experience pool with priority weights, a delayed updated target network, and noise exploration.

Description

Technical Field

[0001] The present invention relates to the field of network communication, and in particular to a multi-relay node joint optimization method and device based on deep reinforcement learning Background Art

[0002] In emergency communications scenarios, infrastructure often fails to meet the basic communication service requirements of rescue operations due to unexpected circumstances. A wireless mesh network (FANET) composed of drone nodes offers advantages such as fast initial deployment, strong fault tolerance, wide coverage, and high flexibility, making it ideal for building emergency communications networks.

[0003] Most current research optimizes relay deployment using heuristic algorithms such as genetic algorithms, computational geometry algorithms, virtual potential field methods, and particle swarm optimization, based on knowledge of the deployment environment, including user node locations, base station locations, obstacle locations, and user node movement trajectories. However, heuristic algorithms are prone to local optima, requiring parameter adjustments when network states change. This leads to poor generalization and, when applied to large networks, a significant time cost in finding the optimal strategy. In contrast, reinforcement learning, a method based on interactive learning between an agent and its environment, offers advantages for solving complex optimization problems with multiple constraints. Its autonomous learning and exploration properties enable reinforcement learning to overcome the limitations of traditional heuristic algorithms when searching for appropriate solutions, improving optimization accuracy, accelerating convergence, and avoiding local optima. This provides a novel approach to solving optimization problems in emergency communication scenarios. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a multi-relay node joint optimization method and device based on deep reinforcement learning, so as to achieve, in emergency communication scenarios, under the premise of ensuring that the communication of the relay link is basically smooth, the management of the number of relay nodes and the dynamic optimization of the position of the relay nodes, so as to ultimately enable the relay link to obtain better link quality, fewer nodes and smaller operating energy consumption.

[0005] To achieve the above objectives, an embodiment of the present invention discloses a multi-relay node joint optimization method based on deep reinforcement learning, which is applied to a command center. The communication relay optimization method includes:

[0006] The status acquisition module installed in the command center periodically obtains the status information of the entire relay link from the relay nodes, where the timing refers to an extremely short period of time Δt, usually within one second. The entire relay link includes all relay nodes, task nodes and the relay network composed of the command center.

[0007] The command center controls the number of relay nodes through the command center node number management algorithm installed on the intelligent body module, wherein the input of the node number management algorithm is the overall link information of the above-mentioned relay link, and the output is to select one of three actions from the action space, and the action space includes: (1) sending a start operation signal to the new relay node; (2) sending a recovery signal to the relay node that departed most recently; (3) monitoring the environmental status and maintaining the number of relay nodes unchanged, wherein the new relay node is a relay node that is dormant in the command center. After receiving the start operation signal, it will start from the command center to execute the relay optimization method, form a relay link with other relay drones, and provide relay for the task node.

[0008] The status information collected by the status collection module installed in the command center includes the relevant link quality R of the two relay nodes last added in the relay link. 1o , R 2o , the link quality R with the minimum link quality among all relay links that can be known at this time o , relative position (x1,y1), (x2,y2), number of relay nodes n o and quantity limit n max In addition, in order to enable the intelligent agent to have certain prediction capabilities, the input state will also include the state of the previous moment.

[0009] The command center status information is:

[0010] S o =[x1,y1,x2,y2,R 1o ,R 2o ,R o ,n n ,n max ] T＝t,t-1

[0011] The link quality R is represented by the link received signal strength (RSSI), T=t is the current time, and T=t-1 is the previous time.

[0012] The node quantity management method is used in the training phase. The intelligent agent module interacts with the simulation environment module. The simulation environment updates parameters according to the actions. The intelligent agent module updates and improves its own strategy according to the reward function of the simulation environment module.

[0013] The simulation environment module is built based on real-world emergency scenarios, ignoring the presence of obstacles. The relay nodes and task nodes in the environment conform to the dynamic constraints of the drone nodes. The channels between nodes meet the drone channel model. The environment model terminates training for a round when the maximum number of time steps is reached or when the relay link is disconnected.

[0014] Among them, the reward function is related to the overall link status, the number of relay nodes and the location of the task node.

[0015] Specific reward value r o =r t +r nn +r ex , where r t It is expressed as the sum of rewards or penalties obtained due to communication quality, r nn represents the reward related to the number of nodes in the relay link, r ex Indicates the reward given due to system termination or exceeding the limit.

[0016] The algorithm of the agent module in the deep reinforcement learning node quantity management method uses a neural network composed of three fully connected layers as the basic network. The network input is the 18-dimensional variable command center state information, and the output is the expected action value Q that can be obtained for each of the three actions of the command center. The expected action value Q is the sum of the reward r obtained by executing the current action and the predicted future reward multiplied by the reward decay exponent γ. The expected action value is:

[0017] Q(s,a)=r+γQ(s',a')

[0018] Among them, s' is the next environmental state, and a' is the action corresponding to the maximum expected action value output by the next environmental command center node quantity management algorithm.

[0019] The agent will also add a certain amount of exploration noise to its action selection during training, that is, it will explore other actions with a probability of ε, and ε will become smaller and smaller as the training progresses.

[0020] The node quantity management algorithm of the agent module has a training network and a target network. The training network updates its strategy based on past experience during its interaction with the simulated environment, and the target network updates its parameters after the training network has updated several times. The network parameter update method is gradient descent, with the goal of making the action value output by the training network approach the sum of the weighted predicted next action value output by the target network and the actual reward value. The updated parameter difference is:

[0021]

[0022] Where a is the learning rate, s', a' are the next node state and the action selected to maximize the action value, θ, θ' are the parameters of the target network and the training network respectively. When the agent selects an action, it will add a certain amount of noise to enhance its efficiency in exploring other actions. The noise will satisfy the normal distribution and the mean of the distribution will be offset towards the node with the poor link quality. It is expressed as

[0023] A multi-relay node joint optimization method based on deep reinforcement learning is applied to relay nodes. It optimizes the relay node position in real time based on the relay node's own status and the status of the overall link, optimizes the communication quality of the relay link, and minimizes mobile power consumption and node number consumption. It includes:

[0024] The status acquisition module installed in the relay node regularly summarizes its own link status information based on its own status and the information obtained from the nodes at both ends, wherein the nodes at both ends are the predecessor and successor nodes of the chain wireless mesh relay network that maintain wireless network communication connectivity with the relay node.

[0025] The relay node adjusts its own position in real time through the relay node position optimization algorithm installed in the relay agent module, where the input of the position optimization algorithm is the state information summarized by the above relay node, and the output is the acceleration of the drone node operation that meets the drone node's own constraints, including the direction and magnitude of the acceleration.

[0026] The status information obtained by the relay node status acquisition module includes the relative coordinates (x r ,y r ), (x m -x r ,y m -y r ), the node's own speed (v x ,v y ), RSSI values ​​R1 and R2 of the two links of the node, and the minimum RSSI value R of the entire link at the left and right ends of the node L , R L , and contains the states of this step and the previous step.

[0027] The relay node status information is:

[0028] S t =[x r ,y r ,x m -x r ,y m -y r ,v x ,v y ,R1,R2,R L ,R R ] T＝t,t-1

[0029] Similar to the command center quantity control method, the relay node position optimization method designs a segmented reward function in the simulation environment module to guide the relay node position optimization agent, where the reward function is segmented according to the relay link quality related to the current node, and is related to the link quality, mobile energy consumption, overall link quality and whether to change the number of nodes.

[0030] Specific reward value r = r t +r p +r n +r ex

[0031] where r t It is expressed as the sum of rewards or penalties obtained due to communication quality, r p Represents the sum of rewards or penalties due to motion power consumption, r n Indicates the reward obtained from the entire link or other nodes, r ex Represents the sum of other rewards or penalties, including the penalties received by the agent for violating the constraints, and the rewards or penalties received by the agent based on the final results after a round of training.

[0032] The node position optimization method stores the empirical data and weights of each agent's interaction with the environment in a memory pool, and randomly selects and eliminates nodes based on the weights. The weights are related to the state of the relay node and its contribution to the overall agent's learning. The minimum weight is 1, and it is decremented by 1 after each use, and is deleted when it reaches 0.

[0033] The relay node agent uses a neural network consisting of three fully connected layers as its basic network. The overall network uses two target evaluation networks, two training evaluation networks, a target action network, and a training action network. The action network inputs the state S observed by the agent and outputs the action A that maximizes the action value. The evaluation network inputs the state S and action A and outputs the calculated action value Q. The ultimate goal of the algorithm is to train the actor target network so that it can reasonably respond to the state S in the environment and output the action that maximizes the cumulative action value. The training evaluation network is updated as follows:

[0034]

[0035]

[0036] in, To train the action network, the action output is based on the next state. r is the reward value after executing the current action. The network uses gradient descent to minimize the square of the difference between the reward value r and the weighted maximum action value output by the action evaluation network with the lowest next action value prediction, and the action value for the current state. The training action then uses gradient descent to maximize the action value of its own output action.

[0037] The training network parameters will be soft-synchronized with the target network every few rounds. The update method is:

[0038] θ i '←τθ i +(1-τ)θ i '

[0039]

[0040] where θ i is the parameter of the i-th training evaluation network, θ i 'Evaluate the parameters of the network for the i-th target, are the parameters of the first training action network, are the parameters of the target action network.

[0041] The embodiment of the present invention discloses a multi-relay node joint optimization device based on deep reinforcement learning, which is applied to a command center. The node quantity management device includes:

[0042] The command center status acquisition module is used to collect the environmental parameters required by the input state of the intelligent agent module at each moment, perform normalization processing, and provide them to the intelligent agent module.

[0043] The command center intelligent agent module is used to manage actions based on the number of nodes that need to be executed according to the state output, and optimize its own output strategy based on the rewards given by the environment.

[0044] The command center memory pool module is used to store the action state reward at the current moment and the state at the next moment generated by the interaction between the command center agent module and the simulation environment module, and provide it to the agent for learning to improve the agent strategy.

[0045] The command center action exploration module is used to determine the specific actions performed by the command center and optimize the output strategy while fully exploring the action space.

[0046] An embodiment of the present invention provides a multi-relay node joint optimization device based on deep reinforcement learning, which is applied to a command center. The location optimization device includes:

[0047] The relay node state acquisition module is used to collect the environmental parameters required by the input state of the intelligent agent module at each moment, perform normalization processing, and provide them to the intelligent agent module.

[0048] The relay node agent module is used to manage the number of nodes that need to be executed according to the state output, and optimize its own output strategy based on the rewards given by the environment.

[0049] The relay node memory pool module is used to store the current action, state, reward and next state and weight generated by the interaction between the relay node agent module and the simulation environment module, and provide them to the agent for learning to improve the agent strategy.

[0050] The relay node action exploration module is used to determine the specific actions performed by the relay node and optimize the output strategy while fully exploring the action space.

[0051] It can be seen from the above technical solution that in the embodiment of the present invention, the command center and relay node intelligent agents generate experience data by interacting with the simulation environment. These data and their weights are stored in the memory pool, and when the intelligent agent meets the iterative optimization conditions, they are randomly taken out with weights for the intelligent agent to learn and optimize its own strategy, ultimately achieving real-time optimization of the relay link in the emergency communication scenario, and being able to learn action modes that adapt to special tasks and autonomously respond to emergencies in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a flowchart of a deep reinforcement learning relay optimization method provided by an embodiment of the present invention.

[0054] Figure 2 This is a deep reinforcement learning relay link topology diagram provided by an embodiment of the present invention.

[0055] Figure 3 This is a diagram of the structure of the deep reinforcement learning command center agent module provided by an embodiment of the present invention.

[0056] Figure 4 This is a structural diagram of the deep reinforcement learning relay node agent module provided by an embodiment of the present invention

[0057] Figure 5 This is a diagram of the command center structure provided by an embodiment of the present invention.

[0058] Figure 6 This is a diagram of the relay node structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] The present invention is described in detail below through specific embodiments.

[0061] See also Figure 1 , Figure 1 This is a flow chart of a relay optimization method provided by an embodiment of the present invention.

[0062] Step 101, the initialization step, includes initializing the agent and the simulation environment. Agent initialization primarily involves initializing the target network in the command center agent and synchronizing it with the training network, as well as the target evaluation network and target action network in the relay node agents and synchronizing them with the training evaluation network and training action network. Initialization of the simulation environment module involves random initialization of environment parameters, the positions of each node, and the target positions of the task nodes.

[0063] Step 102: When the agent collects the status, the status collection modules in the command center and relay nodes collect the status they need and perform normalization processing. The status collection module installed in the command center collects the status information required by the agent module in the command center, including the link quality R of the last two relay nodes added in the relay link. 1o , R 2o , the link quality R with the minimum link quality among all relay links that can be known at this time o , relative position (x1,y1), (x2,y2), number of relay nodes n o and quantity limit n max In addition, in order to enable the intelligent agent to have certain prediction capabilities, the input state will also include the state of the previous moment.

[0064] The command center status information is:

[0065] S o =[x1,y1,x2,y2,R 1o ,R 2o ,R o ,n n ,n max ] T＝t,t-1

[0066] The link quality R is represented by the link received signal strength (RSSI), T=t is the current time, and T=t-1 is the previous time.

[0067] The relay node state acquisition module collects the state information required by the relay node agent module, including the relative coordinates (x r ,y r ), (x m -x r ,y m -y r ), the node's own speed (v x ,v y ), RSSI values ​​R1 and R2 of the two links of the node, and the minimum RSSI value R of the entire link at the left and right ends of the node L , R L , the overall schematic diagram of the left and right sections is as follows Figure 2 As shown in the figure, the final state will also include the states of this step and the previous step.

[0068] The relay node status information is:

[0069] S t =[x r ,y r ,x m -x r ,y m -y r ,v x ,v y ,R1,R2,R L ,R R ] T＝t,t-1

[0070] When the agent outputs an action, the more stable target network in the command center agent adds new nodes or reduces nodes based on the input state output, maintaining the current state value of each of the three actions. Then, a certain amount of exploration noise is added during action selection, that is, other actions are explored with a probability of ε, where ε is less than 1 and will become smaller as training progresses.

[0071] When the relay node agent outputs an action, the more stable target action network in the agent outputs the action that can maximize the current action value based on the input state. Subsequently, the agent will add a certain amount of noise when selecting an action to enhance its efficiency in exploring other actions. The noise will satisfy the normal distribution and the distribution mean will shift toward the end node with the poor current link quality, expressed as a′=a+ε

[0072] Step 103: Interact with the environment. The agent outputs actions to interact with the simulated environment. The environment modifies the relay node actions and the command center actions to satisfy the environmental constraints. It then changes the positions of each node and calculates the link quality and power between the nodes. Finally, the relay node and the command center each calculate the reward value based on their respective reward functions and return it.

[0073] The reward function of the relay node simulation environment takes into account the communication quality and energy consumption requirements of the node at each stage under general circumstances, as well as the influence of other nodes and the requirements of the overall link. It is specifically expressed as:

[0074] r=r t +r p +r n +r ex

[0075] Where r t It is expressed as the sum of rewards or penalties obtained due to communication quality, r p Represents the sum of rewards or penalties due to motion power consumption, r n Indicates the reward obtained from the entire link or other nodes, r ex Represents the sum of other rewards or penalties, including the penalties received by the agent for violating the constraints, and the rewards or penalties received by the agent based on the final results after a round of training.

[0076] In the emergency communication scenario, the quality of the communication link must be guaranteed first. In the multi-relay node deployment problem, the relay node must first ensure the communication quality of the two links related to itself. Let the RSSI of the poorer link be R. Therefore, when designing the reward function, this paper divides it into three stages according to R: dangerous stage, adjustment stage, and rich stage. t , r p The calculation is as follows:

[0077] When R is less than a certain threshold value R1, that is, R0≤R<R1, the communication link quality is poor, the bit error rate increases, the communication quality decreases, the transmission rate decreases, and there is even a high risk of causing connection instability or interruption. Therefore, at this time, r t To avoid the situation where the relay node continues to receive a large penalty value due to the task node needing to perform a rescue mission at the edge of the reachable range, the ratio of the RSSI of the two links is introduced into the reward function. When the RSSI values ​​of the two links of the relay node are relatively balanced, the penalty value is appropriately reduced. The formula is expressed as:

[0078] r t =c(R-R1)·β1

[0079] where β1=α1r o / R1 is a fixed value added to normalize the reward and correct the slope. Increasing the link RSSI ratio c can also guide the agent to avoid getting too close to a node in the early stages of learning, which could cause the other end node to lose connection with the relay node.

[0080] When the link communication status is poor, in order to encourage the agent to find possible ways to get out of the predicament and obtain a relay link with better communication quality, the penalty given by the agent's energy consumption will also be appropriately reduced. The calculation formula is:

[0081] r e =k1e l,t

[0082] where e l,t is the accumulated energy at the current moment, and the calculation formula is:

[0083] e l,t =P(v,a)Δt+γ e e l,t-Δt

[0084] Where γ e (0<γ e <1) is the cumulative energy consumption discount. In order to allow the agent to consider the energy consumption, it can focus on the nearest 1 / γ e In the process of making decisions, the agent can consider the impact of the current decision on the next few steps.

[0085] When R is greater than R1 but less than a larger threshold value R2, the relay node can ensure the communication reliability between the task node and the command center with a certain margin, while optimizing the communication link to provide better support for task execution, reducing transmission delays in the overall relay link and improving overall communication efficiency. Therefore, the reward function gives it a positive reward value at this time, and it increases significantly with the increase of R. The specific formula is as follows:

[0086] r t =(R-R1)·β2

[0087] The normalization coefficient β2 = α2r0 / (R2-R1).

[0088] This state of good communication quality of the relay link is theoretically the state that the intelligent body will maintain for a long time. It is also the stage where the algorithm mainly needs to optimize learning. In this stage, the intelligent body also needs to consider the size of energy consumption during the movement. e The calculation formula is r e =k2e l , where k2>k1, the impact of energy consumption has a greater impact on the acquisition of reward value.

[0089] When R is greater than the threshold value R2, the node has obtained a large RSSI, and the relay link can be regarded as being able to meet the needs of all rescue tasks. At this time, it is difficult to achieve better results by reducing the node spacing or increasing the link quality by other means. Therefore, the reward value given by the reward function will be maintained at a high reward and will change less with the fluctuation of R. The specific formula is:

[0090] r t =α2r0+(R-R2)·β3

[0091] The normalization coefficient β3=α3r0 / (R max -R2).

[0092] At this time, the agent should pay more attention to the size of its energy consumption, so the reward function will be more sensitive to changes in energy consumption, r e =k3e l , where k3>k2.

[0093] In addition, the reward function will also penalize other inappropriate actions of the agent. For example, when the agent performs an action beyond the speed and acceleration limits, the penalty is calculated based on the magnitude of the excess limit.

[0094] r ex1 =α4(|Δa| 2 +|Δv| 2 )

[0095] Where Δa and Δv represent the magnitude of the acceleration and speed exceeding the limit. In addition, the agent will be given a termination reward or penalty based on the final termination state of the algorithm in this round of training. Assume that the environment has performed a total of n rounds of actions at the end, and the agent can perform at most n actions in each round. max Step training, if n<n max It means that the agent unexpectedly terminates this round of training due to other reasons. At this time, the reward function will give it a large penalty value, which is -α5r0. If all training rounds are successfully completed, it will receive a positive value as a reward, where α5 is a large constant, generally above ten.

[0096] And r n It is related to the overall link status and other nodes, and its calculation formula is:

[0097] r n =r nt +r nn

[0098] where r nt is the reward associated with the overall link state, r nnis the reward related to the action of adding or reducing nodes, i.e. the number of nodes, r nt The formula is:

[0099] r nt =(R n -R1)·β n

[0100] Where R n The RSSI of the link with the smallest RSSI in the entire link can reflect the status of the overall link to the greatest extent while facilitating collection. A fixed value added to normalize the reward and correct the slope, α n It is a fixed value preset according to task requirements and training stage. nn The calculation formula is:

[0101] r nn =[F del ·(max(T n ,0,T s ) / T s )-F ad ]β nn

[0102] Among them F del A flag to record whether to reduce the node, 1 if the reduction is successful, otherwise 0, F ad A flag to record whether a node is added, 1 if the addition is successful, otherwise 0, max(T n ,0,T s ) / T s Indicates the number of rounds the node survives. The more rounds the node survives, the more valuable it is and the more rewards it will receive. nn =r0α nn is the normalization coefficient.

[0103] The reward function of the command center simulation environment module takes into account the overall link status and the location of the task node, as well as the number of nodes in the current system. Its calculation formula is:

[0104] r o =r t +r nn +r ex

[0105] Where r t It is expressed as the sum of rewards or penalties obtained due to communication quality, r nn represents the reward related to the number of nodes in the relay link, r ex Indicates the reward given due to system termination or exceeding the limit.

[0106] r t The calculation formula is:

[0107] r t =(R n -R1)·β ot

[0108] Where R n The RSSI of the link with the smallest RSSI in the entire link can reflect the status of the overall link to the greatest extent while facilitating collection. A fixed value added to normalize the reward and correct the slope, α ot These are fixed values ​​preset according to task requirements and training stages.

[0109] r nn The reward is related to the number of nodes. Different from the calculation formula on the relay node side, the calculation formula on the command center side is:

[0110]

[0111] Where F del A flag to record whether to reduce the node, 1 if the reduction is successful, otherwise 0, F ad A flag to record whether a node is added, 1 if the addition is successful, otherwise 0. o1 =r0α nn is the normalization coefficient. r represents the distance between the task node and the command center, d0 represents the relay distance at which the relay node can maintain a good link, and β o2 =r0α o2 is the normalization coefficient.

[0112] In addition, other rewards are penalties for nodes that increase the limit r ex1 =α oex1 r0 and the reward r for successfully completing the entire round of training ex2 =α oex2 r0, and the penalty for unexpectedly ending this round of training -r ex2 .

[0113] The reward function of the command center takes into account the overall link status and the location of the task node, as well as the number of nodes in the current system. Its calculation formula is:

[0114] r o =r t +r nn +r ex

[0115] Where r t It is expressed as the sum of rewards or penalties obtained due to communication quality, r nn represents the reward related to the number of nodes in the relay link, r exIndicates the reward given due to system termination or exceeding the limit.

[0116] r t The calculation formula is:

[0117] r t =(R n -R1)·β ot

[0118] Where R n The RSSI of the link with the smallest RSSI in the entire link can reflect the status of the overall link to the greatest extent while facilitating collection. A fixed value added to normalize the reward and correct the slope, α ot These are fixed values ​​preset according to task requirements and training stages.

[0119] r nn The reward is related to the number of nodes. Different from the calculation formula on the relay node side, the calculation formula on the command center side is:

[0120]

[0121] The first half of the formula is similar to that on the relay node side, β ot =r0α o1 is the normalization coefficient, the second half d r represents the distance between the task node and the command center, d0 represents the relay distance at which the relay node can maintain a good link, and β o2 =r0α o2 is the normalization coefficient. In addition, other rewards are penalties for nodes that increase the limit r ex1 =α oex1 r0 and the reward r for successfully completing the entire round of training ex2 =α oex2 r0, and the penalty for unexpectedly ending this round of training -r ex2 .

[0122] In step 104, when storing experience, the memory pool module in the relay node location optimization method and the memory pool module in the command center node quantity management method both store the five-tuple (s, a, r, s', done) consisting of the current state S, the current action a, the next state S', and the reward value r, along with its priority weight m, into the memory pool. The priority weight of the experience is determined by the loss function value and the state of the relay node.

[0123] Step 105: When the agent extracts experience batches, it will randomly extract experience from the memory pool according to the priority weight, and its weight value will be reduced by 1 after the experience is used. When the experience pool is full, the experience with the lowest weight and the longest time will be eliminated first.

[0124] The structural diagram of the relay node intelligent agent module is as follows: Figure 3 As shown, the overall module comprises two neural networks: an action network (Actor) and a critic network (Critic). The action network inputs the state S observed by the agent and outputs the action A that maximizes the action value. The critic network inputs the state S and the action A and outputs the calculated action value Q. The ultimate goal of the algorithm is to train the Actor target network so that it responds appropriately to the state S in the environment and outputs the action that maximizes the cumulative action value. The training network is iteratively updated during each training session, as shown in the following formula.

[0125]

[0126]

[0127] As shown in the above formula, the gradient descent method is used each time the current action state value prediction is updated. and the next action value prediction The square of the difference between the sum of the current reward value r and the reward value r is smaller. Where s' is the next state, the agent module will select the smallest output from the two target evaluation networks for iterative optimization. The training action network is updated using gradient descent to minimize the action value of its own output action.

[0128] The structural diagram of the command center node quantity management agent is as follows: Figure 4 As shown in Figure 2, similar to the relay node position optimization agent, the node quantity control algorithm also has a training network and a target network. The network parameter update method is gradient descent. The purpose is to make the action value output by the training network approach the sum of the weighted predicted next action value output by the target network and the actual reward value. The specific formula is as follows:

[0129]

[0130] Where a is the learning rate, s', a' are the state of the next node and the action selected to maximize the action value, θ, θ' represent the parameters in the target network and the training network respectively.

[0131] Step 106: When updating the target network action network of the agent, a soft update method is used, that is, the parameters of the training network are added to the target network at a ratio of a smaller value τ. The expression is as follows:

[0132]

[0133] Step 107, reset the environment module, that is, when the relay link is disconnected or the current training round ends and the next round of training begins, the state parameters in the environment are reset, the task node position is reset to the command center, and the task node task target point is randomly set again.

[0134] See also Figure 5 , Figure 5 This is a structural diagram of a relay optimization device provided by an embodiment of the present invention, which is applied to a command center and includes:

[0135] The simulation environment module is used to change the environment state according to the output of the command center during simulation training, and calculate the reward value to feed back to the intelligent agent module.

[0136] The command center status acquisition module is used to collect the environmental parameters required by the input state of the intelligent agent module at each moment, perform normalization processing, and provide them to the intelligent agent module.

[0137] The command center intelligent agent module is used to manage actions based on the number of nodes that need to be executed according to the state output, and optimize its own output strategy based on the rewards given by the environment.

[0138] The decision-making component is relatively stable and is responsible for real-time interaction with the environment, outputting actions based on the observed state. The optimization component, after gaining a certain amount of experience in the memory pool, begins optimizing the agent's parameters to maximize its reward.

[0139] The command center memory pool module is used to store the action state reward at the current moment and the state at the next moment generated by the interaction between the command center agent module and the simulation environment module, and provide it to the agent for learning to improve the agent strategy.

[0140] See also Figure 6 , Figure 6 This is a structural diagram of a relay optimization device provided by an embodiment of the present invention, which is applied to a relay node and includes:

[0141] The simulation environment module is used to change the environment state according to the output of the command center during simulation training, and calculate the reward value to feed back to the intelligent agent module.

[0142] The relay node state acquisition module is used to collect the environmental parameters required by the input state of the intelligent agent module at each moment, perform normalization processing, and provide them to the intelligent agent module.

[0143] The relay node agent module is used to manage the number of nodes that need to be executed according to the state output, and optimize its own output strategy based on the rewards given by the environment.

[0144] The decision-making component is relatively stable and is responsible for real-time interaction with the environment, outputting actions based on the observed state. The optimization component, after gaining a certain amount of experience in the memory pool, begins optimizing the agent's parameters to maximize its reward.

[0145] The relay node memory pool module is used to store the action state reward at the current moment and the state at the next moment generated by the interaction between the command center agent module and the simulation environment module, and provide it to the agent for learning to improve the agent strategy.

[0146] In an embodiment of the present invention, the relay node intelligent agent optimizes its own strategy each time it outputs an action, so that the action state value it outputs is close to the actual situation in the environment, so that when selecting the optimal action, it can also get closer and closer to the optimal action under the actual situation, and ultimately enable the relay link to obtain higher link quality while maintaining smooth operation, lower operating energy consumption and a lower number of nodes while improving the first two optimization goals.

[0147] The device of the embodiment of the present invention is a device that applies the above relay optimization method, so all embodiments of the above relay user method are applicable to the device and can achieve the same or similar beneficial effects.

[0148] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A multi-relay node joint optimization method based on deep reinforcement learning, characterized by: Applied to a command center and relay nodes, the command center is a node that issues tasks to task nodes, the relay nodes are nodes that forward data, and the task nodes are nodes that specifically execute tasks. During the execution of tasks, they must maintain communication with the command center. The method includes: The number of nodes in the overall relay link is controlled by the deep reinforcement learning agent applied by the command center. The number management agent module adopts a multi-layer neural network to form a basic network. The input state space is a multi-dimensional continuously changing vector, which contains a lot of information collected by the command center state acquisition module to provide support for relay optimization. The state information collected by the state acquisition module at least includes state variables related to the link quality in the relay link, state variables related to the position of the relay node in the relay link, state variables related to the current number and number limit of the relay nodes in the relay link, and other state variables related to the number management of the relay nodes in the relay link. All the above state variables are normalized when used as inputs of the agent; its output action is to select an action from an action space containing at least three actions, and the action space at least includes: (1) sending a start-up signal to the new relay node; (2) sending a recovery signal to the relay node that departed recently; (3) monitoring the environmental state and maintaining the number of relay nodes unchanged, wherein the new relay node is a relay node that is dormant in the command center, and after receiving the start-up signal, it will start from the command center to execute the relay optimization method, form a relay link with other relay drones, and provide relay for the task node; The calculation formula on the command center side is: ； Where r nn is the reward related to the number of nodes, F del A flag to record whether to reduce the node, 1 if the reduction is successful, otherwise 0, F ad A flag to record whether a node is added, 1 if the addition is successful, otherwise 0; β o1 =r0α nn is the normalization coefficient; d r represents the distance between the task node and the command center, d0 represents the relay distance at which the relay node can maintain a good link, and β o2 =r0α o2 is the normalization coefficient; The relay node's movement is controlled by a deep reinforcement learning agent applied to the relay node to optimize its relay position. Its output action is at least a two-dimensional continuous variable representing its acceleration. The input state space is a multi-dimensional continuously changing vector containing a variety of information collected by the relay node state acquisition module to support relay optimization. By managing the number of relay nodes and optimizing their positions in the command center, as the task node gradually moves away from the command center to perform its task, the relay link can adjust the number of relay nodes in the link and the position of each relay node in real time, thereby optimizing the link quality of the overall communication link.

2. The multi-relay node joint optimization method based on deep reinforcement learning according to claim 1, characterized in that: Applied to relay nodes, the state information collected by the state acquisition module includes at least state variables related to the link quality in the relay link related to the relay node; state variables related to the positions of other nodes around the relay node; state variables related to the relay node's own factors, the said own factors include speed, acceleration, power consumption; other state variables related to the optimization of the relay node position; all the above state variables in the previous moment, wherein all state variables will be normalized when used as input to the intelligent agent.

3. The multi-relay node joint optimization method based on deep reinforcement learning according to claim 1, characterized in that: Applied to relay nodes, the relay node position optimization agent uses a multi-layer neural network as the basic network, and adopts multiple evaluation networks and multiple action networks as the whole. The action network inputs the relay node state S and outputs the action A. The evaluation network inputs S and A and outputs the calculated value Q. The two networks cooperate with each other to finally complete the training of the Actor target network, so that it can reasonably respond to the state S in the environment and output the action that can maximize the cumulative action value. The action output by the agent will be corrected through the action exploration module.

4. The multi-relay node joint optimization method based on deep reinforcement learning according to claim 1, characterized in that: The agent stores the experience data and weights of each step of the interaction between the agent and the environment into the memory pool, and randomly selects them according to the weights when the agent updates and iterates its own strategy. When the agent stores excessive new experience data, the experience with the minimum weight is eliminated. The weight is at least related to the state of the relay node and its contribution to the overall learning speed of the agent. The minimum weight is 1. After each experience is learned by the agent, its weight is reduced by 1 until it is reduced to 0.

5. The multi-relay node joint optimization method based on deep reinforcement learning according to claim 1, characterized in that: When the agent performs an action, the action exploration module will correct the final output action, so that the agent can strike a balance between exploring more possible actions and executing better actions. The degree of action correction is at least related to the results of the previous few action executions, the reward value and the current training stage, and gradually decreases during the training process.

Images