Multi-antenna charger mobile energy supplement method based on deep reinforcement learning

Abstract

The invention discloses a multi-antenna charger mobile energy supplement method based on deep reinforcement learning, and belongs to the field of wireless rechargeable sensor networks. In consideration of the problems of limited charging coverage of a single-antenna directional mobile charger (DMC) and timeliness of charging response of a sensor node, only charging scheduling of the multi-antenna DMC is considered in current research work, and the effect of improving the charging strategy by the multi-antenna DMC through dynamic adaptation and long-term optimization is not considered. According to the invention, the charging scheduling of the multi-antenna DMC is optimized based on a deep reinforcement learning algorithm DDQN, and the optimal charging cluster is intelligently selected and the charging scheduling strategy of the multi-antenna DMC is optimized at the same time by analyzing the request information of the sensor node and the network state. The all-dimensional charging coverage of the DMC on each cluster sensor node is realized, and the charging request of the sensor node is responded in time. The method provided by the invention can improve the overall charging efficiency of the network, reduce the death rate of the sensor nodes, and achieve the purpose of prolonging the service life of the wireless rechargeable sensor network.

Description

Technical Field

[0001] The present invention relates to the field of mobile energy replenishment of wireless sensor and actuator networks, and specifically to a mobile energy replenishment method of a multi-antenna charger based on deep reinforcement learning. Background Art

[0002] Wireless Rechargeable Sensor Networks (WRSN), which consists of base stations (BS), sensor nodes and mobile chargers (MC), have significant advantages in many fields such as industrial automation and smart cities. The base station is the main data receiving point in the network and is responsible for managing the configuration and maintenance of the entire sensor network. The sensor nodes are equipped with a variety of sensors to monitor and collect various physical or chemical parameters in the environment, and send the collected data to other sensor nodes or directly to the base station through wireless signals. The main task of the mobile charger is to provide energy replenishment for sensor nodes and extend the life cycle of the network. Sensor nodes are usually limited by the power of their internal batteries, so how to effectively manage and replenish the energy of these nodes has become a key issue. Using mobile chargers (MCs) in WRSN to charge nodes through wireless power transfer (WPT) is a potential solution. However, the charging efficiency of existing WPT technologies, especially those based on radio frequency (RF), decreases significantly with the increase of charging distance, which leads to low charging efficiency and energy utilization when the nodes are relatively dispersed or the number of nodes is large.

[0003] In "Optimizing Charging Efficiency and Maintaining Sensor Network Perpetually in Mobile Directional Charging" published in Sensors in 2019, Xianghua Xu et al. designed an approximate algorithm to determine the optimal charging point and charging direction of the mobile charging vehicle, aiming to minimize the number of charging points and maximize the coverage utility of the charging process. At the same time, an algorithm for planning the moving path of the mobile charging vehicle was proposed to maximize the energy replenishment efficiency of the mobile charging vehicle and extend the network life.

[0004] Yuzhu Liang et al. proposed a charging strategy that combines mobile charging with directional charging in the paper "Grouping Reduces Energy Cost in Directionally Rechargeable Wireless Vehicular and Sensor Networks" published in IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY in 2023. First, a greedy charging group generation algorithm is used, and then the minimum covering circle algorithm is used to find the optimal charging center for each charging group. At the charging center, the directional beam angle is rotated to replenish energy for the nodes in the charging group. Finally, the ant colony optimization algorithm is used to plan the optimal mobile path.

[0005] From the existing literature, most of the current research works use a directional mobile charger (DMC) equipped with one antenna to replenish energy for sensor nodes, but a single-antenna DMC can usually only provide energy in one direction or a limited area, and the charging coverage is limited. Charging sensor nodes in a cluster by rotating the antenna angle will prolong the charging time, causing low-energy nodes to die due to long-term lack of energy replenishment. In addition, from the existing literature, there is no research work using deep reinforcement learning methods to conduct multi-antenna DMC charging scheduling methods in wireless rechargeable sensor networks. Most of the existing multi-antenna DMC charging scheduling methods in wireless rechargeable sensor networks use basic algorithms to plan the moving path of the DMC. The moving path obtained by such methods is fixed and cannot adapt to the dynamic changes of the network environment. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned existing research work, the present invention fully considers the characteristics of wireless power transmission technology and the dynamics of wireless rechargeable sensor networks, and designs a multi-antenna charger mobile energy replenishment method based on deep reinforcement learning with the goal of maximizing energy transmission efficiency and minimizing DMC moving path.

[0007] The technical solution adopted by the present invention is: a multi-antenna charger mobile energy replenishment method based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Establish a wireless rechargeable sensor network: including a base station BS, n sensor nodes, and a mobile charger DMC equipped with multiple antennas;

[0009] Step 2: Use clustering algorithm to cluster the sensor nodes and obtain the charging point of DMC by finding the geometric center of the cluster;

[0010] Step 3: Set the cluster charging priority. That is, the more sensor nodes that need to be charged in the cluster, the higher the cluster charging priority. When the energy of the sensor node is lower than the sensor node charging threshold e th,i When the charging request is sent to the base station, the base station sends the information of the cluster where the sensor node to be charged is located to the DMC. The cluster with high priority is more likely to get energy replenishment first. The charging model is based on the Friis free space path loss model.

[0011] Step 4: The DMC is equipped with three antennas with adjustable directions. The DMC moves to the charging point and stays there. The minimum covering dominating set algorithm is used to determine the number and direction of antennas to replenish energy for the sensor nodes in the cluster. The goal is to use the least number of antennas to cover all sensor nodes in the cluster.

[0012] Step 5: As the energy of the sensor nodes is consumed, the DMC determines the next charging cluster through the deep reinforcement learning algorithm DDQN, and moves to the charging point to replenish the energy for the sensor nodes. When the remaining energy of the DMC is not enough to support the next charging task, the MC suspends charging and returns to the base station to replenish energy.

[0013] Specifically, the wireless rechargeable sensor network in Step 1: sensor nodes are randomly deployed in a square area of ​​L*L. The task of the sensor nodes is to perform sensing and communication operations and finally aggregate the data to the base station. The sensor nodes are equipped with rechargeable batteries with a maximum battery capacity of E s When the battery power of a sensor node is lower than that of an adjacent sensor node, the sensor node uses a multi-hop routing protocol, and when the battery power is higher than that of an adjacent sensor node, the sensor node uses a single-hop routing protocol to send data directly to the BS. The base station is deployed at the center of the network and schedules the DMC to replenish energy for the clusters in the charging queue according to the list to be charged. When a cluster is fully charged, its number is removed from the charging queue. The DMC travels at a constant speed with a maximum battery capacity of E. m , the mobile energy consumption is ec m J / m.

[0014] Specifically, in Step 2, the coordinates of sensor node i are known, the clustering radius is R, and the mean-shift algorithm is used to cluster the sensor nodes, which is calculated as follows:

[0015]

[0016] Among them, (x i ,y i ) is the current coordinate of sensor node i, (x i ′,y i ′) is the updated coordinate, (x j,y j ) is the coordinate of the neighbor node j of sensor node i; N(x i ) and N(y i ) represents the set of neighbor nodes of sensor node i in the x and y coordinate directions; K is the kernel function, and the Gaussian kernel is selected; g is the bandwidth parameter, which controls the influence range of the kernel function.

[0017] Specifically, in Step 3, the sensor node charging threshold e th,i It is used to determine whether the energy of a single sensor node is sufficient to maintain its operation. When the energy of a sensor node is lower than this threshold, the sensor node needs to send a charging request to the base station. Each sensor node determines the charging threshold according to its energy consumption rate, which is calculated as follows:

[0018]

[0019] in, represents the energy consumption rate of sensor node i, represents the average waiting time of sensor node i.

[0020] Specifically, in Step 3, the charging model is as follows:

[0021]

[0022] Among them, P R is the received power, G T and G R are the transmitting antenna gain and the receiving antenna gain, L p ,γ,λ,β,α,D represent polarization loss, efficiency of rectifier in receiving module, RF wavelength, parameter of Friis free space equation for adjusting short distance transmission, path loss coefficient, distance between transmitter and receiver, P respectively. T is the transmitting power;

[0023] Each sensor node generates data and transmits the data to the BS at the center of the network. The battery-aware multi-hop energy consumption model is as follows:

[0024]

[0025] Among them, r i is the energy consumption rate of the i-th sensor node, is a set of sensor nodes, k represents Select a sensor from is the data flow rate from the kth sensor to the ith sensor, f i j is the data flow rate from the i-th sensor to the j-th sensor, f i BSis the data flow rate from the i-th sensor to the BS, ρ is the energy consumption rate for receiving one data unit, so the receiving energy consumption of the i-th sensor is and are the energy consumption rates of transmitting a data unit from the i-th sensor to the j-th sensor and to the BS, respectively.

[0026] Specifically, in Step 4, the beam coverage angle of each antenna is 120°. The specific steps of using the minimum covering dominating set algorithm to determine the number of antennas and their directions are as follows:

[0027] A: The coordinates of the known sensor node i are (x i ,y i ), the coordinates of sensor node k are (x k ,y k ), construct the coverable connection between sensor nodes in the cluster, the formula is as follows:

[0028]

[0029] Where r is the maximum communication range of the antenna. If d(i,k)≤r, an edge is established between sensor node i and sensor node k.

[0030] B: Determine the minimum dominating set, set an empty dominating set M and a covered node set, select a sensor node from the uncovered sensor nodes, and add it to the dominating set M until all sensor nodes are covered. The formula is as follows:

[0031]

[0032] Where D(M) represents the set of sensor nodes dominated by the set M, N(v) is the set of neighbors of sensor node v, and V is the set of all sensor nodes.

[0033] C: Optimize the antenna direction. For each sensor node in the set M, calculate the optimal coverage direction. The calculation formula is as follows:

[0034]

[0035] Among them, θ u represents the optimal direction angle of sensor node u; N(u) is the neighbor set of sensor node u; θ is a temporary variable used to traverse possible starting angles; angle(u,v) represents the direction angle from sensor node u to sensor node v; by applying the minimum covering dominating set algorithm, the coverage angle of each antenna is accurately adjusted to the optimal level to ensure that the antenna's 120° sector coverage includes as many sensor nodes as possible, thereby improving the data collection efficiency and energy utilization of the overall network.

[0036] Specifically, in Step 5, the deep reinforcement learning algorithm DDQN is used to optimize the charging strategy. The specific process is as follows:

[0037] State space S

[0038] S= DMC ,S node1 ,S node2 ,…,S node|n| >

[0039] S DMC = <x DMC ,y DMC ,RE DMC >

[0040] S nodei = <x i ,y i ,RE i >

[0041] The state space consists of the states of n sensor nodes (node ​​coordinates and residual energy) and the state of a DMC (coordinates and residual energy), where S DMC The horizontal coordinate x of DMC at time t is represented by DMC , the vertical coordinate y DMC and residual energy RE DMC The triplet formed by S nodei Represents the horizontal coordinate x of sensor node i i , the vertical coordinate y i , residual energy RE i The triplet composed of S node|n| Indicates the state of the nth sensor node;

[0042] Action Space A

[0043] A={a i |i=0,1,…,cluster_num}

[0044] a i Indicates that the i-th charging cluster is selected for charging, a cluster_num Indicates the action of choosing to return to the base station for charging;

[0045] Reward function R

[0046]

[0047] Among them, distance and number_dead_step are the distance moved and the number of dead nodes after the DMC executes one step of decision, ω represents the weight coefficient of the moving distance, σ is the penalty factor for the death of the sensor node, and the total reward R obtained by the DMC after a round of scheduling is completed​total for:

[0048] R total =∑R (10)

[0049] Where T is the number of action steps executed by DMC in one round of scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a model diagram of the wireless rechargeable sensor network of the present invention;

[0051] Figure 2 This is the DDQN network architecture diagram of the present invention;

[0052] Figure 3 This is a flow chart of the mobile energy replenishment method of a multi-antenna charger based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0053] In order to describe the present invention in more detail and facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings and embodiments. The embodiments in this part are used to explain the present invention for the purpose of facilitating understanding, and are not intended to limit the present invention.

[0054] Embodiment 1: A method for mobile energy replenishment of a multi-antenna charger based on deep reinforcement learning, comprising the following steps:

[0055] Step 1: Please see Figure 1 , establish a wireless rechargeable sensor network model: hundreds of sensor nodes are randomly deployed in a certain monitoring area. The base station is deployed at the center of the network. Clusters are divided according to the density of sensor nodes, and the charging point of each cluster is determined. represents the set of sensor nodes; E s Represents the battery capacity of the sensor node; E m Indicates the battery capacity of DMC; RE i (t) represents the battery energy of sensor node i at time t; RE m (t) represents the remaining battery energy of the DMC at time t; ec i represents the energy consumption rate of sensor node i; ec m represents the energy consumption rate of DMC; C represents the set of clusters; P represents the set of charging points; e th,i Represents the charging threshold of the sensor node.

[0056] Initially, sensor node n i Randomly deployed in the L*L square area, the main task of the sensor is to perform sensing and communication operations, and finally aggregate the data to the base station. The sensor node is equipped with a rechargeable battery with a maximum battery capacity of E sWhen the battery power of a sensor node is lower than that of an adjacent sensor node, the sensor node uses a multi-hop routing protocol, and when the battery power is higher than that of an adjacent sensor node, a single-hop routing protocol is used to send data directly to the BS. The base station is deployed at the center of the network and schedules the DMC to replenish energy for the clusters in the charging queue according to the list to be charged. When a cluster is fully charged, its number is removed from the charging queue. The DMC travels at a constant speed with a maximum battery capacity of E m , the mobile energy consumption is ec m J / m.

[0057] Step 2: Please see Figure 1 ,The mean-shift algorithm is used to cluster the sensor nodes, and the charging point of the DMC is obtained by finding the geometric center of the cluster.

[0058] Given the coordinates of sensor node i and the clustering radius R, the mean-shift algorithm is used to cluster the sensor nodes. The calculation is as follows:

[0059]

[0060] Among them, (x i ,y i ) is the current coordinate of sensor node i, (x i ′,y i ′) is the updated coordinate, (x j ,y j ) is the coordinate of the neighbor node j of sensor node i; N(x i ) and N(y i ) represents the set of neighbor nodes of sensor node i in the x and y coordinate directions; K is the kernel function, and the Gaussian kernel is selected; h is the bandwidth parameter, which controls the influence range of the kernel function.

[0061] from Figure 1 It can be seen that after clustering the sensor nodes by the mean-shift algorithm, the sensor nodes in the network are clustered into single-charge clusters (containing one sensor node) and multi-charge clusters (containing multiple sensor nodes). For the single-charge cluster, the location of the sensor node is the DMC charging location, and the DMC directly moves to the location of the sensor node to replenish its energy; for the multi-charge cluster, the centroid of the cluster is the DMC charging location, and the multi-antenna DMC with directional charging function is used to replenish the energy of the sensor nodes.

[0062] Step 3: Set cluster charging priority, that is, the more sensor nodes that need to be charged in the cluster, the higher the cluster charging priority. Specifically, when the energy of the sensor node is lower than the sensor node charging threshold, a charging request is sent to the base station. The base station sends the information of the cluster where the sensor node to be charged is located to the DMC. The cluster with a high priority is more likely to get energy replenishment first. The charging model is based on the Friis free space path loss model.

[0063] The specific parameter settings are shown in the following table.

[0064] Table 1 Parameter settings

[0065]

[0066]

[0067] Sensor node charging threshold e th,i It is used to determine whether the energy of a single sensor node is sufficient to maintain its operation. When the energy of a sensor node is lower than this threshold, the sensor node needs to send a charging request to the base station. Each sensor determines the charging threshold based on its energy consumption rate, which is calculated as follows:

[0068]

[0069] in, represents the energy consumption rate of sensor i, represents the average waiting time of sensor i.

[0070] The charging model is as follows:

[0071]

[0072] Among them, P R is the received power, G T and G R are the transmitting antenna gain and the receiving antenna gain respectively. p ,γ,λ,β,α,D represent polarization loss, efficiency of rectifier in receiving module, RF wavelength, parameter of Friis free space equation for adjusting short distance transmission, path loss coefficient, distance between transmitter and receiver, P respectively. T is the transmission power.

[0073] Each sensor node generates data and transmits the data to the BS at the center of the network. The battery-aware multi-hop energy consumption model is as follows:

[0074]

[0075] Among them, r i is the energy consumption rate of the i-th sensor node, is a set of sensor nodes, k represents Select a sensor from is the data flow rate from the kth sensor to the ith sensor, f i j is the data flow rate from the i-th sensor to the j-th sensor, f i BS is the data flow rate from the i-th sensor to the BS, ρ is the energy consumption rate for receiving one data unit, so the receiving energy consumption of the i-th sensor is and are the energy consumption rates of transmitting a data unit from the i-th sensor to the j-th sensor and to the BS, respectively.

[0076] Step 4: Please see Figure 1 ,The DMC is equipped with three antennas with adjustable directions, and the beam coverage angle of each antenna is 120°. The DMC moves to the charging point and resides there, and the minimum covering dominating set algorithm is used to determine the number of antennas and their directions to replenish energy for the sensor nodes in the cluster, with the goal of using the least number of antennas to cover all the sensor nodes in the cluster.

[0077] The specific steps of using the minimum covering dominating set algorithm to determine the number of antennas and their directions are as follows:

[0078] A: The coordinates of the known sensor node i are (x i ,y i ), the coordinates of sensor node k are (x k ,y k ), construct the coverable connection between sensor nodes in the cluster, the formula is as follows:

[0079]

[0080] Where r is the maximum communication range of the antenna. If d(i,k)≤r, an edge is established between sensor node i and sensor node k.

[0081] B: Determine the minimum dominating set, set an empty dominating set M and a covered sensor node set, select a sensor node from the uncovered sensor nodes, add it to the dominating set M, until all sensor nodes are covered. The formula is as follows:

[0082]

[0083] Where D(M) represents the set of sensor nodes dominated by the set M, N(v) is the set of neighbors of sensor node v, and V is the set of all sensor nodes.

[0084] C: Optimize the antenna direction. For each sensor node in the set M, calculate the optimal coverage direction. The calculation formula is as follows:

[0085]

[0086] Among them, θ u represents the optimal direction angle of sensor node u; N(u) is the neighbor set of sensor node u; θ is a temporary variable used to traverse possible starting angles; angle(u,v) represents the direction angle from sensor node u to sensor node v. By applying the minimum covering dominating set algorithm, the coverage angle of each antenna is precisely adjusted to the optimal value, ensuring that the antenna's 120° sector coverage includes as many sensor nodes as possible, thereby improving the data collection efficiency and energy utilization of the overall network.

[0087] Step 5: Please see Figure 2 ,As the energy of the sensor nodes is consumed, the DMC determines the next charging cluster through the deep reinforcement learning algorithm DDQN, and moves to the charging point to recharge the sensor nodes. When the remaining energy of the DMC is insufficient to support the next charging task, the MC suspends charging and returns to the base station to recharge the energy.

[0088] The deep reinforcement learning algorithm DDQN is used to optimize the charging strategy. The specific process is as follows:

[0089] State space S

[0090] S= DMC ,S node1 ,S node2 ,…,S node|n| >

[0091] S DMC = <x DMC ,y DMC ,RE DMC >

[0092] S nodei = <x i ,y i ,RE i >

[0093] The state space consists of the states of n sensor nodes (node ​​coordinates and residual energy) and the state of a DMC (coordinates and residual energy). DMC The horizontal coordinate x of DMC at time t is represented by DMC , the vertical coordinate y DMC and residual energy RE DMC The triplet formed by S nodei Represents the horizontal coordinate x of sensor node i i ​, the vertical coordinate y i , residual energy RE i The triplet composed of S node|n| Represents the state of the nth sensor node.

[0094] Action Space A

[0095] A={a i |i=0,1,…,cluster_num}

[0096] a i (for i=0 to cluster_num) indicates that the i-th charging cluster is selected for charging. cluster_num Indicates the action of selecting to return to the base station for charging.

[0097] Reward function R

[0098]

[0099] Among them, distance and number_dead_step are the distance moved and the number of dead nodes after the DMC executes one step of decision, ω represents the weight coefficient of the moving distance, and σ is the penalty factor for the death of the sensor node. The total reward R obtained by the DMC after a round of scheduling is completed total for:

[0100] R total =∑R (10)

[0101] Where T is the number of action steps executed by DMC in one round of scheduling.

[0102] The experience replay buffer is used to store quadruplets generated by interactions with the environment, from which samples are randomly drawn to break data correlation. The prediction network is used to estimate the Q value in the current state and select the corresponding action. The target network estimates the maximum action for the next step to calculate the target Q value. The loss is calculated by the Q value difference, and the parameters of the prediction network are updated using back propagation. The target network is periodically updated to the parameters of the prediction network. Finally, the DMC selects a charging cluster or returns to the base station for charging based on the Q value, thereby optimizing the charging path. This process is continuously adjusted through the DDQN algorithm to achieve the optimal charging decision.

[0103] The above is only the specific idea of ​​the present invention, which is for the convenience of understanding by researchers in this field. However, the implementation mode of the present invention is not limited to the above. Relevant technicians in this field can make improvements or modifications based on the present invention. All improvements or modifications based on the concept of the present invention are deemed to be within the protection scope of the present invention.

Claims

1. A multi-antenna charger mobile energy replenishment method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Establish a wireless rechargeable sensor network: including a base station BS, n sensor nodes, and a mobile charger DMC equipped with multiple antennas; Step 2: Use clustering algorithm to cluster the sensor nodes and obtain the charging point of DMC by finding the geometric center of the cluster; Step 3: Set the cluster charging priority. That is, the more sensor nodes that need to be charged in the cluster, the higher the cluster charging priority. When the energy of the sensor node is lower than the sensor node charging threshold e th,i When the charging request is sent to the base station, the base station sends the information of the cluster where the sensor node to be charged is located to the DMC. The cluster with high priority is more likely to get energy replenishment first. The charging model is based on the Friis free space path loss model. Step 4: The DMC is equipped with three antennas with adjustable directions. The DMC moves to the charging point and stays there. The minimum covering dominating set algorithm is used to determine the number and direction of antennas to replenish energy for the sensor nodes in the cluster. The goal is to use the least number of antennas to cover all sensor nodes in the cluster. Step 5: As the energy of the sensor nodes is consumed, the DMC determines the next charging cluster through the deep reinforcement learning algorithm DDQN, and moves to the charging point to replenish the energy for the sensor nodes. When the remaining energy of the DMC is not enough to support the next charging task, the MC suspends charging and returns to the base station to replenish energy.

2. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: Wireless rechargeable sensor network in Step 1: sensor nodes are randomly deployed in the square area of ​​L*L. The task of the sensor nodes is to perform sensing and communication operations and finally aggregate the data to the base station. The sensor nodes are equipped with rechargeable batteries with a maximum battery capacity of E. s When the battery power of a sensor node is lower than that of an adjacent sensor node, the sensor node uses a multi-hop routing protocol, and when the battery power is higher than that of an adjacent sensor node, the sensor node uses a single-hop routing protocol to send data directly to the BS. The base station is deployed at the center of the network and schedules the DMC to replenish energy for the clusters in the charging queue according to the list to be charged. When a cluster is fully charged, its number is removed from the charging queue. The DMC travels at a constant speed with a maximum battery capacity of E. m , the mobile energy consumption is ec m J / m.

3. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: In Step 2, the coordinates of sensor node i are known, the clustering radius is R, and the mean-shift algorithm is used to cluster the sensor nodes. The calculation is as follows: Among them, (x i ,y i ) is the current coordinate of sensor node i, (x′ i ,y′ i ) is the updated coordinate, (x j ,y j ) is the coordinate of the neighbor node j of sensor node i; N(x i ) and N(y i ) represents the set of neighbor nodes of sensor node i in the x and y coordinate directions; K is the kernel function, and the Gaussian kernel is selected; h is the bandwidth parameter, which controls the influence range of the kernel function.

4. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: In Step 3, the sensor node charging threshold e th,i It is used to determine whether the energy of a single sensor node is sufficient to maintain its operation. When the energy of a sensor node is lower than this threshold, the sensor node needs to send a charging request to the base station. Each sensor node determines the charging threshold according to its energy consumption rate, which is calculated as follows: in, represents the energy consumption rate of sensor node i, represents the average waiting time of sensor node i.

5. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: In Step 3, the charging model is as follows: Among them, P R is the received power, G T and G R are the transmitting antenna gain and the receiving antenna gain, L p ,γ,λ,β,α,D represent polarization loss, efficiency of rectifier in receiving module, RF wavelength, parameter of Friis free space equation for adjusting short distance transmission, path loss coefficient, distance between transmitter and receiver, P respectively. T is the transmitting power; Each sensor node generates data and transmits the data to the BS at the center of the network. The battery-aware multi-hop energy consumption model is as follows: Among them, r i is the energy consumption rate of the i-th sensor node, is a set of sensor nodes, k represents Select a sensor from is the data flow rate from the kth sensor to the ith sensor, f i j is the data flow rate from the i-th sensor to the j-th sensor, f i BS is the data flow rate from the i-th sensor to the BS, ρ is the energy consumption rate for receiving one data unit, so the receiving energy consumption of the i-th sensor is and are the energy consumption rates of transmitting a data unit from the i-th sensor to the j-th sensor and to the BS, respectively.

6. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: In Step 4, the beam coverage angle of each antenna is 120°. The specific steps for determining the number of antennas and their directions using the minimum covering dominating set algorithm are as follows: A: The coordinates of the known sensor node i are (x i ,y i ), the coordinates of sensor node k are (x k ,y k ), construct the coverable connection between sensor nodes in the cluster, the formula is as follows: Where r is the maximum communication range of the antenna. If d(i,k)≤r, an edge is established between sensor node i and sensor node k. B: Determine the minimum dominating set, set an empty dominating set M and a covered node set, select a sensor node from the uncovered sensor nodes, add it to the dominating set M, until all sensor nodes are covered. The formula is as follows: Where D(M) represents the set of sensor nodes dominated by the set M, N(v) is the set of neighbors of sensor node v, and V is the set of all sensor nodes; C: Optimize the antenna direction. For each sensor node in the set M, calculate the optimal coverage direction. The calculation formula is as follows: Among them, θ u represents the optimal direction angle of sensor node u; N(u) is the neighbor set of sensor node u; θ is a temporary variable used to traverse possible starting angles; angle(u,v) represents the direction angle from sensor node u to sensor node v; By applying the minimum covering dominating set algorithm, the coverage angle of each antenna is precisely adjusted to the optimal level, ensuring that as many sensor nodes as possible are included in the 120° sector coverage of the antenna.

7. The multi-antenna charger mobile energy replenishment method based on deep reinforcement learning according to claim 1 is characterized in that: In Step 5, the deep reinforcement learning algorithm DDQN is used to optimize the charging strategy. The specific process is as follows: State space S S=<S DMC ,S node1 ,S node2 ,…,S node|n| > S DMC =<x DNC ,y DMC ,RE DMC > S nodei =<x i ,y i ,RE i > The state space consists of the states of n sensor nodes and one DMC, where S DMC The horizontal coordinate x of DMC at time t is represented by DMC , the vertical coordinate y DMC and residual energy RE DMC The triplet formed by S nodei Represents the horizontal coordinate x of sensor node i i , the vertical coordinate y i , residual energy RE i The triplet composed of S node|n| Indicates the state of the nth sensor node; Action Space A A={a i |i=0,1,…,cluster_num} a i Indicates that the i-th charging cluster is selected for charging, a cluster_num Indicates the action of choosing to return to the base station for charging; Reward function R Among them, distance and number_dead_step are the distance moved and the number of dead nodes after the DMC executes one step of decision, ω represents the weight coefficient of the moving distance, σ is the penalty factor for the death of the sensor node, and the total reward R obtained by the DMC after a round of scheduling is completed total for: R total =ΣR (10) Where T is the number of action steps executed by DMC in one round of scheduling.

Images