Unmanned aerial vehicle self-adaptive charging method based on reinforcement learning and dormancy mechanism

Abstract

The invention discloses an unmanned aerial vehicle self-adaptive charging method based on reinforcement learning and a dormancy mechanism. The method provides a wireless charging model different from a traditional wireless charging model. The system utility is maximized by considering the energy consumption of the sensing nodes and the hovering position of the unmanned aerial vehicle. The system effectiveness of the invention not only comprises maximization of the total amount of sensor network data. The minimization of the number of death nodes and the minimization of the flight distance of the unmanned aerial vehicle are also optimization targets. On one hand, reinforcement learning and a dormancy strategy based on a matrix completion algorithm are combined, and the survival time of the sensing nodes is greatly prolonged; and on the other hand, a self-adaptive charging capacity strategy of the sensing equipment is provided according to the data collection capability of different sensing equipment. A sensing node in the Internet of Things system adjusts parameters such as alarm electric quantity and rated electric quantity in real time according to the number of events in the current area. Finally, through a large number of simulation experiments, compared with a common method, the method provided by the invention can improve system benefits by 40% or more.

Description

Technical Field

[0001] The present invention relates to the fields of Internet of Things, smart city sensor networks and group intelligence sensing sensor networks, and in particular to adaptive charging planning of unmanned aerial vehicles based on reinforcement learning and sleep mechanisms. Background Art

[0002] Taking the IoT sensor network as an example, in the IoT sensor network of smart cities, basic data is mainly provided by sensing devices distributed in various areas. These sensing devices are regarded as sensing nodes in the IoT sensor network. In order to ensure the normal operation of these nodes, they must be maintained regularly. However, due to the limited energy of the sensing nodes, once the energy is exhausted, the data in the node coverage area cannot be effectively collected, which seriously affects the data collection ability of the entire network. In order to solve this problem and extend the effective working time of the sensing nodes, various energy replenishment schemes have been proposed, such as wireless power supply by drones, power supply by mobile charging vehicles, and manual battery replacement. Among them, wireless power supply by drones has become the main energy replenishment method in IoT sensor networks. First, with the help of wireless charging technology, it effectively avoids the problem of frequent battery replacement. Secondly, wireless power supply by drones greatly reduces the operating risks of workers in dangerous areas. For example, in high-risk industries such as chemical, oil and gas, wireless charging reduces the need to lay cables and perform maintenance in flammable and explosive environments, thereby reducing the risk of safety accidents. In addition, compared with mobile charging vehicles, drones can provide power for sensor nodes deployed at high altitudes, solving the charging problem that is difficult to reach by ground power supply means. Finally, wireless power supply for drones has greater flexibility, does not rely on fixed ground battery replacement piles, and can be deployed freely.

[0003] The typical process of wireless power supply for drones is as follows: the drone starts from the base station and wirelessly powers the sensor nodes in the network one by one. In this process, the drone's energy must support its own flight and power the sensing nodes. Although wireless power supply technology is quite mature, the limited energy reserves of drones still pose challenges to the continuous power supply of IoT sensor networks. For example, when the drone's energy is about to run out, it must return to the base station or a nearby power station in time for energy replenishment. Due to the limited energy storage capacity of drones, there is a risk that all sensor nodes cannot be wirelessly powered within the effective time. Therefore, drone wireless charging is still a challenging research area in IoT sensor networks.

[0004] Therefore, the present invention provides a dormant adaptive charging method for UAV based on reinforcement learning, aiming to solve the problem of how to design a reasonable charging path under limited energy conditions to maximize the system utility. Summary of the invention

[0005] The present invention provides a UAV adaptive charging method based on reinforcement learning and sleep mechanism, aiming to solve the problem of how to design a reasonable charging path under limited energy conditions to maximize the system utility. The system utility of the present invention not only includes maximizing the amount of data collected by the IoT sensor network during the working cycle, but also optimizes the minimization of the number of dead nodes and the minimization of the UAV flight distance.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] In response to the challenges in existing research, this paper proposes a drone adaptive charging method based on reinforcement learning and sleep mechanism, with the goal of maximizing system utility. Its main innovations are summarized as follows:

[0008] An adaptive power strategy for sensor devices is proposed: the drone adaptively adjusts the maximum charging capacity of the sensor node according to the data collection rate of the sensor node. This greatly shortens the traditional single-node charging time and enables more nodes to be charged in time. If the sensor node has sufficient power, it will not enter the drone's charging candidate queue, thereby giving priority to ensuring the power supply of key location nodes and significantly extending the overall life of the IoT sensor network.

[0009] A dormant charging strategy for sensor devices is proposed: To address the problem of sensor nodes losing their data collection capabilities after power failure, a dormant strategy is proposed to put nodes that meet the dormant conditions into a low-power waiting mode until they are charged. This strategy significantly improves the utility of the system and extends the overall life of the IoT sensor network.

[0010] A UAV adaptive charging method based on reinforcement learning and sleep mechanism is proposed: by combining reinforcement learning strategy, sensor equipment adaptive power strategy and sleepable charging strategy, the UAV can obtain the optimal charging path, thereby maximizing the system benefit.

[0011] The implementation process of a drone adaptive charging method based on reinforcement learning and sleep mechanism can be briefly shown as follows:

[0012] Step 1: The IoT sensor network sets the adaptive safety state, adaptive alarm state, adaptive pre-sleep power and death state of the sensor device according to the adaptive charging strategy of the sensor device; the system collects the state information of the sensor device in the IoT sensor network in real time;

[0013] Step 2: When the power of the sensor device reaches the adaptive pre-sleep state power threshold set by the adaptive charging strategy, the sensor device applies to the system for sleep, and the system judges the sleep application according to the dormant charging strategy of the sensor device;

[0014] Step three, combining reinforcement learning strategy, adaptive charging strategy of sensor equipment and dormant charging strategy, the flight path of the drone is planned to maximize the amount of data collected by the IoT sensor network; the drone starts from the base station according to the planned flight path that maximizes the system benefits, performs data collection and wireless power supply in sequence, and finally returns to the base station.

[0015] Finally, the effectiveness of the present invention is verified through a large number of simulations. The experimental results show that a drone adaptive charging method based on reinforcement learning and sleep mechanism can significantly improve the system utility and meet the requirements of maximizing the system utility. A large number of simulation results prove that this scheme greatly improves the performance and life of the IoT sensor network.

[0016] In summary, the present invention studies a method of using drones to replenish energy for sensor nodes in a sensor network. The goal is to maximize the total amount of data collected by the system. A new charging scheme is proposed. First, an adaptive power strategy for sensor nodes is proposed. The sensor nodes in the Internet of Things system adjust the alarm power, rated power and other parameters in real time according to the number of events in the current area. The adaptive power strategy enables drones to quickly respond to sensor nodes to be charged. In addition, we combine reinforcement learning technology with the sleep mechanism to greatly improve the survival time and work efficiency of sensor nodes. Compared with general methods, the advantages of the present invention are:

[0017] (1) The method of the present invention has higher system utility when facing different numbers of sensor devices. The flight path selected using the method of the present invention can achieve higher system utility. When the number of sensor devices is small, it can be found that the energy of the drone can meet the needs of the sensor network, and the present invention is not much different from the general method. However, as the scale of the network continues to expand, the advantages of the present invention are greatly reflected, and the system utility of the present invention is significantly improved by 40% compared with the general method.

[0018] (2) The drone using the method of the present invention has lower energy consumption. The flight path selected using the method of the present invention can achieve higher system utility under the same energy consumption. Under the same drone energy, the system utility of the present invention is significantly higher than that of the general method. This shows from the side that when achieving the same system benefit, the energy required by the present invention is lower.

[0019] The method of the present invention can be more flexible to meet the actual needs of the sensor network. In a specific implementation environment, the energy consumption rate of the sensor equipment may be affected by outdoor climate such as temperature and humidity. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a schematic diagram of a scene of a model of an embodiment of the present invention.

[0021] Figure 2 Schematic diagram of the operating principle of the model of the embodiment of the present invention.

[0022] Figure 3 The figure is a comparison diagram of the system effectiveness of the method of the present invention and the general method in the embodiment of the present invention with different numbers of sensor nodes.

[0023] Figure 4 This is a comparison chart of the system effectiveness of the method of the present invention and the general method in the embodiments of the present invention under different initial energies carried by drones.

[0024] Figure 5 The figure is a comparison chart of the system effectiveness of the method of the present invention and the general method in the embodiment of the present invention under different energy consumption rates of the sensor equipment. DETAILED DESCRIPTION

[0025] The present invention is further described below with reference to the accompanying drawings and examples. The embodiments of the present invention are based on the following model:

[0026] Figure 1 The schematic diagram of the scenario of the model of the embodiment of the present invention. The model of the present invention includes three key parts: wireless power supply equipment drone, roadside intelligent sensor equipment and local base station. The drone starts from the local base station and selects the best path to perform the task according to the drone dormant adaptive charging planning strategy based on reinforcement learning. The drone hovers at the designated hovering charging position of each sensor device in turn to perform wireless power supply and data collection.

[0027] The present invention proposes an adaptive charging method for unmanned aerial vehicles based on reinforcement learning and a sleep mechanism. The main optimization measures include: (1) Adaptive charging strategy for sensor devices: First, the state of sensor devices in the sensor network is set through this strategy. This measure effectively improves the probability of sensor devices obtaining energy in a timely manner and provides sufficient data support for the subsequent path planning of the unmanned aerial vehicle. (2) Dormant strategy for sensor devices: Without affecting the overall quality of the sensor network, the survival time of the sensor device is further extended through the sleep strategy. This strategy causes the sensor node to enter a low-power mode when the energy is insufficient, and resumes normal operation when charging. (3) An adaptive charging method for unmanned aerial vehicles based on reinforcement learning and a sleep mechanism: By combining the strategy of reinforcement learning and a sleep mechanism, the unmanned aerial vehicle can plan a flight path that maximizes the system utility. This path not only increases the amount of data collected, but also reduces the number of node deaths and the flight distance of the unmanned aerial vehicle. The system utility can be expressed as follows:

[0028]

[0029] where ω 1 ,ω 2 ,ω3 , is the weight, reflecting the importance of each attribute in the utility function. Since the system hopes to collect as much data as possible. 1 The total amount of data received by the base station during the working cycle is DS, the total flight distance of the drone is DD, and the total number of dead nodes in the sensor network is DN. Represents sensor node Optimal wireless power supply hovering position and sensor nodes The optimal wireless power supply hovering position spatial distance. ζ(i) Represents the total amount of data collected by the i-th sensor node. N represents the number of nodes in the sensor network. is an indicator function. When sensor node i dies, otherwise

[0030] The specific steps for implementing the present invention are introduced below in conjunction with specific examples.

[0031] Step 1: Set different states for each sensor device according to the sensor device's adaptive charging strategy.

[0032] In the adaptive charging strategy of sensor devices, sensor nodes can be divided into adaptive safety state, adaptive alarm state, adaptive pre-sleep state, and death state. The specific division of each state is distinguished by an adaptive power threshold:

[0033]

[0034] in,

[0035]

[0036] in Represents the battery level of the sensor node, Represents the control coefficient of the adaptive alarm power, represents the control coefficient of the adaptive power, Represents the control coefficient of adaptive sleep power, D i represents the data collection rate of the sensor nodes, is a constant, E op(i) Represents the adaptive rated charging capacity of the sensor device.

[0037] Step 2: According to the dormant strategy of the sensor device. In each round of UAV charging, if the power of the sensor node is less than the dormant power set by the dormant strategy of the sensor device, the system hides the data value of the data matrix corresponding to the sensor node, uses the matrix filling algorithm to complete the matrix, and compares it with the original data matrix to calculate the error between the completed matrix and the real matrix. The error The calculation is expressed as follows:

[0038]

[0039] If its error Less than or equal to the system setting The system will allow this node to sleep. rl(i) represents the true value of the sensor device data matrix, D sl(i) Represents the fill value of the sensing device.

[0040] Step 3, plan a flight path for the drone that maximizes the system utility. A drone adaptive charging method based on reinforcement learning and sleep mechanism. Figure 2 The schematic diagram of the operating principle of the model of the embodiment of the present invention. Based on the state set by the adaptive charging strategy of the sensor device, the overall state of the sensor network is input into the environment of the reinforcement learning model, and the system model uses convolution scanning to read the state of the environment. And through the deep Q learning network, it is continuously trained, and after each action is selected, the sensor device can be dormant. The sensor that meets the conditions is put into dormancy and the environmental state is updated. Finally, the UAV flight path that maximizes the system utility is obtained.

[0041] Among them, the state space of this model is represented by S network:

[0042] S network={s 1 ,s 2 ,..,s i ,...,s N}

[0043] where s i It includes four parts: the dormant state of the sensor node, the alarm state, the remaining power, and the distance from the drone to node i. The drone is allowed to charge the sensor node only when it is in the alarm state and the remaining power is greater than 0. When the sensor node is in the dormant state, its data is obtained using the matrix filling algorithm, and the power consumption is the dormant power. Therefore, the state of the MDP problem is a 4*N+1 vector, where N represents the number of sensor nodes in the IoT sensor network.

[0044] The action space of this model is composed of N sensor nodes in the IoT perception network. Selecting the i-th node means that the next charging object of the drone is the i-th sensor node of the corresponding IoT perception network. The action space can be briefly described as follows:

[0045] action={a|a∈{1,2,...,i,...N}}

[0046] Reward return: The goal of the system is to maximize the amount of data collected, so the reward return is related to the amount of data that the current surviving sensor node can collect. si The amount of data will affect the reward. At the same time, the system hopes that the drone moves less distance so that the sensor node can obtain more energy. And the number of dead nodes in the system should be as small as possible. Therefore, the drone is in state s i The instantaneous reward after taking action As shown below:

[0047]

[0048] Among them, λ 1 ,λ 2 ,λ 2 are the corresponding weights. After the drone takes action from node i, the moving distance to reach node j is d ij When the drone takes action from node i and arrives at node j for charging, the number of dead nodes in the sensor network is DN ij .

[0049] The drone agent constantly senses the environmental changes of the IOT network, and each action taken will receive a corresponding reward, thereby continuously updating the environmental state. The total reward obtained by the drone agent will be calculated in the final state. Through the learned DQN neural network function, the drone agent can collect as many rewards as possible to obtain the best charging path.

[0050] Step 4: The drone wirelessly powers the sensor devices and collects data in turn according to the flight path planned to maximize the system utility. When the drone battery reaches the alarm value, it returns to the base station to replenish energy and prepare for the next wireless power supply and data collection.

[0051] Figure 3 The figure is a comparison diagram of the system effectiveness of the method of the present invention and the general method in the embodiment of the present invention with different numbers of sensor nodes. Figure 3 The experimental results of the system utility in the comparative experiment using the method of the present invention and the general method are compared. When the number of sensor devices is small, it can be found that the energy of the drone can meet the needs of the sensor network, and the difference between the present invention and the general method is small. However, as the scale of the network continues to expand, the advantages of the present invention are greatly reflected, and the system utility of the present invention is significantly improved by 40% compared with the general method.

[0052] Figure 4 This is a comparison chart of the system effectiveness of the method of the present invention and the general method in the embodiment of the present invention under different UAV carrying initial energies. Figure 4It can be seen that under the same drone energy, the system utility of the present invention is significantly higher than that of the general method. This shows from the side that when achieving the same system benefit, the energy required by the present invention is lower.

[0053] exist Figure 5 It can be seen that even in the case of different energy consumption of the sensor equipment, the system benefit of the present invention is still better than the general method to a certain extent. It can be seen that the system benefit is still maximized in the case of extreme energy consumption of the sensor equipment.

Claims

1. A UAV adaptive charging method based on reinforcement learning and sleep mechanism, characterized by: The following steps are involved: Step 1: The IoT sensor network sets the adaptive safety state, adaptive alarm state, adaptive pre-sleep power and death state of the sensor device according to the adaptive charging strategy of the sensor device; The system collects the status information of sensor devices in the IoT sensor network in real time; Step 2: When the power of the sensor device reaches the adaptive pre-sleep state power threshold set by the adaptive charging strategy, the sensor device applies to the system for sleep, and the system judges the sleep application according to the dormant charging strategy of the sensor device; Step 3: Combine the reinforcement learning strategy, the adaptive charging strategy of the sensor equipment and the dormant charging strategy to plan the flight path of the drone to maximize the amount of data collected by the IoT sensor network; the drone starts from the base station according to the planned flight path that maximizes the system benefits, performs data collection and wireless power supply in sequence, and finally returns to the base station.

2. According to the method for adaptive charging of a drone based on reinforcement learning and a dormancy mechanism according to claim 1, it is characterized in that the specific steps of calculating the adaptive charging state of the sensor device in step 1 are: According to the adaptive charging strategy of the sensor device, the sensor node can be divided into adaptive safety state, adaptive alarm state, adaptive pre-sleep state, and death state. Each state is divided according to an adaptive power threshold: S_state (i) Indicates the current state of the sensor device with serial number i; safe state indicates that the sensor device is in a safe state and has sufficient power; alarm state indicates that the sensor device is in an alarm state and can enter the UAV power supply sequence; pre_sleep state indicates that the sensor device is in a pre-sleep state; dead state indicates that the sensor device has entered a sleep state due to exhaustion of power; (i) Indicates the remaining power of the sensor device i; Eal (i) Represents the alarm power threshold of the adaptive strategy of the sensor device; E sl(i) represents the pre-sleep power threshold of the adaptive strategy of the sensor device; the optimal charging power threshold E of the adaptive strategy of the sensor device i op(i) , Alarm power threshold E al(i) And the calculation of the pre-sleep power threshold is as follows: Represents the battery level of the sensor node, Represents the control coefficient of the adaptive alarm power, represents the control coefficient of the adaptive power, Represents the control coefficient of adaptive sleep power, D i represents the data collection rate of the sensor nodes, is a constant, E op(i) Represents the adaptive rated charging capacity of the sensor device.

3. According to the method of adaptive charging of unmanned aerial vehicles based on reinforcement learning and sleep mechanism in claim 1, it is characterized in that in step 2, the sensor device applies for sleep state, and the specific method for the system to approve or reject the sleep application of the sensor device is: In each round of drone charging, if the power of a sensor node is lower than the dormant power threshold set by the dormant charging strategy of the sensor device, the system will hide the data value of the data matrix corresponding to the sensor node; After hiding the data values, the system uses a matrix filling algorithm to complete the matrix and compares the completed matrix with the original data matrix; Calculate the error between the completed filled matrix and the real matrix To determine whether the sensor device has entered a dormant state, the error The calculation is expressed as follows: If the error Less than or equal to the system setting The system will allow this node to sleep.

4. The method for adaptive charging of a drone based on reinforcement learning and sleep mechanism according to claim 1, characterized in that: In step 3, the reinforcement learning strategy is used, combined with the adaptive charging strategy of the sensor equipment and the dormant charging strategy of the sensor equipment to plan the path of the drone, and finally the flight path of the drone that maximizes the data collection amount of the IoT sensor network is obtained; the specific path planning method is: The deep Q learning algorithm is used as the main body of the UAV path planning. For the state of the sensor network, the state of each sensor device is set through the adaptive charging strategy of the sensor device. When the current battery of the UAV is too low, it is regarded as the intelligent agent stops planning and this training ends.

Images