Rechargeable unmanned aerial vehicle data acquisition method based on enhanced pointer network

Abstract

The invention discloses a rechargeable unmanned aerial vehicle data acquisition method based on an enhanced pointer network, and the method comprises the steps: firstly constructing a rechargeable unmanned aerial vehicle scene, and obtaining the related data of an unmanned aerial vehicle; secondly, constructing a pointer network short-term prediction model, performing state encoding on the acquired data, and inputting a result after state encoding into an encoder to obtain an intermediate variable; hidden layer information is obtained through the intermediate variable by using a decoder, and the action of the unmanned aerial vehicle at the next moment is obtained by combining an attention mechanism with short-term prediction until all sensor data are collected. And finally, training the pointer network short-term prediction model by utilizing a strategy gradient method, generating an actor network and a reviewer network, and calculating an update gradient by utilizing a reward function to continuously update so as to converge rewards. According to the invention, for the data acquisition problem of the rechargeable unmanned aerial vehicle, the subsequent decision behavior is effectively optimized, and the probability of making a better behavior strategy under the current electric quantity condition of the unmanned aerial vehicle is increased.

Description

Technical Field

[0001] The present invention relates to the fields of drone data collection, deep reinforcement learning, and more specifically, to a rechargeable drone data collection method based on a reinforcement pointer network. Background Art

[0002] In recent years, with the rapid development of Internet of Things technology, wireless sensor networks use multiple sensors to obtain surrounding perceptible information and analyze and process the information, improving the efficiency and real-time performance of data acquisition. When natural disasters cause ground equipment to fail to connect, or when ground communication is difficult in complex terrain, drones provide a flexible and efficient method for collecting sensor data, which can reduce the energy consumption of transmitting data, as well as the deployment and maintenance costs of wireless sensors. Under limited energy constraints, drones will plan an optimal flight path based on mission requirements and environmental information, and visit the sensor collection nodes and charging piles in order.

[0003] As a complex combinatorial optimization problem, UAV path planning has attracted the attention of researchers at home and abroad in recent years. This problem can be modeled as a variant of the traveling salesman problem. In early studies, researchers tended to use precise algorithms to solve such problems. Common solutions include dynamic programming, branch and bound, and linear programming, etc. However, as the scale of the problem increases, there may be a dimension explosion problem, and it is difficult to grasp the optimal balance between the speed of solution and the optimal solution, making this problem of great research value. A large number of intelligent optimization algorithms have been proposed, such as ant colony algorithm, particle swarm algorithm, genetic algorithm, taboo search algorithm, simulated annealing algorithm, etc. Although intelligent optimization algorithms have shown certain advantages in dealing with problems, they also face the challenges of complexity of state transition and relatively slow optimization speed. Summary of the invention

[0004] The purpose of the present invention is to address the shortcomings of existing rechargeable UAV data collection methods and propose a rechargeable UAV data collection method based on enhanced pointer network combined with short-term prediction, which effectively improves the collection efficiency.

[0005] The present invention mainly solves the problem of data collection of rechargeable drones by the following method: the rechargeable drone data problem is modeled as a variant of the traveling salesman problem, and through the combination of pointer network and reinforcement learning, short-term prediction is used to improve the effect of the solution and shorten the calculation time of the solution.

[0006] The specific implementation steps of the present invention are as follows:

[0007] S1: Build a rechargeable drone scene and obtain drone-related data. This includes obtaining the distribution of sensors and charging piles, and drone indicators: power consumption per meter during flight, flight distance per second, power consumption when collecting unit data, data volume collected per second, and drone charging rate during charging; obtain dynamic information of drones and sensors at every moment, including the location of the drone, the current power of the drone, and the remaining data volume of the sensor point.

[0008] S2: Build a pointer network short-term prediction model, perform state encoding on the data obtained in S1, input the state encoding result into the encoder to obtain the intermediate variable; then use the decoder to obtain the hidden layer information through the intermediate variable, and use the attention mechanism combined with short-term prediction to obtain the action of the drone at the next moment, and repeat this cycle until all sensor data are collected. Specifically include:

[0009] S2.1: Take the current battery level and remaining data of the drone as a dynamic input set in Represents the power e of the drone at the current time point t t and the remaining data As a dynamic input collection.

[0010] The charging pile and sensor position coordinate set is taken as the static input set N = (n0, n1, ..., n n ), where there is a charging station n0 and n sensors collecting data; static input and dynamic input components are obtained The output result of the state encoding As input to the encoder module.

[0011] S2.2: The encoder module uses the first embedding layer composed of a convolutional network to Collection Mapping to a one-dimensional vector As an intermediate variable input into the decoder, After the first embedding layer, we get n i , After the embedding layer, we get

[0012] S2.3: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information h t , and the current UAV position information x at each time step t After the second embedding layer consisting of a convolutional network, we get At each time step of decoding, the hidden layer information h tand the position of the drone at that moment Use the attention mechanism combined with the short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the sensor number that the drone will go to for data collection or return to the charging station for charging at the next moment; finally, a greedy strategy is used to select the action with the maximum probability value as the drone's next behavior.

[0013] S3: Use the policy gradient method to train the pointer network short-term prediction model, adopt the Actor-Critic framework, generate the actor network and the critic network, use the reward function to calculate the update gradient to continuously update the network, and finally make the reward converge.

[0014] Beneficial effects of the invention: The invention combines the results of short-term prediction fusion in the algorithm, and aims at the data collection problem of rechargeable UAVs, so that the UAVs can take their own power factors into consideration when making decisions, which can effectively optimize their subsequent decision-making behaviors and increase the probability of the UAVs making better behavior strategies under the current power conditions. At the same time, the invention also reduces the time of understanding and improves the effect of understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a short-term prediction model diagram of the pointer network of the present invention;

[0016] Figure 2 It is the algorithm flow chart of the present invention;

[0017] Figure 3 This is a visualization effect diagram of the short-term prediction of the deep enhanced pointer network of the present invention;

[0018] Figure 4 This is a comparison chart of the short-term prediction convergence effect of the deep enhanced pointer network of the present invention;

[0019] Figure 5 This is a comparison chart of the time effect of solving the short-term prediction learning algorithm of the deep reinforcement pointer network of the present invention. DETAILED DESCRIPTION

[0020] The specific implementation modes of the present invention are described in more detail below with reference to the accompanying drawings.

[0021] Although the following detailed description provides a deep understanding of the present invention, the present invention can be implemented in a variety of different ways, which may be different from those described herein. Professionals can make appropriate adjustments and changes based on the core principles of the present invention without departing from its essence. Therefore, the scope of protection of the present invention should not be limited to the specific embodiments detailed herein.

[0022] The details include:

[0023] S1: Build a scenario and obtain data: In a search area, obtain a location distribution N=(n0,n1,...,n n ), where i∈(1,2,...,n), the amount of remaining data for each sensor is d i , the drone is v x Flying at a constant speed and v d The transmission rate is v p The ideal charging efficiency is used to charge the drone; at the same time, the dynamic information of the drone and the sensor is obtained at every moment, including the location of the drone, the current power, and the remaining data volume of the sensor point.

[0024] S2: Construct a pointer network short-term prediction model, such as Figure 1 As shown. By encoding the data obtained in S1, the encoded result is input into the encoder to obtain an intermediate variable. The decoder obtains the hidden layer state through the intermediate variable, and the hidden layer and the dynamic information of the current drone are combined with the attention mechanism to obtain the action of the drone at the next moment. This cycle continues until all sensor data are collected.

[0025] Specifically include:

[0026] S2.1: Input status code:

[0027] State encoding: The current battery level and remaining data of the drone are used as a dynamic input set in Represents the power e of the drone at the current time point t t and the remaining data As a dynamic input set. The power and location information of the drone in this set will change with time t. The charging pile and sensor location coordinate set is taken as the static input set N = (n0, n1, ..., n n ), the data in this set will not change with the change of time t. Static input and dynamic input components are obtained The output result of the state encoding As input to the encoder module.

[0028] S2.2: Encoder module: The encoder module uses a first embedding layer consisting of a convolutional network to Collection Mapped to a one-dimensional vector As an intermediate variable input into the decoder, After the first embedding layer, we get n i , After the embedding layer, we get This module connects the states of each point and weakens the association between the points, making the output result independent of the sequence representation of the input state.

[0029] S2.3: Decoder module: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information h t , and the current UAV position information x at each time step t After the second embedding layer consisting of a convolutional network, we get At each time step of decoding, the hidden layer information h t and the position of the drone at that moment Use the attention mechanism combined with the short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the sensor number that the drone will go to for data collection or return to the charging station for charging at the next moment. Finally, the greedy strategy is used to select the action with the maximum probability value as the next moment behavior of the drone. The short-term prediction model is constructed as follows.

[0030] S2.3.1: Construction of short-term prediction model: By obtaining the location and power information of the drone at the current time t, the power consumed by all possible behaviors of the drone at the next time t+1 is calculated. The remaining power information matrix and position information matrix of all possible actions of the drone at time t+1 are calculated, and then the remaining power matrix of the drone at two time steps is obtained by calculating the remaining power after returning to the charging station at time t+2. in

[0031] In the formula, e t The current power of the drone. The power consumed by the charging pile is returned at time t+2. It is the remaining power for the actions made at that moment for the short-term prediction of the future.

[0032] S2.3.2 Attention Mechanism: Static Input and Dynamic Input Set at Time t after Embedding and the decoder output state h at decoding step t t , calculate the alignment vector

[0033] Where v a , W a are the training parameters.

[0034] The conditional probability is calculated by weighted accumulation combined with the context vector of the input sequence:

[0035] The value calculated using the embedded input and context vector is then normalized through the Glimpse mechanism softmax function to output the probability Combine short-term predictions to output the final action probability distribution in is the result obtained by accumulating and weighting the attention mechanism results and passing through the Glimpse mechanism, λ is a hyperparameter, and v c , W c is the training parameter.

[0036] S3: Use the policy gradient method to train the network model. The Actor-Critic framework is used to generate an actor network and a critic network. The critic network evaluates the effect of the actor network and collaboratively updates the network parameters. The weight vector associated with the actor network is θ, and the weight vector associated with the critic network is φ. During training, m sample problems are extracted from M groups of data, and Monte Carlo simulation is used to generate feasible sequence solutions relative to the current policy. The network uses the reward function to calculate the update gradient to continuously update the network parameters, and finally converge the reward.

[0037] The critic network update gradient formula is Where R i is the reward value obtained by solving the sequence, Represents the critic network's response to the i-th sample sequence Estimate the value of the reward received.

[0038] The actor network update gradient formula is In the formula For the sequence The output of the solution Y i The probability product of the drone’s behavior selection.

[0039] The reward function design includes:

[0040] S3.1: Design reward function. In order to avoid the impact of sparse rewards on network updates, the reward value assigned by the strategy is as follows: R = aT r +bT h +cT c +r

[0041] The parameters in the formula can be calculated through the dynamic information and indicators of the drone at each moment: Indicates the total time the drone takes to fly. π represents the total length of the distance flown by the UAV under this solution sequence; It represents the time required for the drone to hover over the sensor and collect data in a single flight. π It represents the amount of data that the drone needs to collect to complete its mission; Indicates the time it takes for the drone to be fully charged at the charging station, E π represents the sum of all the electricity consumed by the drone to complete the task; a, b, c are hyperparameters, which are the proportional weights of the reward function, and r is the penalty given to the drone when completing the task with negative electricity.

[0042] The algorithm flow chart of the present invention is as follows Figure 2 visible.

[0043] The present invention builds a model for the data collection problem of rechargeable drones and uses deep reinforcement learning methods for training and solving. The trained model can directly perform end-to-end reasoning on the input sequence without the need for traditional iterative solution steps, combined with the idea of ​​short-term prediction, thereby significantly improving the efficiency of problem solving and the effect of understanding.

[0044] The results of the present invention are visualized as shown in FIG. Figure 3 As shown, Figure 3 The result graph of 50 sensor collection points, where the five-pointed star represents the location of the charging pile, the dot represents the location of the sensor, the dotted arrow represents the direction of the drone's flight, and the number above the sensor represents the remaining power of the drone after it flies to the location and completes the collection task. Figure 4 It can be seen that (hereinafter, the pointer network is referred to as PN, and the method of the present invention is referred to as SF-PN), where Table 1 shows the reward values ​​of the two algorithms after convergence and the effect improved by the present invention under different numbers of sensors (10-50). The smaller the reward value, the less time the drone takes to complete the data collection task:

[0045] Table 1

[0046]

[0047] When the number of sensors is small, the SF-PN algorithm will be in a relatively stable state from the beginning of training; when the number of sensors increases, the SF-PN algorithm and the PN algorithm will have a rapid convergence process in the first few rounds, and the performance of the SF-PN algorithm is better than that of the PN algorithm. In scenarios of the same scale, the convergence speed and effect of the SF-PN algorithm are better than those of the PN algorithm. The convergence rate of the algorithm has increased by more than 10%, and the optimal result of the algorithm has increased by 7% to 17%. This is due to the fusion of short-term predictions in the algorithm, which increases the probability that the drone will make a better behavioral strategy considering the current power level. Figure 5It can be seen that the computational time of the proposed SF-PN algorithm is lower than that of the PN algorithm, and the computational efficiency is significantly improved with the increase in the number of sensors.

Claims

1. A rechargeable drone data collection method based on an enhanced pointer network, characterized in that: The following steps are involved: S1: Build a rechargeable drone scene and obtain drone-related data; S2: Build a pointer network short-term prediction model, perform state encoding on the data obtained in S1, and input the state encoding result into the encoder to obtain the intermediate variable; then use the decoder to obtain the hidden layer information through the intermediate variable, and use the attention mechanism combined with short-term prediction to obtain the action of the drone at the next moment until all sensor data are collected; S3: Use the policy gradient method to train the pointer network short-term prediction model, adopt the Actor-Critic framework, generate the actor network and the critic network, and use the reward function to calculate the update gradient to continuously update and converge the reward.

2. The rechargeable drone data collection method based on enhanced pointer network according to claim 1 is characterized in that: The drone-related data includes obtaining the distribution locations of sensors, the distribution locations of charging piles, drone indicators, and dynamic information of drones and sensors at every moment; The drone indicators include: power consumption per meter during flight, flight distance per second, power consumption when collecting unit data, data volume collected per second, and drone charging rate during charging; The dynamic information of the drone and the sensor at each moment includes: the location of the drone, the current power of the drone, and the remaining data volume of the sensor point.

3. The rechargeable drone data collection method based on enhanced pointer network according to claim 2 is characterized in that: The specific implementation process of step S2 is as follows: S2.1: Take the current battery level and remaining data of the drone as a dynamic input set in Represents the power e of the drone at the current time point t t and the remaining data As a dynamic input set, the power and location information of the drones in this set change with time t; The charging pile and sensor position coordinate set is taken as the static input set N = (n0, n1, ..., n n ), where there is a charging station n0 and n sensors collecting data. The data in this set does not change with the change of time t; static input and dynamic input components are obtained The output result of the state encoding As input to the encoder module; S2.2: The encoder module uses the first embedding layer composed of a convolutional network to Collection Mapping to a one-dimensional vector As an intermediate variable input into the decoder, After the first embedding layer, we get n i , After the embedding layer, we get S2.3: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information ht, and the current drone position information x at each time step is t After the second embedding layer consisting of a convolutional network, we get At each time step of decoding, the hidden layer information h t and the position of the drone at that moment Use the attention mechanism combined with the short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the sensor number that the drone will go to for data collection or return to the charging station for charging at the next moment; finally, a greedy strategy is used to select the action with the maximum probability value as the drone's next behavior.

4. The rechargeable drone data collection method based on enhanced pointer network according to claim 3 is characterized in that: The short-term prediction model is implemented as follows: by obtaining the current time t drone position and power information, the power consumed by all possible behaviors of the drone at the next time t+1 is calculated. The remaining power information matrix and position information matrix of all possible actions of the drone at time t+1 are calculated, and then the remaining power matrix of the drone at two time steps is obtained by calculating the remaining power after returning to the charging station at time t+2. in In the formula, e t The current power of the drone. The power consumed by the charging pile is returned at time t+2. The remaining power for the short-term prediction of the future at that moment; The attention mechanism is implemented as follows: through the static input and dynamic input set at time t after embedding and the decoder output state h at decoding step t t , calculate the alignment vector Where v a , W a are the training parameters; The conditional probability is calculated by weighted accumulation combined with the context vector of the input sequence: The value calculated using the embedded input and context vector is then normalized through the Glimpse mechanism softmax function to output the probability Combine short-term predictions to output the final action probability distribution in is the result obtained by accumulating and weighting the attention mechanism results and passing through the Glimpse mechanism, λ is a hyperparameter, and v c , W c is the training parameter.

5. The rechargeable drone data collection method based on enhanced pointer network according to claim 4 is characterized in that: The reward function is R = aT r +bT h +cT c +r; The parameters in the formula are calculated through the dynamic information and indicators of the drone at each moment: T r Indicates the total time the drone takes to fly; T h It represents the time required for the drone to hover over the sensor and collect data in a single flight; T c It represents the time it takes for the drone to be fully charged at the charging station; a, b, c are hyperparameters, which are the proportional weights of the reward function; r is the penalty given to the drone when completing the task with negative power.

Images