Unmanned aerial vehicle network multi-hop data processing method

Abstract

The invention discloses an unmanned aerial vehicle network multi-hop data processing method. The method comprises the following steps: step 1, constructing an unmanned aerial vehicle network data processing mathematical model; 2, distributing computing resources by adopting improved reinforcement learning aiming at the constructed mathematical model; step 3, optimizing the allocated computing resources by adopting an improved grey wolf optimization algorithm; and 4, carrying out shared information updating on the optimized computing resources based on a consensus method, and updating the average return estimation of the unmanned aerial vehicle by calculating a learning progress and a consensus weight by adopting a learning-based consensus method. According to the unmanned aerial vehicle network data processing mathematical model, the calculation process can be simplified, the algorithm optimization efficiency can be improved, the improved deep reinforcement learning model can balance task processing and unloading strategies, the optimal overall performance is achieved in the aspects of delay and energy efficiency, gradient estimation variance is reduced, and the algorithm optimization efficiency is improved. A learning-based consensus method enables the unmanned aerial vehicle to calculate global information, and benefits from experience of other unmanned aerial vehicles to more comprehensively understand the environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle data processing, and in particular to a method for processing multi-hop data in an unmanned aerial vehicle network. Background Art

[0002] Smart city is a new form of urban management that integrates information technology, intelligent social management and services. It creates a safe, comfortable and convenient modern living environment for urban residents with the help of new generation information technologies such as cloud computing, mobile Internet and Internet of Things. While meeting the needs of citizens, smart city seeks a friendly interactive relationship between citizens, local services and infrastructure, provides citizens with detailed information on the status of the city, promotes interaction between citizens and urban infrastructure, and improves the service efficiency of government departments.

[0003] Intelligent transportation system is one of the main building blocks of smart cities. In the past two decades, it has become an effective way to improve the performance of transportation systems, enhance travel safety and provide more choices for travelers. In intelligent transportation systems, drones play an important role. They can move within the monitoring area of ​​the sensor network, collect information from sensors, buffer data, and then transmit it to the data management center of the intelligent transportation system, thereby greatly improving the efficiency of data transmission.

[0004] Compared with traditional wireless networks, the mobility of drones is unique. Designing a suitable mobility solution can significantly reduce message conflicts and losses, reduce the burden of node data forwarding, evenly distribute the network energy consumption burden, and collect data faster. Compared with ground data collectors, drones have a higher movement speed, which can increase the access speed of nodes, shorten the data processing delay, have higher bandwidth, and have fewer obstacles during data collection, a larger wireless signal coverage range, and can reduce communication delays and the energy consumed by sensor data transmission. These characteristics of drones bring obvious advantages to data processing and greatly expand the application field of wireless sensor networks.

[0005] At present, many experts and scholars have conducted extensive research on multi-hop data processing methods for UAV networks, and have made great breakthroughs in this field. However, most of these methods have defects, such as inaccurate UAV network data processing models, high network latency, and high energy consumption for processing tasks. Summary of the invention

[0006] In view of this, the present invention proposes a drone network multi-hop data processing method to solve the problems existing in the prior art of inaccurate drone network data processing model, high network delay, and high energy consumption of processing tasks.

[0007] The specific technical solutions of the present invention are as follows:

[0008] A method for processing multi-hop data in a drone network, comprising:

[0009] Step 1: Construct a mathematical model for UAV network data processing. The operation of UAVs in free space is considered during construction. It is assumed that there is a line-of-sight path between UAVs. The model is constructed by analyzing the channel gain between UAVs, data transmission rate, energy and computing power of UAVs, energy consumption when processing and unloading tasks, and duration of data processing.

[0010] Step 2: Improved reinforcement learning is used to allocate computing resources for the constructed mathematical model. Improved reinforcement learning defines an optimization problem with different elements and allocates rewards according to the contribution of the drone's offloading strategy to the task delay and energy consumption, so as to achieve a balance between task processing and offloading strategy.

[0011] Step 3: optimize the allocated computing resources using an improved gray wolf optimization algorithm. The gray wolf optimization algorithm integrates the sine-cosine algorithm into the basic gray wolf optimization algorithm. The gray wolves in the basic gray wolf optimization algorithm have specific social hierarchy and group behavior characteristics.

[0012] Step 4: Update the shared information of the optimized computing resources based on the consensus method. Use the learning-based consensus method to update the average return estimate of the drone by calculating the learning progress and consensus weight, and then update the reward function of deep reinforcement learning.

[0013] Specifically, in step one, when constructing a mathematical model for UAV network data processing, the situation in which the UAV operates in a free space at a considerable height from the ground and there is a line-of-sight path is considered. The model is constructed by analyzing the relationship between the channel gain between UAVs and various factors, the relationship between the data transmission rate and related factors, the relationship between the total energy consumption and various factors, and the relationship between the processing data duration and related factors. The total energy consumption is related to the energy consumption of the data processing part and the energy consumption of unloading data, and the data processing duration is related to the local processing time and the unloading time.

[0014] Specifically, in step 2, in the step of allocating computing resources using improved reinforcement learning, the optimization problem of UAV network data processing is defined as a form containing multiple elements, where the state space defines the environment state, the action space consists of discrete action sets of the UAV, and the discrete action sets represent two situations: complete data offloading or full data processing. The state utility value in a given action is determined based on multiple factors, and rewards are allocated based on the contribution of each UAV's offloading strategy to minimizing the overall delay and energy consumption of completing the task. An effective balance between task processing and offloading strategies is achieved through a UAV strategy containing multiple elements to achieve the best overall performance in terms of delay and energy efficiency.

[0015] Specifically, in step 2, in the step of allocating computing resources using improved reinforcement learning, the improved deep reinforcement learning model adopted can effectively balance task processing and offloading strategies, achieve the best overall performance in terms of latency and energy efficiency, and can reduce the variance in gradient estimation by introducing specific values.

[0016] Specifically, in step three, in the step of optimizing computing resources using an improved gray wolf optimization algorithm, the gray wolves in the improved gray wolf optimization algorithm include four types: alpha wolf, beta wolf, delta wolf and omega wolf. They follow a strict social hierarchy and have the main characteristics of a group in hunting, searching, besieging and attacking prey.

[0017] Specifically, in step three, in the step of optimizing computing resources using the improved gray wolf optimization algorithm, the computing resources are optimized by integrating the sine-cosine algorithm into the basic gray wolf optimization algorithm and using a specific position update formula and an improved Alpha Wolf motion formula.

[0018] Specifically, in step four, in the shared information update step based on the consensus method, a learning-based consensus method is adopted. First, the difference between the current and previous average return estimates of each drone is calculated to obtain the learning progress, and then the consensus weight is calculated according to the learning rates of the two drones. The consensus weight is updated using a weight smoothing factor related to the consensus matrix of the relationship between the drone and other drones in the environment.

[0019] Specifically, in step four, the shared information update step based on the consensus method, each drone updates its average reward estimate according to the calculated consensus weight, taking into account the weighted sum of rewards of all drones, with the weights coming from the consensus matrix.

[0020] Specifically, in step four, in the shared information update step based on the consensus method, the drone estimates the updated average reward and updates the reward function of deep reinforcement learning based on local observations, minimizing the difference between the estimated reward and the actual reward value by iteratively adjusting the parameters.

[0021] Specifically, in step four, the consensus-based shared information update step, the adopted learning-based consensus method enables the drones to calculate global information and benefit from the experience of other drones.

[0022] The beneficial effects of the present invention are:

[0023] 1. The established mathematical model for drone network data processing can simplify the calculation process and greatly improve the efficiency of algorithm optimization.

[0024] 2. The improved deep reinforcement learning model can effectively balance task processing and offloading strategies to achieve the best overall performance in terms of latency and energy efficiency, and the introduction of baseline values ​​can reduce the variance in gradient estimation.

[0025] 3. A learning-based consensus method is adopted. Given that the local information of the reinforcement learning algorithm alone depends on the local average reward of all drones, which is insufficient to calculate the global estimate of the action value and advantage function, this method enables drones to calculate global information, benefit from the experience of other drones, and thus have a more comprehensive understanding of the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 The present invention is a flowchart of a method for processing multi-hop data in an unmanned aerial vehicle network. DETAILED DESCRIPTION

[0028] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] The present invention proposes a method for multi-hop data processing in a drone network. First, a mathematical model for drone network data processing is established. Then, the established mathematical model for drone network data processing is allocated computing resources using improved reinforcement learning. Next, the allocated computing resources are optimized using an improved gray wolf optimization algorithm. Finally, the optimized computing resources are updated with shared information based on a consensus method. The specific steps of the method of the present invention are as follows:

[0030] Step 1: Establish a mathematical model for drone network data processing.

[0031] Since the UAVs are operating in free space at a considerable height from the ground, the present invention assumes that there is a line-of-sight path between the UAVs and the channel gain h between the two UAVs i and j is i，j (t) can be expressed as in, It represents the proportional factor of signal attenuation due to path loss in free space propagation, f is the frequency of the transmitted signal, c is the speed of light propagation in free space, μ LoS is the attenuation coefficient, di，j (t) represents the Euclidean distance between the two drones at time t, Rice(v, δ) represents the Rice distribution, and v and δ are the parameters of the Rice distribution. At time t, the data transmission rate of drone i offloading data to the adjacent drone i+1 can be expressed as Among them, B is the bandwidth allocated to each node drone, P n is the noise power, h i,i+1 (t) is the channel gain between two UAVs i and i+1, P i (t) is the data transmission power of drone i. Each drone has its own energy and computing power. The drone decides to process the αth received task and then unload the α-1th received task. Therefore, at time t, the total energy consumption of drone i is the sum of the energy consumed by processing the data and the energy consumed by unloading the remaining data to the neighboring drones. The specific formula can be expressed as Among them, E i is the CPU energy consumption required for UAV i to execute a task cycle, β represents the proportion of data processed locally on UAV i, s and x represent the task size and the clock cycle required to process a task, respectively. The energy required to offload data to the next UAV i+1 can be expressed as The duration D of the drone processing data at time t t Depends on the time T that the drone takes to process data locally comp and the time taken to uninstall T off , the specific formula can be expressed as Among them, R i (t) represents the available computing resources of UAV i, D t =max(T comp (t),T off (t)).

[0032] The final result is a mathematical model for processing drone network data. Beneficial effects: The mathematical model for processing drone network data established in this step can simplify the calculation process and greatly improve the efficiency of algorithm optimization.

[0033] Step 2: Use improved reinforcement learning to allocate computing resources to the mathematical model of drone network data processing in step 1.

[0034] The present invention defines the optimization problem of drone network data processing as <S, A, P, R>, where S, A, P, and R are the state space, action space, transition probability, and reward function of reinforcement learning, respectively. The state space S at time t defines the state of the environment, and the action space A consists of the discrete action set taken by the drone at time t. The specific formula can be expressed as Among them, a value of 1 means that the received data is completely offloaded to the next drone, a value of 0 means that the drone can process all the data without offloading any part, k is the weight, n is the number of drones, and the utility value W of maintaining the state in a given action is i (t) = p1(E i，comp (t)+E i，off (t))+p2M i (t), where M i (t) represents the delay cost at time t, and p1 and p2 are related weights. In the present invention, rewards are allocated according to the contribution of each drone's offloading strategy to minimizing the overall delay and energy consumption of completing the task, guiding the drones to implement a strategy that effectively balances task processing and offloading to achieve the best overall performance in terms of delay and energy efficiency. The specific strategy of drone i can be expressed as Among them, γ i is the discount factor of drone i, a is the action taken by the strategy in state s, T max is the long-term average value J(θ) of drone i and the action value Q according to the maximum task completion time allowed by the application. θ and advantage function The gradient of can be expressed as in, represents the gradient of the objective function J(θ) relative to UAV i, Denote as the baseline value, giving the average expected reward for drone i, considering the behaviors of all other drones except i.

[0035] The final result is the computational allocation resources of each drone calculated using the improved deep reinforcement learning model. Beneficial Effects: The improved deep reinforcement learning model used in this step can effectively balance the strategy of task processing and offloading to achieve the best overall performance in terms of latency and energy efficiency, while the introduction of baseline values ​​can reduce the variance in gradient estimation.

[0036] Step 3: Use the improved grey wolf optimization algorithm to optimize the computing resources allocated in step 2.

[0037] The present invention adopts an improved gray wolf optimization algorithm to optimize the computing resources allocated in step 2. There are four main types of gray wolves in the gray wolf optimization algorithm, namely, alpha wolf, beta wolf, delta wolf and omega wolf. They follow a strict social hierarchy and gradually decrease. Hunting, searching, besieging and attacking prey are the main characteristics of the group. In order to improve the movement of the wolf pack, the sine cosine algorithm is integrated into the basic gray wolf optimization algorithm. The specific formula can be expressed as X(t+1)=X+g1sing2|g3·J(θ)-X|, X(t+1)=X+g1cosg2|g3·J(θ)-X|, where X is the position of the gray wolf in the gray wolf optimization algorithm, g1, g2 and g3 are weight factors. After the sine cosine algorithm improves the basic gray wolf optimization algorithm, the movement of the alpha wolf can be expressed by the formula: Among them, C1=2g2.

[0038] What we finally get is the computing resources optimized by the improved Grey Wolf Optimization Algorithm.

[0039] Step 4: Share information updates based on the consensus method for the computing resources optimized in step 3.

[0040] This paper proposes a consensus method based on learning, which enables drones to update their strategies based on shared information. Each drone calculates its learning progress, which is the difference between its current and previous average reward estimates ε. The specific learning progress calculation formula for drone i at time t can be expressed as Using this learning process, the drones calculate the consensus weight. The consensus weight between two drones i and j can be expressed as C i，j =σ i l i +σ j l j , where σ i and σ j are the learning rates of drones i and j respectively, and the weight smoothing factor ρ is used to update the consensus weight. The specific formula can be expressed as C t [i, j] = σ·C t-1 [i,j]+(1-ρ)C i，j , where C t The consensus matrix represents the relationship between the drone and all other drones in the environment at time t. Once the consensus weights are calculated, each drone updates its average reward estimate. The drone no longer relies solely on its own reward, but considers the weighted sum of the rewards of all drones, where the weights come from the consensus matrix. The updated average reward estimate of each drone is expressed as in, represents the average return of the drone at time t. Then, the drone estimates the updated average return Based on local observations, the reward function R(s, a) of deep reinforcement learning is updated, and the parameters are iteratively adjusted to minimize the difference between the estimated reward and the actual reward value.

[0041] The final result is a shared information update for drones based on a sharing method. Beneficial effects: The local information alone of the reinforcement learning algorithm used in this step is not enough to calculate the global estimate of the action value and advantage function, because it relies on the local average reward of all drones. The learning-based consensus method enables drones to calculate global information and benefit from the experience of other drones, thereby having a more comprehensive understanding of the environment.

[0042] The beneficial effects of the present invention are:

[0043] (1) The mathematical model for drone network data processing established by the present invention can simplify the calculation process and greatly improve the efficiency of algorithm optimization.

[0044] (2) The improved deep reinforcement learning model adopted in the present invention can effectively balance the strategies of task processing and offloading to achieve the best overall performance in terms of latency and energy efficiency. At the same time, the introduction of baseline values ​​can reduce the variance in gradient estimation.

[0045] (3) The local information alone of the reinforcement learning algorithm adopted in the present invention is not sufficient to calculate the global estimate of the action value and advantage function because it relies on the local average reward of all drones. The learning-based consensus method enables drones to calculate global information and benefit from the experience of other drones, thereby having a more comprehensive understanding of the environment.

[0046] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing multi-hop data in a drone network, characterized in that: include: Step 1: Construct a mathematical model for UAV network data processing. The operation of UAVs in free space is considered during construction. It is assumed that there is a line-of-sight path between UAVs. The model is constructed by analyzing the channel gain between UAVs, data transmission rate, energy and computing power of UAVs, energy consumption when processing and unloading tasks, and duration of data processing. Step 2: Improved reinforcement learning is used to allocate computing resources for the constructed mathematical model. Improved reinforcement learning defines an optimization problem with different elements and allocates rewards according to the contribution of the drone's offloading strategy to the task delay and energy consumption, so as to achieve a balance between task processing and offloading strategy. Step 3: optimize the allocated computing resources using an improved gray wolf optimization algorithm. The gray wolf optimization algorithm integrates the sine-cosine algorithm into the basic gray wolf optimization algorithm. The gray wolves in the basic gray wolf optimization algorithm have specific social hierarchy and group behavior characteristics. Step 4: Update the shared information of the optimized computing resources based on the consensus method. Use the learning-based consensus method to update the average return estimate of the drone by calculating the learning progress and consensus weight, and then update the reward function of deep reinforcement learning.

2. A method for processing multi-hop data in a drone network as claimed in claim 1, characterized in that: In step one, when constructing a mathematical model for UAV network data processing, the situation where the UAV is operating in a free space at a considerable height from the ground and there is a line-of-sight path is considered. The model is constructed by analyzing the relationship between the channel gain between UAVs and various factors, the relationship between the data transmission rate and related factors, the relationship between the total energy consumption and various factors, and the relationship between the processing data duration and related factors. The total energy consumption is related to the energy consumption of the data processing part and the energy consumption of the unloading data, and the processing data duration is related to the local processing time and the unloading time.

3. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step 2, in the step of allocating computing resources using improved reinforcement learning, the optimization problem of UAV network data processing is defined in the form of multiple elements, where the state space defines the environment state, the action space consists of discrete action sets of the UAV, and the discrete action sets represent the two situations of complete data offloading or full processing. The state utility value in a given action is determined based on multiple factors, and rewards are allocated based on the contribution of each UAV's offloading strategy to minimizing the overall delay and energy consumption of completing the task. An effective balance between task processing and offloading strategies is achieved through a UAV strategy containing multiple elements to achieve the best overall performance in terms of delay and energy efficiency.

4. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step 2, in the step of allocating computing resources using improved reinforcement learning, the improved deep reinforcement learning model adopted can effectively balance task processing and offloading strategies, achieve the best overall performance in terms of latency and energy efficiency, and can reduce the variance in gradient estimation by introducing specific values.

5. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step three, the improved gray wolf optimization algorithm is used to optimize computing resources. The gray wolves in the improved gray wolf optimization algorithm include four types: alpha wolf, beta wolf, delta wolf and omega wolf. They follow a strict social hierarchy and have the main characteristics of a group in hunting, searching, besieging and attacking prey.

6. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step three, in the step of optimizing computing resources using the improved gray wolf optimization algorithm, the computing resources are optimized by integrating the sine-cosine algorithm into the basic gray wolf optimization algorithm and using a specific position update formula and an improved alpha wolf motion formula.

7. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step four, in the shared information update step based on the consensus method, a learning-based consensus method is used. First, the difference between the current and previous average return estimates of each drone is calculated to obtain the learning progress, and then the consensus weight is calculated based on the learning rates of the two drones. The consensus weight is updated using a weight smoothing factor related to the consensus matrix of the relationship between the drone and other drones in the environment.

8. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step four, the consensus-based shared information update step, each drone updates its average reward estimate according to the calculated consensus weights, taking into account the weighted sum of rewards of all drones, with the weights coming from the consensus matrix.

9. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step four, in the shared information update step based on the consensus method, the drone estimates the updated average reward and updates the reward function of deep reinforcement learning based on local observations, minimizing the difference between the estimated reward and the actual reward value by iteratively adjusting the parameters.

10. The method for processing multi-hop data in a drone network according to claim 1, characterized in that: In step four, the consensus-based shared information update step, the adopted learning-based consensus method enables drones to calculate global information and benefit from the experience of other drones.

Images