Reinforcement learning-based trusted access and resource allocation method for low-altitude intelligent network

Abstract

The present invention discloses a method for trusted access and resource allocation of a low-altitude intelligent network based on reinforcement learning, which belongs to the field of communication network security. The present invention constructs a reputation value for each drone based on ADS-B return information, the number of negative interactions, and computing performance, and uses blockchain to store and update the reputation value. By observing the number of drones accessing the base station, the amount of computing resources required for the task, the computing resources currently held by the base station, the reputation value of the drone, and the number of historical negative interactions between the drone and the base station, the base station constructs the system state and uses it as the input of the reinforcement learning model. Based on the output of the reinforcement learning model, the base station selects the number and ID of drones allowed to access, the corresponding computing resource allocation strategy, and the block size, thereby improving the throughput while reducing the task computing delay, transmission delay, and the number of negative interactions, thereby comprehensively improving the trusted access capability and task computing efficiency of the low-altitude intelligent network.

Description

Technical Field

[0001] The present invention relates to the field of communication network security technology, and in particular to a low-altitude intelligent network trusted access and resource allocation method based on reinforcement learning. Background Art

[0002] Drones may generate some computing tasks during flight. Due to the limitations of their own computing resources, drones can seek help from ground base stations that have more computing resources. However, some malicious drones may abuse the resources of ground base stations.

[0003] Since reinforcement learning can observe environmental changes in real time and dynamically formulate strategies to achieve optimal results, it can control access and allocate resources for drones. This can effectively avoid the risks caused by the abuse of computing resources of ground base stations, thereby improving the computing efficiency and security of low-altitude intelligent networks. How to use reinforcement learning to safely and efficiently perform access control and resource allocation is an urgent problem that needs to be solved. Summary of the Invention

[0004] The present invention provides a low-altitude intelligent network trusted access and resource allocation method based on reinforcement learning, which uses blockchain and reinforcement learning methods to perform access control and resource allocation of drones, reduce latency, and improve the security of the low-altitude intelligent network system.

[0005] The embodiment of the present invention provides a method for trusted access and resource allocation of a low-altitude intelligent network based on reinforcement learning, comprising the following steps:

[0006] Step 1) obtaining a low-altitude intelligent network system, assuming that the low-altitude intelligent network system includes M drones and N base stations;

[0007] Step 2) Construct a risk-aware reinforcement learning model for the jth base station among N base stations, 1≤j≤N, where the risk-aware reinforcement learning model includes a Q network and an E network, and initializes the weight parameters of the Q network and the E network to θ (0) and And the initial learning rate is α and the discount factor is γ;

[0008] Step 3) Base station j constructs the state vector s of time slot k (k) ;

[0009] Step 4) Base station j calculates the state vector s (k) Through policy set A (k) Select a joint optimization strategy for access control, resource allocation, and block size;

[0010] Step 5) Base station j calculates the benefit;

[0011] Step 6) Base station j evaluates the risk value and obtains the number of positive and negative interactions between the drone and base station i;

[0012] Step 7) Base station j uploads the drone ID, drone coordinate information, number of positive interactions, and number of negative interactions to the system blockchain;

[0013] Step 8) Base station j stores the experience including state vector, selection strategy, benefit value, and risk value into the experience pool;

[0014] Step 9) Base station j randomly samples Z experiences from the experience pool to form a batch sample;

[0015] Step 10) Update the weight parameters θ of the Q network and E network of risk-aware reinforcement learning (k) and

[0016] Step 11) Repeat steps 3) to 10) until the ground base station learns a stable access control strategy, resource allocation strategy and block size strategy, and the benefit u (k) converges to a stable value.

[0017] Optionally, in one embodiment of the present invention, in step 3), base station j constructs the state vector s of time slot k (k) include:

[0018] At time slot k, the ground base station j obtains the number m of drones that are connected within the communication range. (k) , obtain the available computing resources r held by itself (k) ;

[0019] Based on the ADS-B return information, the coordinate information transmitted by the drone is received by using an ADS-B ground receiver The amount of computing resources each drone wants to obtain

[0020] Get the reputation value of each drone from the blockchain Measure the number of negative interactions of each drone in the previous time slot where 1≤i≤m (k) , then the state vector s of the k-time slot constructed by the ground base station j is (k) for:

[0021]

[0022] Optionally, in one embodiment of the present invention, in step 4), the policy set is defined as Among them, a1 (k) A Boolean value set representing whether no one is allowed to access, a2 (k)Represents the allocation strategy of computing resources, a3 (k) Represents the strategy for generating block size, ground base station j converts the state vector s (k) They are input into the Q network and E network respectively, and the Q network outputs the long-term discounted expected benefit Q(s) of all participating node selection strategies in the current state. (k) ,a (k) θ (k) ), updated according to the iterative Bellman equation:

[0023]

[0024] E network outputs the long-term risk level of all participating node selection strategies in the current state Update according to the iterative Bellman equation:

[0025]

[0026] Among them, u (k) and ρ (k) They are the benefit value and risk value at time slot k, respectively. According to the long-term discounted expected benefit Q(s (k) ,a (k) θ (k) ) and long-term risk levels The strategy distribution function is constructed by improving the Boltzmann distribution function:

[0027]

[0028] According to the policy distribution function π(s (k) ,a (k) ), the ground base station j obtains the access control, resource allocation and block size strategy a (k) .

[0029] Optionally, in one embodiment of the present invention, in step 5), the specific steps of calculating the benefit value by the ground base station j are:

[0030] Ground base station j evaluates drone reputation through blockchain Calculating latency Transmission delay and the throughput q of uploaded blocks (k) , calculate the benefit by the following formula:

[0031]

[0032] Among them, w T 、w Y 、w Q Weight parameters representing latency, number of negative interactions, and throughput.

[0033] Optionally, in one embodiment of the present invention, in step 6), the specific steps of the ground base station j evaluating the risk value are:

[0034] Set the thresholds for reputation, latency, negative interactions, and throughput to be and

[0035] According to the reputation value v (k) Total delay Number of negative interactions Y(·) (k) and throughput q (k) Whether it exceeds the threshold to judge the short-term risk value ρ (k) Size:

[0036]

[0037] Among them, W T 、W Y 、W Q are the weight parameters respectively.

[0038] Optionally, in one embodiment of the present invention, in step 7), the ground base station j sends the ID information g of all drones to the ground station. (k) , coordinate information d (k) , number of positive interactions x (k) and the number of negative interactions y (k) Constructed as historical information c (k) ={g (k) ,d (k) ,x (k) ,y (k)}, and form a transaction and upload it to the blockchain, which calculates and updates the reputation value based on the number of positive and negative interactions.

[0039] Optionally, in one embodiment of the present invention, the ground base station j converts the state vector s (k) 、Select strategy a (k) 、Effect (k) and risk value ρ (k) Constructed as an empirical sequence f (k) ={s (k) ,a (k) ,u (k) ,ρ (k)}, and store the experience sequence into the experience pool D = {f (1) ,f (2) ,…,f (k)}middle.

[0040] Optionally, in one embodiment of the present invention, in step 9), the ground base station j randomly samples Z experience {φ (z)} 1≤z≤kForm batch samples where z follows a uniform distribution from 1 to k.

[0041] Optionally, in one embodiment of the present invention, in step 10), the weight parameters of ground base station j for Q network and E network are as follows:

[0042] Ground base station j uses the Adam optimization algorithm to update the weight parameters θ of the Q network (k) :

[0043]

[0044] Use the Adam optimization algorithm to update the weight parameters of the E network

[0045]

[0046] Among them, β is the discount factor for weight parameter update.

[0047] The reinforcement learning-based trusted access and resource allocation method for low-altitude intelligent networks in an embodiment of the present invention combines the adaptability of reinforcement learning with the tamper-proof nature of blockchain. It stores important information about drones through blockchain and uses reinforcement learning to control access and allocate resources for drones based on the status of ground base stations, thereby assisting drones in completing computing tasks more efficiently, optimizing the security of the low-altitude intelligent network system, and reducing transmission latency.

[0048] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0050] Figure 1 This is a flowchart of a method for trusted access and resource allocation of a low-altitude intelligent network based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0052] Figure 1 This is a flowchart of a method for trusted access and resource allocation of a low-altitude intelligent network based on reinforcement learning according to an embodiment of the present invention.

[0053] like Figure 1 As shown, the low-altitude intelligent network trusted access and resource allocation method based on reinforcement learning includes the following steps:

[0054] Step 1) Obtain a low-altitude intelligent network system, assuming that the low-altitude intelligent network system includes M drones and N base stations.

[0055] Step 2) Construct a risk-aware reinforcement learning model for the jth base station among N base stations, 1≤j≤N, where the risk-aware reinforcement learning model includes a Q network and an E network, and initializes the weight parameters of the Q network and the E network to θ (0) and And the initial learning rate is α and the discount factor is γ.

[0056] Step 3) Base station j constructs the state vector s of time slot k (k) .

[0057] In step 3), base station j constructs the state vector s of time slot k (k) include:

[0058] At time slot k, the ground base station j obtains the number m of drones that are connected within the communication range. (k) , obtain the available computing resources r (k) ;

[0059] Based on the ADS-B return information, the coordinate information transmitted by the drone is received by using an ADS-B ground receiver The amount of computing resources each drone wants to obtain

[0060] Get the reputation value of each drone from the blockchain Based on the above information, measure the number of negative interactions of each drone in the previous time slot where 1≤i≤m (k) , then the state vector s of the k-time slot constructed by the ground base station j is (k) for:

[0061]

[0062] Step 4) Base station j calculates the state vector s (k) Through policy set A (k) Select a joint optimization strategy for admission control, resource allocation, and block size.

[0063] In step 4), the policy set is defined as Among them, a1 (k) A Boolean value set representing whether no one is allowed to access, a2 (k) Represents the allocation strategy of computing resources, a3(k) Represents the strategy for generating block size, ground base station j converts the state vector s (k) They are input into the Q network and E network respectively, and the Q network outputs the long-term discounted expected benefit Q(s) of all participating node selection strategies in the current state. (k) ,a (k) θ (k) ), updated according to the iterative Bellman equation:

[0064]

[0065] E network outputs the long-term risk level of all participating node selection strategies in the current state Update according to the iterative Bellman equation:

[0066]

[0067] Among them, u (k) and ρ (k) They are the benefit value and risk value at time slot k, respectively. According to the long-term discounted expected benefit Q(s (k) ,a (k) θ (k) ) and long-term risk levels The strategy distribution function is constructed by improving the Boltzmann distribution function:

[0068]

[0069] According to the policy distribution function π(s (k) ,a (k) ), the ground base station j obtains the access control, resource allocation and block size strategy a (k) .

[0070] Step 5) Base station j calculates the benefit.

[0071] In step 5), the specific steps for ground base station j to calculate the benefit value are:

[0072] Ground base station j evaluates drone reputation through blockchain Calculating latency Transmission delay and the throughput q of uploaded blocks (k) , calculate the benefit by the following formula:

[0073]

[0074] Among them, w T 、w Y 、w Q Weight parameters representing latency, number of negative interactions, and throughput.

[0075] Step 6) Base station j evaluates the risk value and obtains the number of positive interactions and negative interactions between the drone and base station i.

[0076] In step 6), the specific steps of the ground base station j evaluating the risk value are:

[0077] Set the thresholds for reputation, latency, negative interactions, and throughput to be and

[0078] According to the reputation value v (k) Total delay Number of negative interactions Y(·) (k) and throughput q (k) Whether it exceeds the threshold to judge the short-term risk value ρ (k) Size:

[0079]

[0080] Among them, W T 、W Y 、W Q are the weight parameters respectively.

[0081] Step 7) Base station j uploads the drone ID, drone coordinate information, number of positive interactions, and number of negative interactions to the system blockchain.

[0082] In step 7), the ground base station j sends the ID information g of all drones to the ground station j. (k) , coordinate information d (k) , number of positive interactions x (k) and the number of negative interactions y (k) Constructed as historical information c (k) ={g (k) ,d (k) ,x (k) ,y (k)}, and form a transaction and upload it to the blockchain, which calculates and updates the reputation value based on the number of positive and negative interactions.

[0083] Step 8) Base station j stores the experience including state vector, selection strategy, benefit value, and risk value into the experience pool.

[0084] In step 8), the ground base station j sends the state vector s (k) 、Select strategy a (k) 、Effect (k) and risk value ρ (k) Constructed as an empirical sequence f (k) ={s (k) ,a (k) ,u (k) ,ρ (k)}, and store the experience sequence into the experience pool D = {f (1) ,f (2) ,…,f (k)}middle.

[0085] Step 9) Base station j randomly samples Z experiences from the experience pool to form a batch sample.

[0086] In step 9), the ground base station j randomly samples Z experience {φ (z)} 1≤z≤k Form batch samples where z follows a uniform distribution from 1 to k.

[0087] Step 10) Update the weight parameters θ of the Q network and E network of risk-aware reinforcement learning (k) and

[0088] In step 10), the weight parameters of ground base station j for Q network and E network are as follows:

[0089] Ground base station j uses the Adam optimization algorithm to update the weight parameters θ of the Q network (k) :

[0090]

[0091] Use the Adam optimization algorithm to update the weight parameters of the E network

[0092]

[0093] Among them, β is the discount factor for weight parameter update.

[0094] Step 11) Repeat steps 3) to 10) until the ground base station learns a stable access control strategy, resource allocation strategy and block size strategy, and the benefit u (k) converges to a stable value.

[0095] The following describes a specific embodiment of the present invention's method for trusted access and resource allocation to a low-altitude intelligent network based on reinforcement learning.

[0096] Step 1: The low-altitude intelligent network system includes two drones and one ground base station;

[0097] Step 2: Construct a risk-aware reinforcement learning model for the base station, including a Q network and an E network composed of fully connected layers. The Q network and the E network each contain an input layer, two hidden layers, and an output layer. The input layer consists of 12 neurons, the two hidden layers consist of 32 and 16 neurons respectively, and the output layer consists of 64 neurons. Initialize the weight parameters θ of the Q network and the E network.(0) and Initialize the learning rate α to 0.01 and the discount factor γ to 0.8; set the benefit function weight parameters to 1, the number of randomly sampled experiences Z = 32, the delay requirement for determining whether the model training has timed out τ = 0.05s, and the credit value risk requirement Delay risk requirements Negative interaction risk requirements Throughput risk requirements

[0098] Step 3: Based on the ADS-B return information and feedback information, at time slot k, the ground base station obtains the number m of drones that are connected within the communication range. (k) , obtain the available computing resources r (k) ; Get the drone ID by using the ADS-B ground receiver and receive the coordinate information transmitted by the drone The amount of computing resources each drone wants to obtain Get the reputation value of each drone from the blockchain Measure the number of negative interactions of each drone in the previous time slot where 1≤i≤m (k) . Construct the state vector s (k) as follows:

[0099]

[0100] Step 4: The ground base station converts the state vector s (k) Input them into Q network and E network respectively, and according to the long-term discounted expected benefit Q(s) output by Q network (k) ,a (k) θ (k) ) and the long-term risk level of the E network output Get the strategy distribution function π(s (k) ,a (k) ), select the joint optimization strategy a of access control, resource allocation and block size according to the policy distribution (k) ∈A.

[0101] Step 5: The ground base station can obtain the drone’s reputation value evaluated by the blockchain based on the selected action strategy Calculating latency Transmission delay and the throughput q of uploaded blocks (k) Calculate the benefit value u through the above content (k) as follows:

[0102]

[0103] Step 6: Set the reputation value v( k ) Total delay Number of negative interactions Y(·) ( k ) and throughput q (k) It is required by reputation value, delay, number of negative interactions, and throughput and Compare and obtain the risk value ρ of the ground base station (k) :

[0104]

[0105] Step 7: The ground base station will send the ID information of all drones to the (k) , coordinate information d (k) , number of positive interactions x (k) and the number of negative interactions y (k) Constructed as historical information c ( k ) ={g ( k ) ,d ( k ) ,x ( k ) ,y ( k )}And form a transaction and upload it to the blockchain, which calculates and updates the reputation value based on the number of positive and negative interactions.

[0106] Step 8: The ground base station changes the state s (k) 、Select strategy a (k) 、Effect (k) and risk value ρ (k) Constructed as an empirical sequence f (k) ={s (k) ,a (k) ,u (k) ,ρ (k)}, and store the experience sequence into the experience pool D = {f (1) ,f (2) ,…,f (k)}middle.

[0107] Step 9: The ground base station randomly samples 32 experiences from the experience pool to form a batch sample;

[0108] Step 10: The ground base station uses the Adam optimization algorithm to update the weight parameters θ of the Q network and E network respectively (k) and

[0109]

[0110] Step 11: Repeat steps 3 to 10 until the ground base station learns a stable access control strategy, resource allocation strategy, and block size strategy.

[0111] According to the trusted access and resource allocation method for low-altitude intelligent network based on reinforcement learning proposed in an embodiment of the present invention, the access control and resource allocation strategy of drones are jointly optimized to realize intelligent trusted computing offloading and defend against malicious attacks. The present invention builds the reputation value of each drone based on ADS-B return information, the number of negative interactions and computing performance, and uses blockchain to store and update the reputation value. By observing the number of drones accessing the base station of the invention, the amount of computing resources required for the task, the computing resources currently held by the base station, the reputation value of the drone and the number of historical negative interactions between the drone and the base station, the base station builds the system state and uses it as the input of the reinforcement learning model. Based on the output of the reinforcement learning model, the base station selects the number and ID of drones allowed to access, the corresponding computing resource allocation strategy and block size, thereby improving the throughput while reducing the task computing delay, transmission delay and the number of negative interactions, thereby comprehensively improving the trusted access capability and task computing efficiency of the low-altitude intelligent network.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A method for trusted access and resource allocation of low-altitude intelligent network based on reinforcement learning, characterized in that: The following steps are involved: Step 1) obtaining a low-altitude intelligent network system, assuming that the low-altitude intelligent network system includes M drones and N base stations; Step 2) Construct a risk-aware reinforcement learning model for the jth base station among N base stations, 1≤j≤N, where the risk-aware reinforcement learning model includes a Q network and an E network, and initializes the weight parameters of the Q network and the E network to θ (0) and And the initial learning rate is α and the discount factor is γ; Step 3) Base station j constructs the state vector s of time slot k (k) ; Step 4) Base station j calculates the state vector s (k) Through policy set A (k) Select a joint optimization strategy for access control, resource allocation, and block size; Step 5) Base station j calculates the benefit; Step 6) Base station j evaluates the risk value and obtains the number of positive and negative interactions between the drone and base station i; Step 7) Base station j uploads the drone ID, drone coordinate information, number of positive interactions, and number of negative interactions to the system blockchain; Step 8) Base station j stores the experience including state vector, selection strategy, benefit value, and risk value into the experience pool; Step 9) Base station j randomly samples Z experiences from the experience pool to form a batch sample; Step 10) Update the weight parameters θ of the Q network and E network of risk-aware reinforcement learning (k) and Step 11) Repeat steps 3) to 10) until the ground base station learns a stable access control strategy, resource allocation strategy and block size strategy, and the benefit u (k) converges to a stable value.

2. The method according to claim 1, characterized in that In step 3), base station j constructs the state vector s of time slot k (k) include: At time slot k, the ground base station j obtains the number m of drones applying for access within the communication range. (k) , obtain the available computing resources r held by itself (k) ; Based on the ADS-B return information, the coordinate information transmitted by the drone is received by using an ADS-B ground receiver The amount of computing resources each drone wants to obtain Get the reputation value of each drone from the blockchain Measure the number of negative interactions of each drone in the previous time slot where 1≤i≤m (k) , then the state vector s of the k-time slot constructed by the ground base station j is (k) for:

3. The method according to claim 1, characterized in that In step 4), the policy set is defined as a (k) ={a1 (k) ,a2 (k) ,a3 (k) }∈A, where a1 (k) A Boolean value set representing whether drone access is allowed, a2 (k) Represents the allocation strategy of computing resources, a3 (k) Represents the strategy for generating block size, ground base station j converts the state vector s (k) They are input into the Q network and E network respectively, and the Q network outputs the long-term discounted expected benefit Q(s) of all participating node selection strategies in the current state. (k) ,a (k) θ (k) ), updated according to the iterative Bellman equation: E network outputs the long-term risk level of all participating node selection strategies in the current state Update according to the iterative Bellman equation: Among them, u (k) and ρ (k) They are the benefit value and risk value at time slot k, respectively. According to the long-term discounted expected benefit Q(s (k) ,a (k) θ (k) ) and long-term risk levels The strategy distribution function is constructed by improving the Boltzmann distribution function: According to the policy distribution function π(s (k) ,a (k) ), the ground base station j obtains the access control, resource allocation and block size strategy a (k) .

4. The method according to claim 1, wherein In step 5), the specific steps for ground base station j to calculate the benefit value are: Ground base station j evaluates drone reputation through blockchain Calculating latency Transmission delay and the throughput q of uploaded blocks (k) , calculate the benefit by the following formula: Among them, w T 、w Y 、w Q Weight parameters representing latency, number of negative interactions, and throughput.

5. The method according to claim 1, wherein In step 6), the specific steps of the ground base station j evaluating the risk value are: Set the thresholds for reputation, latency, negative interactions, and throughput to be and According to the reputation value v (k) Total delay Number of negative interactions Y(·) (k) and throughput q (k) Whether it exceeds the threshold to judge the risk value ρ (k) Size: Among them, W T 、W Y 、W Q are the weight parameters respectively.

6. The method according to claim 1, characterized in that In step 7), the ground base station j sends the ID information g of all drones to the ground station j. (k) , coordinate information d (k) , number of positive interactions x (k) and the number of negative interactions y (k) Constructed as historical information c (k) ={g (k) ,d (k) ,x (k) ,y (k) }, and form a transaction and upload it to the blockchain, which calculates and updates the reputation value based on the number of positive and negative interactions.

7. The method according to claim 1, characterized in that In step 8), the ground base station j sends the state vector s (k) 、Select strategy a (k) 、Effect (k) and risk value ρ (k) Constructed as an empirical sequence f (k) ={s (k) ,a (k) ,u (k) ,ρ (k) }, and store the experience sequence into the experience pool D = {f (1) ,f (2) ,…,f (k) }middle.

8. The method according to claim 1, characterized in that In step 9), the ground base station j randomly samples Z experience {φ (z) } 1≤z≤k Form batch samples where z follows a uniform distribution from 1 to k.

9. The method according to claim 1, characterized in that In step 10), the weight parameters of ground base station j for Q network and E network are as follows: Ground base station j uses the Adam optimization algorithm to update the weight parameters θ of the Q network (k) : Use the Adam optimization algorithm to update the weight parameters of the E network Among them, β is the discount factor for weight parameter update.

Images