Air-ground cooperative communication hierarchical decision-making method based on environmental cognition

Through the spatial and ground collaborative communication layered decision-making method based on environmental cognition, using graph convolution network and hierarchical reinforcement learning, the spectrum and spatial resource utilization problems of spatial and ground collaborative communication in complex urban environments are solved, and the transmission rate and decision-making efficiency are improved. Adapt to dynamic interference and electromagnetic environment changes.

CN119071817AActive Publication Date: 2024-12-03NANJING UNIV OF AERONAUTICS & ASTRONAUTICS

Patent Information

Application Number
CN202411059865.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-04
Publication Date
2024-12-03
Estimated Expiration
2044-08-04

AI Technical Summary

Technical Problem

In complex urban environments, it is difficult to effectively utilize spectrum resources and spatial domain resources in air-ground collaborative communication, traditional reinforcement learning methods are difficult to achieve rapid convergence and multi-dimensional decision-making, and traditional clustering algorithms have high computational complexity and cannot adapt to dynamic interference and electromagnetic environment changes.

Method used

Adopt the spatial and ground collaborative communication hierarchical decision-making method based on environmental cognition, and aggregate multiple environmental features through graph convolution network model for node clustering, and optimize communication decision-making using hierarchical reinforcement learning based on the action-judge framework to coordinate the spatial and ground Communication modeling is a Markov decision-making process, which is decomposed into a ground node cluster and a drone association decision-making layer, a drone trajectory planning layer, and a ground node communication decision-making layer.

Benefits of technology

It significantly improves the transmission rate of multiple nodes in the air-ground collaborative communication network, improves decision-making efficiency, can effectively combat dynamic interference, and adapt to changes in complex urban environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071817A_ABST
    Figure CN119071817A_ABST
Patent Text Reader

Abstract

The invention discloses an air-ground cooperative communication hierarchical decision-making method based on environmental cognition, and the method comprises the steps: employing an unmanned plane to assist communication among multiple ground nodes for malicious or unintentional dynamic interference in a complex urban environment; the method comprises the following steps: dividing air-ground cooperative communication assisted by an unmanned aerial vehicle into three layers of subtasks, constructing a complex decision problem of air-ground cooperative communication, modeling as a Markov decision process, constructing a hierarchical structure based on an action device-evaluator framework, and optimizing a communication decision of multiple nodes in an air-ground cooperative network by using a hierarchical reinforcement learning method; a graph convolution network model is constructed, various environment characteristics including building shielding, node positions, task queues of nodes and frequency use conditions of the nodes in the air-ground cooperative communication network are comprehensively considered based on environment cognition, and multi-node clustering optimization is achieved. According to the method, the characteristics influencing the node transmission performance are effectively extracted from the complex urban environment, and the multi-node transmission rate in the air-ground cooperative communication network is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a hierarchical decision-making method for air-ground collaborative communication based on environmental cognition. Background Art

[0002] The deployment of 6G networks faces many challenges, especially in the allocation of spectrum resources. The growing demand for high-throughput applications and large-scale device access makes it essential to effectively solve congestion problems and establish intelligent connections between user devices. However, in complex urban environments, ground channels are affected by building occlusion and small-scale fading, and the transmission rate is limited. UAVs can not only assist ground node task offloading and alleviate the problems caused by insufficient computing power, but also provide dynamic network deployment and reduce resource unfairness. In addition, by utilizing the line-of-sight link between UAVs and ground nodes, higher channel quality and faster transmission rate can effectively reduce data transmission latency and power consumption. In the context of air-ground collaborative communication, scattered buildings constitute a complex electromagnetic environment, which can affect the transmission of communication signals and can also serve as a barrier against malicious interference. At present, there is a lack of air-ground collaborative communication methods that start from environmental cognition, comprehensively utilize spatial and frequency domain resources in complex urban environments, and effectively combat dynamic interference.

[0003] In a complex and fast-changing electromagnetic environment, since the transmission data of each node is continuously generated throughout the time slot, the communication decision made in the time slot cannot obtain the information of the data packet transmission volume generated in the time slot. In addition, since the overhead of decision and data preparation cannot be ignored, the unloading scheduling decision for processing / unloading these task queues can only be executed in the next time slot. Therefore, the joint learning strategy of air-ground collaborative communication decision and UAV trajectory planning is regarded as a sequential decision problem and modeled as a Markov decision process. Traditional reinforcement learning methods are difficult to achieve the requirements of fast convergence and multi-dimensional decision-making of UAVs in complex urban environments due to their sensitivity to hyperparameters, dependence on a single strategy, and shortcomings in solving multi-dimensional decision-making problems. In addition, in large-scale multi-agent reinforcement learning, the agent does not need to observe the state or actions of all participants, but only needs to observe the state or actions of some related agents in the neighborhood. Therefore, clustering agents with the same tasks in complex scenes can reduce the required storage capacity and computing overhead and improve decision-making efficiency. However, in the context of air-ground collaborative communication, traditional clustering algorithms have high computational complexity and cannot take into account the complex and changeable electromagnetic environment in the city and the time-varying nature of cluster network topology. In addition, using only single features such as distance and density to cluster nodes is obviously insufficient. Summary of the invention

[0004] The purpose of the present invention is to provide a hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, which can effectively extract the characteristics of transmission performance from complex urban environments and significantly improve the transmission rate of multiple nodes in the air-ground collaborative communication network.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] A hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, the air-ground collaborative communication hierarchical decision-making method comprising the following steps:

[0007] Step 1: Set a complex urban environment as the scene. Aiming at malicious or unintentional dynamic interference in the scene, use drones to assist multiple ground nodes to communicate. The data transmission task generated by each ground node adopts any of the following communication methods: no communication, direct communication between ground nodes, and drone-assisted communication;

[0008] Step 2: According to the complexity of the task and the granularity of the decision, the UAV-assisted air-ground collaborative communication is divided into three layers of subtasks: the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer, to construct the complex decision problem of air-ground collaborative communication in this scenario; the complex decision problem of air-ground collaborative communication is modeled as a Markov decision process, and the hierarchical reinforcement learning method based on the actor-critic framework is used to optimize the communication decisions of multiple nodes in the air-ground collaborative network;

[0009] Step 3: Build a graph convolutional network model, use the graph convolutional network to aggregate multiple environmental features for node clustering, and comprehensively consider multiple environmental features including building occlusion, node location, node task queue, and node frequency in the air-ground collaborative communication network based on environmental cognition to optimize clustering of multiple nodes;

[0010] Step 4: Until the hierarchical decision algorithm of air-ground collaborative communication based on environmental cognition converges, the action selection decision is the optimal decision.

[0011] Step 1 further comprises the following steps:

[0012] Set a complex urban environment as the scene, which includes M drones with single antenna connection U = {U 1 , U 2 , ..., U m}, m = 1, 2, ..., M, and N ground nodes u = {u 1 ,u 2 , ..., u n}, N = 1, 2, ..., N. M drones are used to assist N ground nodes in air-ground collaborative communication in scattered buildings to deal with interference signals. The ground nodes are divided into K ground node clusters. The drones always fly at the same height H, and the ground nodes are on the horizontal plane and fixed in position;

[0013] When a ground node has a data transmission task, in each time slot, the data transmission task generated by each ground node adopts any of the following communication modes: no communication, direct communication between ground nodes, and communication assisted by UAVs. In addition, the data transmission task generated by a single ground node in a single time slot can be processed by only one UAV.

[0014] The communication channel between the UAV and the ground node is described by the logarithmic range loss model, which includes a constant attenuation factor in small-scale fading and obstacle occlusion; the information rate of the nth, n∈{1, 2, ..., N}th ground node located at a constant position on the ground is given by: Among them, the transmission power P, noise power N and path loss γ of the nth ground node n ; The information rate when the drone forwards the task queue is where γ m represents the path loss when the drone forwards the task queue; when the path loss index in vacuum is set to mortar, the path loss of node n is Here, small-scale fading is modeled as a Rayleigh distributed random variable X with a scaling factor σ = 1: Rayleigh ; The attenuation due to obstacle occlusion is modeled by a discrete factor β∈{1,0.01}, where β=0.01 if there is a building occlusion, and β=1 otherwise; The path loss of signal transmission between ground nodes is calculated by the following formula: Among them, P r Indicates the signal receiving power, P t Indicates the signal transmission power, G r Represents the gain of the receiving antenna, G t represents the gain of the transmitting antenna, L represents the system loss, d represents the distance between the receiving end and the transmitting end, λ represents the wavelength of the signal, and γ represents the path loss exponent;

[0015] Assume that all ground nodes use frequency band F = {f 1 , f 2 , ..., f i}, i∈{1, 2, ..., I} for communication, and there are I frequencies to choose from; suppose there is a jammer in the scene that transmits interference signals at different frequencies in each time slot, interfering with the data transmission between ground nodes, and the interference signal frequency of the jammer is repeated at the time slot period F j ={f 1 , f2 , ..., f j}, j∈{1, 2, ..., J}; in each time slot, when the ground node selects a frequency point for communication, if the selected frequency point is the same as the frequency point of the interference signal sent by the jammer in the current time slot, it is considered that the current ground node communication is interfered, and if multiple ground nodes select the same frequency point in the same time slot, it is considered that there is communication interference between these ground nodes; when interference and mutual interference occur, the transmission rate is calculated by the following formula Among them, G j is the channel gain from the interference signal to the interfered node, P j is the interference signal power, G i is the channel gain of the mutual interference signal, A is the set of mutual interference numbers of ground nodes, γ n is the path loss of node n, P is the transmission power of the nth ground node, and N is the noise power;

[0016] At the beginning of each time slot, the amount of data transmission randomly generated by each ground node is Δt, each ground node has a data accumulation queue Δb, and each drone has a data accumulation queue Δm. It is assumed that half of each time slot τ is used for transmission and half for reception, and each ground node can only transmit data to one ground node per time slot, and the ground node can only receive data transmitted by the corresponding ground node;

[0017] At the beginning of the time slot, since the amount of data randomly generated by the ground node is ω, the cumulative amount of task data to be transmitted by the ground node is Δ n =Δ n +Δ t , when the ground node chooses to use drones to assist in communication, if the transmission can be completed, that is, Δ t ≤R n At (t)·τ / 2, the amount of data received by the UAV and forwarding is completed. The current amount of data received by the uplink UAV queue is: Δ n =Δ m +Δ t , the ground node queue minus the current data volume: Δ n =Δ n -Δ t ; Current data volume forwarded in downlink UAV queue: Δ m =Δ m -Δ t ; If the transmission cannot be completed, Δ t >R n At (t)·τ / 2, the maximum amount of data that the UAV can currently receive in the uplink is: Δ m =Δ m +R n (t)·τ / 2, the maximum amount of data that can be sent by the ground node queue minus: Δn =Δ n -R n (t)·τ / 2; the maximum amount of data that can be forwarded by the UAV in the downlink: Δ m =Δ m -R m (t)·τ / 2, where R n (t) and R m (t) represents the information rate of the ground node and the UAV respectively; the unsent data is sent in the next time slot; when the ground node chooses direct communication, if the transmission can be completed, that is, Δ t ≤R n (t)·τ / 2, the ground node queue minus the current data volume: Δ n =Δ n -Δ t , if the transmission cannot be completed, that is, Δ t >R n (t)·τ / 2, the maximum amount of data that can be sent by the ground node queue is subtracted: Δ n =Δ n -R n (t)·τ / 2; the unsent data is sent in the next time slot.

[0018] Step 2 further includes:

[0019] In the air-ground collaborative communication scenario, the joint learning strategy of air-ground collaborative communication decision and UAV trajectory planning is reduced to a sequential decision problem. The semi-Markov decision process is used to model the hierarchical decision problem in air-ground collaborative communication, and problem planning and strategy learning are performed for different time slots. Specifically:

[0020] Different tasks are modeled as semi-Markov decision processes with different time scales. A hierarchical structure is constructed through task relationships, and the multi-task and multi-granularity complex decision-making problems in air-ground collaborative communication scenarios are transformed into a top-down hierarchical decision-making with three layers: the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer.

[0021] Define a time slot interval Δ, deploy node clustering, trajectory planning and data transmission agents on UAVs and ground nodes respectively. At the beginning of each Δ time slot, each ground node is randomly clustered, and then UAVs are selected as auxiliary communication nodes of the ground node cluster. Each UAV makes trajectory planning for the next Δ time slots based on its state, and the ground node performs data transmission in each time slot to minimize the accumulated data packets of each node.

[0022] Each ground node re-formulates the ground node clustering decision every Δ time slots. Each ground node performs data transmission in each time slot t according to the latest cluster association decision issued by the ground node cluster. When the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated;

[0023] In the air-ground collaborative communication scenario, based on the actor*judge framework, two fully connected neural networks are used to approximate the policy function and the state-value function respectively. The actor is composed of the policy network and participates in the generation of the agent's actions. The judge evaluates the actions taken by the actor by using the parameterized state-value function.

[0024] At the end of Δ time slots, the temporal difference error (TD error) is calculated based on the accumulated rewards and the output of the value network, and the TD mean square error is minimized to guide the update of the policy network and its own value network parameters;

[0025] Repeat the above steps until the algorithm converges, and take the action selection decision at this time as the optimal decision.

[0026] Furthermore, the UAV trajectory planning layer as an intermediate layer inherits the top-level ground node cluster and UAV associated clustering information a k,t , the ground node communication decision layer receives information from the top and middle layers as the bottom layer k,t and a m,t , for the ground node agent n, a n,t Determine the communication decisions of ground nodes;

[0027] In the decision layer of ground node cluster and drone association, state s k,t For k,t =[p 1 , p 2 , ..., p M , a k,t-1 ], where p 1 , p 2 , ..., p M Represents the positions of M drones, a k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by ground node cluster k k,t+Δ Defined as the accumulation of queues between all ground nodes and drones in the cluster

[0028]

[0029] In the UAV trajectory planning layer, the state s m,t For m,t =[p m , gm , a k,t-1 ], where p m and q m They represent the location of UAV m and the accumulated task queue information, respectively. k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by drone m m,t+Δ It is defined as the queue accumulation between all ground nodes in the cluster associated with the drone

[0030]

[0031] In the ground node communication decision layer, the state s k,n,t For k,n,t =[p m , p n , a n,t-1, f n ,q n ], where a n,t-1 represents the action decision of the time slot on the ground node n, p n 、f n ,q n They represent the location, communication frequency, and accumulated task queue information of ground node n respectively; the reward obtained by ground node n is defined as the accumulated task queue volume r of ground node n k,n,t =q n .

[0032] Furthermore, the process of optimizing the communication decision of multiple nodes in the air-ground collaborative network using the hierarchical reinforcement learning method based on the actor*critic framework includes the following steps:

[0033] Define the cluster association strategy network parameters θ of the ground node cluster k k and cluster association value network parameter w k , the association decision between the ground node cluster and the drone cluster in time slot t is expressed as a k,t =π(·|s k,t θ k ), where in state s k,t According to the cluster association strategy network parameter θ k The output result is selected with a certain probability. k,t ,satisfy The decision to associate the ground node cluster with the drone cluster ends after Δ time slots, based on the accumulated reward r k,t+Δ And the output V(s) of the cluster association value network k,t ;w k ), calculate the TD error δ k,t =r k,t+Δ +ηV(s k,t+Δ ;w k )-V(sk,t ;w k ), where η is the loss coefficient; the cluster association value network parameters are updated with the goal of minimizing the TD mean square error, and the loss function is defined as The cluster association value network along the loss function L(w k ) The update formula of the gradient direction update parameter is expressed as Where α is the weight factor, and the cluster association strategy function network is guided by the TD error along the loss function J(θ k ) updates the cluster association network parameters in the gradient direction, where the loss function is defined as J(θ k )=logπ(s k,t θ k )·δ k,t , the update formula of the cluster association strategy network parameters in each Δ time slot is expressed as

[0034] Define the trajectory planning strategy network parameters θ of UAV m m and trajectory planning value network parameter w m , the trajectory planning decision in time slot t is represented as a m,t =π(·|s m,t θ m ); When the trajectory planning decision ends after Δ time slots, the cumulative reward r obtained by the drone m,t+Δ And the output V(s) of the trajectory planning value network m,t ;w m ), calculate the TD error δ of UAV m m,t =r m,t+Δ +ηV(s m,t+Δ ;w m )-V(s m,t ;w m );Define the communication decision strategy network parameters θ of the ground node n in each cluster k,n and the communication decision value network parameter w k,n , the communication decision action of the ground node is represented as a k,n,t =π(·|s k,n,t θ k,n ); The communication decision action of the ground node is made in each time slot, and the reward r is obtained at the end of the executed data transmission action k,n,t , TD error is expressed as δ k,n,t =r k,n,t +ηV(s k,n,t+1 ;w k,n )-V(s k,n,t ;w k,n ).

[0035] Furthermore, step 3 includes the following sub-steps:

[0036] Step 3.1, abstract all nodes into a graph model representation G = (V, E), where V represents the node set and E represents the edge set;

[0037] Step 3.2, construct the adjacency matrix A and the feature matrix X; the dimension of the adjacency matrix A is N×N, which is used to reflect the communication relationship between nodes. If there is a communication relationship between node i and node j, i, j∈{1, 2, …, N}, then A ij =1, otherwise A ij =0, the dimension of the feature matrix X is N×F, where F represents the number of classes of node features;

[0038] Step 3.3, construct a graph convolution model. The number of neurons in the input layer is equal to the number of columns of the feature matrix X. The number of neurons in the output layer is determined by the number of clusters of the node clusters. The Glorot method is used to initialize the uniformly distributed weight matrix W.

[0039] Step 3.4: Preprocess the adjacency matrix A and feature matrix X and input them into the graph convolutional network. Then, the prediction matrix Y is obtained through forward propagation. in and Represent the preprocessed adjacency matrix A and feature matrix X, W respectively ( 1) and W ( 2) Represent the weight matrices of the first and second layers in the graph convolutional network, respectively. ReLU and softmax represent two selected activation functions.

[0040] Step 3.5, calculate the cross entropy loss function between the prediction matrix and the label matrix output by the graph convolutional network to update the weight matrix parameters. Where N is the number of nodes, K is the number of clusters into which the nodes are clustered, and Y ij and P ij Represent the prediction matrix and label matrix respectively;

[0041] Step 3.6: Repeat steps 3.4-3.5 until the algorithm converges. If the prediction accuracy of the graph convolutional network meets the requirements, the pre-trained weight matrix W is applied to the hierarchical decision-making algorithm based on the actor-judge framework to extract multiple environmental features to recognize the complex urban environment.

[0042] Furthermore, in the decision layer for associating ground node clusters with UAVs, the adjacency matrix A reflects the queue transmission relationship between the ground node clusters and the UAVs, and the feature matrix X includes the positions of all ground nodes and UAVs and the cumulative queue volume; in the ground node communication decision layer, the adjacency matrix A reflects the queue transmission relationship between ground nodes, and the feature matrix X includes the positions of ground nodes and whether frequency mutual interference occurs between ground nodes.

[0043] Furthermore, in step 3.4, the calculation formula of the preprocessed adjacency matrix A is Where D represents the degree matrix of the adjacency matrix A, I represents the identity matrix; the calculation formula of the preprocessed feature matrix X is in and δ x Respectively represent the mean and variance of the feature matrix X;

[0044] In the graph convolutional network, the forward propagation from hidden layer l to hidden layer l+1 is expressed as Where σ represents the ReLU activation function, H l and W l denote the output matrix and weight matrix of hidden layer l respectively.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention proposes a hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, which effectively extracts the characteristics of transmission performance from complex urban environments and significantly improves the transmission rate of multiple nodes in the air-ground collaborative communication network. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the hierarchical decision-making method for air-ground collaborative communication based on environmental cognition of the present invention.

[0048] Figure 2 This is a model diagram of the air-ground collaborative communication hierarchical decision-making system based on environmental cognition of the present invention.

[0049] Figure 3 This is a specific operation flow chart of the air-ground collaborative communication hierarchical decision-making method based on environmental cognition of the present invention.

[0050] Figure 4 This is a specific operation flow chart of the pre-trained graph convolutional network of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0052] A hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, characterized in that the hierarchical decision-making method for air-ground collaborative communication comprises the following steps:

[0053] Step 1: Set a complex urban environment as the scene. Aiming at malicious or unintentional dynamic interference in the scene, use drones to assist multiple ground nodes to communicate. The data transmission task generated by each ground node adopts any of the following communication methods: no communication, direct communication between ground nodes, and drone-assisted communication;

[0054] Step 2: According to the complexity of the task and the granularity of the decision, the UAV-assisted air-ground collaborative communication is divided into three layers of subtasks: the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer, to construct the complex decision problem of air-ground collaborative communication in this scenario; the complex decision problem of air-ground collaborative communication is modeled as a Markov decision process, and the hierarchical reinforcement learning method based on the actor-critic framework is used to optimize the communication decisions of multiple nodes in the air-ground collaborative network;

[0055] Step 3: Build a graph convolutional network model, use the graph convolutional network to aggregate multiple environmental features for node clustering, and comprehensively consider multiple environmental features including building occlusion, node location, node task queue, and node frequency in the air-ground collaborative communication network based on environmental cognition to optimize clustering of multiple nodes;

[0056] Step 4: Until the hierarchical decision algorithm of air-ground collaborative communication based on environmental cognition converges, the action selection decision is the optimal decision.

[0057] This embodiment proposes a hierarchical decision-making method for air-ground collaborative communication based on environmental cognition. Its main steps, system model, and specific operations are as follows: Figure 1-4 As shown, specifically including the following:

[0058] Set a complex urban environment as the background, such as Figure 2 As shown, the scenario includes M drones with single antenna connection U = {U 1 , U 2 , ..., U m}, m = 1, 2, ..., M, and N ground nodes u = {u 1 ,u 2 , ..., u n}, n = 1, 2, ..., N are equipped with the same single antenna to perform air-ground collaborative communication in the dispersed building clusters to deal with interference signals, and the ground nodes are divided into K ground node clusters. Assume that in this scenario, as the time slot changes, the drone always flies at the same height H, and the ground nodes are on the horizontal plane and fixed in position.

[0059] When a ground node has a data transmission task (receiving or sending data packets), in each time slot, the data transmission task generated by each ground node can choose three communication modes, namely no communication, direct communication between ground nodes, and using drone-assisted communication, and the data transmission task generated by a single ground node in a single time slot can and can only rely on one drone to process. Assuming that all ground nodes can and can only establish a connection with one drone per time slot, if the ground node cluster k is associated with the mth drone, then when the data transmission task generated by each node in the ground node cluster k chooses to communicate through drones, data transmission can only be performed through drone m. When a new cluster association decision is generated, the ground node cluster sends the command to each ground node in the cluster. When the node further executes data transmission, it will strictly abide by the cluster association decision of the time slot, and the ground node can choose to communicate with other ground nodes in each time slot.

[0060] The communication channel between the UAV and the ground node is described by a logarithmic range loss model that includes a constant attenuation factor in small-scale fading and obstacle occlusion. The information rate of the nth, n∈{1, 2, ..., N}th ground node located at a constant position on the ground is given by: Among them, the transmission power P, noise power N and path loss γ of the nth user n , Similarly, the information rate when the drone forwards the task queue is where γ m represents the path loss when the drone forwards the task queue. When the path loss exponent in vacuum is set to α, the path loss of node n is as follows Here, small-scale fading is modeled as a Rayleigh distributed random variable X with a scaling factor σ = 1: Rayleigh ; The attenuation due to obstacle occlusion is modeled by a discrete factor β∈{1,0.01}, where β=0.01 if there is a building occlusion, and β=1 otherwise. The path loss of signal transmission between ground nodes (ground-to-ground transmission) is calculated by the following formula: Among them, P r Indicates the signal receiving power, P t Indicates the signal transmission power, G r Represents the gain of the receiving antenna, G t represents the gain of the transmitting antenna, L represents the system loss, d represents the distance between the receiving end and the transmitting end, λ represents the wavelength of the signal, and γ represents the path loss exponent.

[0061] Assume that all ground nodes use frequency band F = {f 1 , f 2 , ..., f i}, i∈{1, 2, ..., I} for communication, and there are I frequencies to choose from; suppose there is a jammer in the scene that transmits interference signals at different frequencies in each time slot, interfering with the data transmission between ground nodes, and the interference signal frequency of the jammer is repeated at the time slot period F j ={f 1 , f 2 , ..., f j}, j∈{1, 2, ..., J}; in each time slot, when the ground node selects a frequency point for communication, if the selected frequency point is the same as the frequency point of the interference signal sent by the jammer in the current time slot, it is considered that the current ground node communication is interfered, and if multiple ground nodes select the same frequency point in the same time slot, it is considered that there is communication interference between these ground nodes; when interference and mutual interference occur, the transmission rate is calculated by the following formula Among them, G j is the channel gain from the interference signal to the interfered node, P j is the interference signal power, G i is the channel gain of the mutual interference signal, A is the set of mutual interference numbers of ground nodes, γ n is the path loss of node n, P is the transmission power of the nth ground node, and N is the noise power.

[0062] The amount of data transmission randomly generated by each ground node in each time slot is Δt, each ground node has a data accumulation queue Δn, and each drone has a data accumulation queue Δm. It is assumed that half of each time slot, i.e. τ / 2, is used for transmission and half of the time is used for reception. Moreover, each ground node can only transmit data to one ground node in each time slot, and the ground node can only receive data transmitted by the corresponding ground node.

[0063] At the beginning of the time slot, since the amount of data randomly generated by the ground node is Δt, the cumulative amount of task data to be transmitted by the ground node is Δ n =Δ n +Δ t , when the ground node chooses to use drones to assist in communication, if the transmission can be completed, that is, Δ t ≤R n At (t)·τ / 2, the UAV receives the data volume and completes the forwarding, that is, the uplink UAV queue receives the current data volume, and the ground node queue subtracts the current data volume: Δ m =Δ m +Δ t , Δ n =Δ n -Δ t ; Current data volume forwarded in downlink UAV queue: Δ m =Δ m -Δ t ; If the transmission cannot be completed, Δt >R n At (t)·τ / 2, the maximum amount of data that the UAV can currently receive in the uplink, and the maximum amount of data that the ground node queue can currently send is subtracted: Δ m =Δ m +R n (t)·τ / 2,Δ n =Δ n -R n (t)·τ / 2; the maximum amount of data that can be forwarded by the UAV in the downlink: Δ m =Δ m -R m (t)·τ / 2, where R n (t) and R m (t) represents the information rate of the ground node and the UAV respectively; the unsent data is sent in the next time slot. When the ground node chooses direct communication, if the transmission can be completed, that is, Δ t ≤R n (t)·τ / 2, the ground node queue minus the current data volume: Δ n =Δ n -Δ t , if the transmission cannot be completed, that is, Δ t >R n (t)·τ / 2, the maximum amount of data that can be sent by the ground node queue is subtracted: Δ n =Δ n -R n (t)·τ / 2; the unsent data is sent in the next time slot.

[0064] In the air-ground collaborative communication scenario, since the communication decision made in a time slot cannot obtain the information of the data packet transmission volume generated in the time slot, and since the data transmission overhead cannot be ignored, the joint learning strategy of air-ground collaborative communication decision and UAV trajectory planning is reduced to a sequential decision problem and modeled as a Markov decision process. In addition, the ground node communication decision and the UAV trajectory planning decision have different time scales. Simply modeling multiple decision problems into a Markov decision process cannot solve the problem that complex decisions need to span multiple time slots. The semi-Markov decision process (SMDP) is used to model the hierarchical decision problem in air-ground collaborative communication to achieve problem planning and strategy learning in different time slots. The decision scheme in SMDP can last for multiple time steps to simulate decision-making behaviors with multiple granularities. It can divide the complex decision problem in the air-ground collaborative communication scenario into the ground node communication decision problem of small time slots and the cluster association decision problem of large time slots to achieve problem planning and strategy learning in different time slots. For multi-task and multi-granularity complex decision-making problems, different tasks can be modeled as SMDPs with different time scales, and a hierarchical structure can be constructed through task relationships. The hierarchical structure allows complex decision-making problems to be further subdivided into multiple layers of sub-problems for separate decisions. The multi-task and multi-granularity complex decision-making problems in the air-ground collaborative communication scenario are transformed into a top-down three-layer hierarchical decision, namely, the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer.

[0065] Define the time slot interval Δ, deploy node clustering, trajectory planning and data transmission agents on UAVs and ground nodes respectively. At the beginning of each Δ time slot, each ground node is randomly clustered, and then selects UAVs as auxiliary communication nodes of the ground node cluster. Each UAV makes trajectory planning for the next Δ time slots based on the state, and the ground node needs to perform data transmission in each time slot to minimize the accumulated data packets of each node. Each ground node re-formulates the ground node clustering decision every Δ time slots, and each ground node needs to perform data transmission in each time slot t according to the latest cluster association decision issued by the ground node cluster. When the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated.

[0066] The drone trajectory planning layer, as an intermediate layer, inherits the clustering information of the top-level ground node clusters and drone associations. k,t , the ground node communication decision layer receives information from the top and middle layers as the bottom layer k,t and a m,t , for the ground node agent n, a n,t Determines the communication decision of the ground node. In the decision layer of the ground node cluster and the drone, the state s k,t Fork,t =[ p1 , p 2 , ..., p M , a k,t-1 ], where p 1 , p 2 , ..., P M Represents the positions of M drones, a k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by ground node cluster k k,t+Δ It can be defined as the accumulation of queues between all ground nodes and UAVs in the cluster, that is, In the UAV trajectory planning layer, the state s m,t For m,t =[p m ,q m , a k,t-1 ], where p m and q m They represent the location of UAV m and the accumulated task queue information, respectively. k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by drone m m,t+Δ It can be defined as the queue accumulation between the drone and all ground nodes in the cluster, that is, In the ground node communication decision layer, the state s k,n,t For k,n,t =[p m , p n , a n,t-1 , f n ,q n ], where p m represents the position of UAV m, a n,t-1 represents the action decision of the time slot on the ground node n, p n 、f n ,q n They represent the location, communication frequency, and accumulated task queue information of ground node n respectively; the reward obtained by ground node n can be defined as the accumulated task queue amount of ground node n, that is, r k,n,t =q n .

[0067] In the air-ground collaborative communication scenario, Figure 3As shown in the figure, based on the Actor-Critic (AC) framework, two fully connected neural networks are used to approximate the policy function π(s; θ) and the state value function V(s; w). The actor is composed of the policy network π(s; θ) and participates in the generation of the agent's actions. The critic evaluates the actions taken by the actor by using the parameterized state value function V(s; w). For the ground node communication decision, drone trajectory planning, and ground node cluster and drone association decision in the scene, the action space is discrete, so the Softmax strategy is adopted.

[0068] Define the policy network parameters θ of ground node cluster k k and the value network parameter w k , the association decision between the ground node cluster and the drone cluster in time slot t can be expressed as a k,t =π(.|s k,t θ k ), where in state s k,t According to the policy network parameters θ k The output result is selected with a certain probability. k,t ,satisfy The scheme ends after Δ time slots, based on the cumulative reward r k,t+Δ And the output V(s) of the value network k,t ;w k ), calculate the TD error δ k,t =r k,t+Δ +ηV(s k,t+Δ ;w k )-V(s k,t ;w k ), where η is the loss coefficient. The value network parameters are updated with the goal of minimizing the TD mean square error. The loss function is defined as The update formula of the value network parameters can be expressed as Where α is the weight factor. Under the guidance of TD error, the policy function network follows the loss function J(θ k ) updates the network parameters in the gradient direction, where the loss function is defined as J(θ k )=logπ(s k,t θ k )·δ k,t The update formula of the policy network parameters in each Δ time slot can be expressed as

[0069] Similarly, define the policy network parameters θ of drone m m and the value network parameter w m , the trajectory planning decision in time slot t can be expressed as a m,t =π(·|s m,t θm ). When the scheme ends after Δ time slots, the cumulative reward r obtained by the drone m,t+Δ And the output V(s) of the value network m,t ;w m ), calculate the TD error δ of UAV m m,t =r m,t+Δ +ηV(s m,t+Δ ;w m )-V(s m,t ;w m ). Define the communication decision strategy network parameters θ of the ground node n in each cluster k,n and the value network parameter w k,n The communication decision action of the ground node can be expressed as a k,n,t =π(·|s k,n,t θ k,n ). Different from the above two schemes, the communication decision action of the ground node needs to make a decision in each time slot and obtain the reward r at the end of the executed data transmission action. k,n,t , the TD error can be expressed as δ k,n,t =r k,n,t +ηV(s k,n,t+1 ;w k,n )-V(s k,n,t ;w k,n ). The updating method of the value network and the policy network executed by drone m and ground node n is the same as that of the cluster association decision layer.

[0070] Repeat the above steps until the algorithm converges, at which point the action selection decision is the optimal decision.

[0071] Environmental cognition is introduced in the decision layer of ground node cluster and drone association and the decision layer of ground node communication. Based on environmental cognition, various environmental features such as building occlusion, node location, node task queue, and node frequency usage in the air-ground collaborative communication network are comprehensively considered. The graph convolutional network can aggregate various environmental features for node clustering to achieve multi-node clustering optimization. The process of pre-training graph convolutional network is as follows: Figure 4 As shown in the figure, the output weight matrix W is the pre-trained graph convolutional network model.

[0072] First, all nodes are abstracted into a graph model representation G = (V, E), where v represents the node set and E represents the edge set. Construct the adjacency matrix A and the feature matrix X, where the dimension of the adjacency matrix A is N × N, reflecting the communication relationship between nodes. If there is a communication relationship between node i and node j, i, j ∈ {1, 2, ..., N}, then A ij =1, otherwise A ij=0, the dimension of the feature matrix X is N×F, where F represents the number of classes of node features; in the decision layer for the association between ground node clusters and UAVs, the adjacency matrix A reflects the queue transmission relationship between ground node clusters and UAVs, and the feature matrix X includes the positions of all ground nodes and UAVs and the cumulative queue volume; in the ground node communication decision layer, the adjacency matrix A reflects the queue transmission relationship between ground nodes, and the feature matrix X includes the positions of ground nodes and whether frequency mutual interference occurs between ground nodes.

[0073] Next, a graph convolution model is constructed. The number of neurons in the input layer is equal to the number of columns of the feature matrix X. The number of neurons in the output layer is determined by the number of clusters of the nodes. The Glorot method is used to initialize the uniformly distributed weight matrix W. The adjacency matrix A and the feature matrix X are preprocessed and input into the graph convolution network. The calculation formula of the preprocessed adjacency matrix A is: Where D represents the degree matrix of the adjacency matrix A, I represents the identity matrix; the calculation formula of the preprocessed feature matrix X is in and δ x They represent the mean and variance of the feature matrix X respectively.

[0074] Then, the prediction matrix Y is obtained through forward propagation, that is, in and Represent the preprocessed adjacency matrix A and feature matrix X, W respectively (1) and W (2) Respectively represent the weight matrices of the first and second layers in the graph convolutional network, ReLU and softmax represent two selected activation functions; in the graph convolutional network, the specific calculation formula for the forward propagation from the hidden layer to the hidden layer l+1 can be expressed as Where σ represents the ReLU activation function, H l and W l denote the output matrix and weight matrix of hidden layer l respectively.

[0075] Finally, the cross entropy loss function between the prediction matrix and the label matrix output by the graph convolutional network is calculated to update the weight matrix parameters, that is, Where N is the number of nodes, K is the number of clusters into which the nodes are clustered, and Y ij and P ij Represent the prediction matrix and label matrix respectively. Repeat the above steps 3.5-3.6 until the algorithm converges. At this time, if the prediction accuracy of the graph convolutional network meets the requirements, the pre-trained weight matrix W can be applied to the hierarchical decision-making algorithm based on the actor-judge framework to extract multiple environmental features to realize the cognition of complex urban environments.

[0076] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0081] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, It is characterized in that The air-ground collaborative communication hierarchical decision method comprises the following steps: Step 1: Set a complex urban environment as the scene. Aiming at malicious or unintentional dynamic interference in the scene, use drones to assist multiple ground nodes to communicate. The data transmission task generated by each ground node adopts any of the following communication methods: no communication, direct communication between ground nodes, and drone-assisted communication; Step 2: According to the complexity of the task and the granularity of the decision, the UAV-assisted air-ground collaborative communication is divided into three layers of subtasks: the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer, to construct the complex decision problem of air-ground collaborative communication in this scenario; the complex decision problem of air-ground collaborative communication is modeled as a Markov decision process, and the hierarchical reinforcement learning method based on the actor-critic framework is used to optimize the communication decisions of multiple nodes in the air-ground collaborative network; Step 3: Build a graph convolutional network model, use the graph convolutional network to aggregate multiple environmental features for node clustering, and comprehensively consider multiple environmental features including building occlusion, node location, node task queue, and node frequency in the air-ground collaborative communication network based on environmental cognition to optimize clustering of multiple nodes; Step 4: Until the hierarchical decision algorithm of air-ground collaborative communication based on environmental cognition converges, the action selection decision is the optimal decision.

2. According to the hierarchical decision-making method for air-ground collaborative communication based on environmental cognition according to claim 1, It is characterized in that Step 1 further comprises the following steps: Set a complex urban environment as the scene, which includes M drones with single antenna connection U = {U 1 , U 2 , ..., U m }, m = 1, 2, ..., M, and N ground nodes u = {u 1 ,u 2 , ..., u n }, n = 1, 2, ..., N; M drones are used to assist N ground nodes in air-ground collaborative communication in scattered buildings to deal with interference signals, and the ground nodes are divided into K ground node clusters. The drones always fly at the same height H, and the ground nodes are on the horizontal plane and fixed in position; When a ground node has a data transmission task, in each time slot, the data transmission task generated by each ground node adopts any of the following communication modes: no communication, direct communication between ground nodes, and communication assisted by UAVs. In addition, the data transmission task generated by a single ground node in a single time slot can be processed by only one UAV. The communication channel between the UAV and the ground node is described by the logarithmic range loss model, which includes a constant attenuation factor in small-scale fading and obstacle occlusion; the information rate of the nth, n∈{1, 2, ..., N}th ground node located at a constant position on the ground is given by: Among them, the transmission power P, noise power N and path loss γ of the nth ground node n ; The information rate when the drone forwards the task queue is where γ m represents the path loss when the drone forwards the task queue; when the path loss index in vacuum is set to mortar, the path loss of node n is Here, small-scale fading is modeled as a Rayleigh distributed random variable X with a scaling factor σ = 1: Rayleigh ; The attenuation due to obstacle occlusion is modeled by a discrete factor β∈{1,0.01}, where β=0.01 if there is a building occlusion, and β=1 otherwise; The path loss of signal transmission between ground nodes is calculated by the following formula: Among them, P r Indicates the signal receiving power, P t Indicates the signal transmission power, G r Represents the gain of the receiving antenna, G t represents the gain of the transmitting antenna, L represents the system loss, d represents the distance between the receiving end and the transmitting end, λ represents the wavelength of the signal, and γ represents the path loss exponent; Assume that all ground nodes use frequency band F = {f 1 , f 2 , ..., f i }, i∈{1, 2, ..., I} for communication, and there are I frequencies to choose from; suppose there is a jammer in the scene that transmits interference signals at different frequencies in each time slot, interfering with the data transmission between ground nodes, and the interference signal frequency of the jammer is repeated at the time slot period F j ={f 1 , f 2 , ..., f j }, j∈{1, 2, ..., J}; in each time slot, when the ground node selects a frequency point for communication, if the selected frequency point is the same as the frequency point of the interference signal sent by the jammer in the current time slot, it is considered that the current ground node communication is interfered, and if multiple ground nodes select the same frequency point in the same time slot, it is considered that there is communication interference between these ground nodes; when interference and mutual interference occur, the transmission rate is calculated by the following formula Among them, G j is the channel gain from the interference signal to the interfered node, P j is the interference signal power, G i is the channel gain of the mutual interference signal, A is the set of mutual interference numbers of ground nodes, γ n is the path loss of node n, P is the transmission power of the nth ground node, and N is the noise power; At the beginning of each time slot, the amount of data transmission randomly generated by each ground node is Δt, each ground node has a data accumulation queue Δn, and each drone has a data accumulation queue Δm. It is assumed that half of each time slot τ is used for transmission and half for reception, and each ground node can only transmit data to one ground node per time slot, and the ground node can only receive data transmitted by the corresponding ground node; At the beginning of the time slot, since the amount of data randomly generated by the ground node is Δt, the cumulative amount of task data to be transmitted by the ground node is Δ n =Δ n +Δ t , when the ground node chooses to use drones to assist in communication, if the transmission can be completed, that is, Δ t ≤R n At (t)·τ / 2, the amount of data received by the UAV and forwarding is completed. The current amount of data received by the uplink UAV queue is: Δ m =Δ m +Δ t , ground node queue minus the current data volume: Δ n =Δ n -Δ t ; Current data volume forwarded in downlink UAV queue: Δ m =Δ m -Δ t ; If the transmission cannot be completed, Δ t >R n At (t)·τ / 2, the maximum amount of data that the UAV can currently receive in the uplink is: Δ m =Δ m +R n (t)·τ / 2, the maximum amount of data that can be sent by the ground node queue minus: Δ n =Δ n -R n (t)·τ / 2; the maximum amount of data that can be forwarded by the UAV in the downlink: Δ m =Δ m -R m (t)·τ / 2, where R n (t) and R m (t) represents the information rate of the ground node and the UAV respectively; the unsent data is sent in the next time slot; when the ground node chooses direct communication, if the transmission can be completed, that is, Δ t ≤R n (t)·τ / 2, the ground node queue minus the current data volume: Δ n =Δ n -Δ t , if the transmission cannot be completed, that is, △ t >R n (t)·τ / 2, the maximum amount of data that can be sent by the ground node queue is subtracted: n =△ n -R n (t)·τ / 2; the unsent data is sent in the next time slot.

3. According to the hierarchical decision-making method for air-ground collaborative communication based on environmental cognition according to claim 1, It is characterized in that Step 2 further includes: In the air-ground collaborative communication scenario, the joint learning strategy of air-ground collaborative communication decision and UAV trajectory planning is reduced to a sequential decision problem. The semi-Markov decision process is used to model the hierarchical decision problem in air-ground collaborative communication, and problem planning and strategy learning are performed for different time slots. Specifically: Different tasks are modeled as semi-Markov decision processes with different time scales. A hierarchical structure is constructed through task relationships, and the multi-task and multi-granularity complex decision-making problems in air-ground collaborative communication scenarios are transformed into a top-down hierarchical decision-making with three layers: the ground node cluster and UAV association decision layer, the UAV trajectory planning layer, and the ground node communication decision layer. Define time slot intervals △, deploy node clustering, trajectory planning and data transmission agents on drones and ground nodes respectively. At the beginning of each △ time slot, each ground node is randomly clustered, and then drones are selected as auxiliary communication nodes of the ground node cluster. Each drone makes trajectory planning for the next △ time slots based on its state, and the ground node performs data transmission in each time slot to minimize the accumulated data packets of each node. Each ground node re-formulates the ground node clustering decision every △ time slots. Each ground node performs data transmission in each time slot t according to the latest cluster association decision issued by the ground node cluster. When the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent △ time slots until a new round of trajectory planning is generated. In the air-ground collaborative communication scenario, based on the actor-judge framework, two fully connected neural networks are used to approximate the policy function and the state-value function respectively. The actor is composed of the policy network and participates in the generation of the agent's actions. The judge evaluates the actions taken by the actor by using the parameterized state-value function. At the end of Δ time slots, the temporal difference error TD is calculated based on the accumulated rewards and the output of the value network, and the policy network and its own value network parameters are updated with the goal of minimizing the TD mean square error; Repeat the above steps until the algorithm converges, and take the action selection decision at this time as the optimal decision.

4. According to claim 3, the hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, It is characterized in that The UAV trajectory planning layer is an intermediate layer that inherits the top-level ground node cluster and the UAV associated clustering information a k,t , the ground node communication decision layer receives information from the top and middle layers as the bottom layer k,t and a m,t , for the ground node agent n, a n,t Determine the communication decisions of ground nodes; In the decision layer of ground node cluster and drone association, state s k,t For k,t =[p 1 , p 2 , ..., p M , a k,t-1 ], where p 1 , p 2 , ..., p M Represents the positions of M drones, a k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by ground node cluster k k,t+Δ Defined as the accumulation of queues between all ground nodes and drones in the cluster In the UAV trajectory planning layer, the state s m,t For m,t =[p m ,q m , a k,t-1 ], where p m and q m They represent the location of UAV m and the accumulated task queue information, respectively. k,t-1 represents the action decision of the kth ground node cluster in the time slot; the reward r obtained by drone m m,t+Δ It is defined as the queue accumulation between all ground nodes in the cluster associated with the drone In the ground node communication decision layer, the state s k,n,t For k,n,t =[p m , p n , a n,t-1 , f n ,θ n ], where a n,t-1 represents the action decision of the time slot on the ground node n, p n 、f n ,q n They represent the location, communication frequency, and accumulated task queue information of ground node n respectively; The reward obtained by ground node n is defined as the accumulated task queue volume r of ground node n k,n,t =q n .

5. According to claim 4, the hierarchical decision-making method for air-ground collaborative communication based on environmental cognition, It is characterized in that The process of optimizing the communication decision of multiple nodes in the air-ground collaborative network using the hierarchical reinforcement learning method based on the actor-critic framework includes the following steps: Define the cluster association strategy network parameters θ of the ground node cluster k k and cluster association value network parameter w k , the association decision between the ground node cluster and the drone cluster in time slot t is expressed as a k,t =π(·|s k,t θ k ), where in state s k,t According to the cluster association strategy network parameter θ k The output result is selected with a certain probability. k,t ,satisfy The decision to associate the ground node cluster with the drone cluster ends after Δ time slots, based on the accumulated reward r k,t+Δ And the output V(s) of the cluster association value network k,t ;w k ), calculate the TD error δ k,t =r k,t+Δ +ηV(s k,t+Δ ;w k )-V(s k,t ;w k ), where η is the loss coefficient; the cluster association value network parameters are updated with the goal of minimizing the TD mean square error, and the loss function is defined as The cluster association value network along the loss function L(w k ) The update formula of the gradient direction update parameter is expressed as Where α is the weight factor, and the cluster association strategy function network is guided by the TD error along the loss function J(θ k ) updates the cluster association network parameters in the gradient direction, where the loss function is defined as J(θ k )=logπ(s k,t θ k )·δ k,t , the update formula of the cluster association strategy network parameters in each Δ time slot is expressed as Define the trajectory planning strategy network parameters θ of UAV m m and trajectory planning value network parameter w m , the trajectory planning decision in time slot t is represented as a m,t =π(·|s m,t θ m ); When the trajectory planning decision ends after Δ time slots, the cumulative reward r obtained by the drone m,t+Δ And the output V(s) of the trajectory planning value network m,t ;w m ), calculate the TD error δ of UAV m m,t =t m,t+Δ +ηV(s m,t+Δ ;w m )-V(s m,t ;w m );Define the communication decision strategy network parameters θ of the ground node n in each cluster k,n and the communication decision value network parameter w k,n , the communication decision action of the ground node is represented as a k,n,t =π(·|s k,n,t θ k,n ); The communication decision action of the ground node is made in each time slot, and the reward r is obtained at the end of the executed data transmission action k,n,t , TD error is expressed as δ k,n,t =r k,n,t +ηV(s k,n,t+1 ;w k,n )-V(s k,n,t ;w k,n ).

6. The hierarchical decision-making method for air-ground collaborative communication based on environmental cognition according to claim 1, It is characterized in that Step 3 includes the following sub-steps: Step 3.1, abstract all nodes into a graph model representation G = (V, E), where V represents the node set and E represents the edge set; Step 3.2, construct the adjacency matrix A and the feature matrix X; The dimension of the adjacency matrix A is N×N, which is used to reflect the communication relationship between nodes. If there is a communication relationship between node i and node j, i, j∈{1, 2, …, N}, then A ii =1, otherwise A ij =0, the dimension of the feature matrix X is N×F, where F represents the number of classes of node features; Step 3.3, construct a graph convolution model. The number of neurons in the input layer is equal to the number of columns of the feature matrix X. The number of neurons in the output layer is determined by the number of clusters of the node clusters. The Glorot method is used to initialize the uniformly distributed weight matrix W. Step 3.4: Preprocess the adjacency matrix A and feature matrix X and input them into the graph convolutional network. Then, the prediction matrix Y is obtained through forward propagation. in and Represent the preprocessed adjacency matrix A and feature matrix X, W respectively (1) and W (2) They represent the weight matrices of the first and second layers in the graph convolutional network respectively, and ReLU and softmax represent two selected activation functions; Step 3.5, calculate the cross entropy loss function between the prediction matrix and the label matrix output by the graph convolutional network to update the weight matrix parameters. Where N is the number of nodes, K is the number of clusters into which the nodes are clustered, and Y ij and P ij Represent the prediction matrix and label matrix respectively; Step 3.6: Repeat steps 3.4-3.5 until the algorithm converges. If the prediction accuracy of the graph convolutional network meets the requirements, the pre-trained weight matrix W is applied to the hierarchical decision-making algorithm based on the actor-judge framework to extract multiple environmental features to recognize the complex urban environment.

7. The hierarchical decision-making method for air-ground collaborative communication based on environmental cognition according to claim 6, It is characterized in that In the decision layer of ground node cluster and UAV association, the adjacency matrix A reflects the queue transmission relationship between ground node cluster and UAV, and the feature matrix X includes the positions of all ground nodes and UAVs and the cumulative queue volume; In the ground node communication decision layer, the adjacency matrix A reflects the queue transmission relationship between ground nodes, and the feature matrix X includes the location of the ground nodes and whether frequency interference occurs between the ground nodes.

8. The hierarchical decision-making method for air-ground collaborative communication based on environmental cognition according to claim 6, It is characterized in that In step 3.4, the calculation formula for the preprocessed adjacency matrix A is Where D represents the degree matrix of the adjacency matrix A, and I represents the identity matrix; The calculation formula of the preprocessing feature matrix X is: in and δ x Respectively represent the mean and variance of the feature matrix X; In the graph convolutional network, the forward propagation from the hidden layer to the hidden layer l+1 is expressed as Where σ represents the ReLU activation function, H l and W l Denote the output matrix and weight matrix of the hidden layer, respectively.

Citation Information

Patent Citations

  • Route establishment method and system for self-organizing network of unmanned aerial vehicle

    CN112822752A

  • Air combat multi-intention strategy autonomous generation method based on deep reinforcement learning

    CN116432030A

  • Cell association, trajectory planning and unloading scheduling combined dynamic decision-making method

    CN116669069A

  • Multi-machine hunting method and device for hierarchical collaborative learning, electronic equipment and medium

    CN117350326A

  • Method for optimal scheduling of energy-carrying communication trajectory of unmanned aerial vehicle

    CN118226885A

Cited By

  • Task cognition-driven low-altitude electromagnetic defense resource collaborative dynamic management method and system

    CN120050683A