Air-ground collaborative network topology robustness modeling method based on reinforcement learning

Abstract

The invention discloses an air-ground collaborative network topology robustness modeling method based on reinforcement learning. The method comprises the following steps: all network nodes in the air and on the ground are managed and controlled in a centralized manner; constructing a complete ground network through a wired link; determining the communication capacity and the communication distance of the air unmanned aerial vehicle nodes, and determining all ground and air node sets in the communication range of each unmanned aerial vehicle; applying a reinforcement learning method to determine relevant parameters such as a state set, an action set and a feedback function; the number of algorithm iterations is set, algorithm training is carried out in a time slot in which no large position change occurs to nodes of the unmanned aerial vehicle in the air, and construction of a network topology model is completed; and when the position of the aerial unmanned aerial vehicle node changes greatly, the step 6 is executed repeatedly, and the updated network topology model is established again. The method can improve the construction dynamics and stability of the air-ground cooperative network topology, improves the robustness and survivability, and is suitable for an air-ground cooperative network scene of a high-speed motion unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of space-ground integrated information network virtualization and machine learning, and specifically relates to a reinforcement learning-based air-ground collaborative network topology robustness modeling method. Background Art

[0002] With the rapid development of drone technology, satellite communications and ground mobile networks, Air-Ground Integrated Networks (AGIN) have shown great potential in a variety of application scenarios, including emergency rescue, disaster monitoring, air combat, etc. AGIN effectively integrates ground communication networks with air communication networks to achieve seamless transmission and sharing of information, provide a wider coverage and higher network reliability, which is of great significance to improving the quality of communication services and meeting the needs of modern society for high-speed and stable network connections.

[0003] However, the development of existing network technologies in various fields of space and ground is different, and the network structures vary greatly. The UAV network has wide coverage, high mobility, strong time-varying topology but limited resources, and the ground network has abundant resources but small coverage and poor mobility. These characteristics make AGIN face a series of key problems in topology design and maintenance, such as the impact of dynamic changes in network topology on communication reliability, coordination and optimization between different levels, and network robustness and stability. Therefore, building an effective topology robustness modeling method to improve the adaptability of AGIN in complex environments is one of the main challenges currently faced.

[0004] At present, the topology modeling of the integrated space-ground network mainly includes topology optimization based on graph theory, simulation evaluation methods and traditional network optimization algorithms. Among them, the graph theory method is more intuitive when dealing with network topology, but it usually fails to fully consider the impact of the dynamic environment; the simulation method can simulate the actual environment well, but it is limited by computing resources and usually consumes a lot of time, which is not suitable for network scenarios with strong node mobility; traditional optimization algorithms often ignore the coordination and optimization of heterogeneous networks when facing the particularity of air-ground collaboration. Therefore, the existing methods still have certain limitations in adapting to dynamic changes and improving network robustness. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning, which specifically includes the following steps:

[0006] S1. Centrally manage and control all network nodes in the air and on the ground;

[0007] The aerial nodes are all drones, and the ground nodes include the ground command center and ground communication relay. The drones can establish wireless communication links with each other and the ground communication relay.

[0008] S2, build a complete ground network through wired links, assuming that the ground network topology has been determined in advance;

[0009] The current ground communication relay can establish wired communication links with other ground communication relays and the ground command and control center, assuming that the ground network topology has been determined in advance;

[0010] Assuming that each node on the ground is in a fixed position, appropriate wired communication links are established according to the position of each node to construct a fixed network topology on the ground;

[0011] S3, determine the communication capacity and communication distance of the aerial drone nodes, and determine the set of all ground and aerial nodes within the communication range of each drone;

[0012] S4. Apply reinforcement learning methods to determine relevant parameters such as state set, action set, and feedback function;

[0013] S5. Set the number of algorithm iterations, perform algorithm training in the time slot when the aerial drone nodes do not undergo significant position changes, and complete the network topology model construction;

[0014] S6. When the position of the aerial drone node changes significantly, step S5 is repeated to re-establish an updated network topology model;

[0015] The above process is repeated until the number of reinforcement learning iterations reaches the upper limit L, and the final air-ground collaborative network topology model is obtained; subsequently, when the position of the aerial drone node changes significantly, the above process is repeated to re-establish the updated network topology model.

[0016] In step S2 of an embodiment of the present invention, in the air-ground collaborative network, a physical network G is defined as {V g ,V s ,E gg ,E gs ,E ss}, where V g Represents the ground node set, denoted as V g = {V c ,V r}, where V c Represents the ground command and control center assembly, V r represents the ground communication relay set, V s Represents the set of aerial drone nodes; E gg Represents the ground link set, denoted as E gg ={Ecr ,E rr}, where E cr Represents the set of wired links established between all ground command and control centers and ground communication relays, E rr Represents the set of wired links established between any two ground communication relays; E gs Represents the set of wireless communication links established between all ground communication relays and aerial drones; E ss represents the wireless communication link between the aerial drones; where V g 、V s 、E gg All are static collections, E gs 、E ss It is a dynamic collection, and the collection elements are updated in real time based on the wireless communication link connected to the drone.

[0017] In step S3 of another embodiment of the present invention, it is assumed that the relevant parameters of each drone are the same, and the communication distance is represented by R, that is, the drone can only establish a wireless communication link with other nodes within a sphere with itself as the center and R as the radius. Since drone B and ground communication relay E are not within the communication range of drone A, drone A can only establish a wireless communication link with drone C and ground communication relay D; the communication capacity is represented by M, that is, each drone can establish a wireless communication link with up to M drones and ground communication relays.

[0018] In step S4 of another embodiment of the present invention, assuming that the number of training iterations of reinforcement learning is L, each iterative training will construct at most one wireless link, and the environment state of reinforcement learning is defined as the total number of wireless links established after the algorithm starts training, including two wireless links between drones and between drones and ground communication relays. Therefore, the state set is represented by S={0,1,2,3,...,L}, and the state set size is L+1;

[0019] The action of reinforcement learning is defined as selecting the current drone, establishing a wireless communication link with another node within the communication range, selecting a drone based on the number of communicable nodes within the communication range of the current drone, and establishing a wireless communication link with other drones or ground communication relays other than the drone within the communication range. Assuming that there is at least one drone and ground communication relay node within the communication range of each drone in the air, the total number of other drones and ground communication relay nodes within the communication range of the drone, m, can be a positive integer greater than or equal to 2. When m>M, m is represented as M. + , then the value range of m is the set {2,3,...,M,M +}, a total of M elements, and define the label at the same time:

[0020]

[0021] The action is represented by A(t)=(m,Conn), and the action set size is M*2;

[0022] The feedback function is expressed as:

[0023]

[0024] Where S(t) represents the state of reinforcement learning at time t; Sub represents the number of subnetworks directly connected to the current drone nodes, N represents the total number of drone nodes, Sub∈[1,N]; d max and d min Respectively represent the maximum and minimum node degrees of all drone nodes; α and β are weight coefficients;

[0025] The update formula for setting the Q value is:

[0026] Q(S(t),A(t))=R(S(t),A(t))+γ*max{Q(S(t+1),A(t+1))} (3)

[0027] Among them, Q represents the total reward expected from taking an action in a given state, Q(S(t), A(t)) represents the Q value corresponding to the state S(t) and action A(t) in the reinforcement learning Q table, R(S(t), A(t)) represents the feedback value obtained by selecting action A(t) in the reinforcement learning state S(t), and γ is the greed factor.

[0028] In a specific embodiment of the present invention, the greed factor is set to 0.8.

[0029] In step S5 of another embodiment of the present invention, a wireless communication link is established for the air-ground collaborative network using a reinforcement learning method, which specifically includes the following steps:

[0030] S5.1. Initialize the reinforcement learning Q table to an all-zero matrix with L+1 rows and M*2 columns, where the row elements represent all possible states and the column elements represent all possible actions.

[0031] S5.2, set the initial state S(0) = 0;

[0032] S5.3, set the current state at time t to S(t), that is, the total number of links between drones and drones, and between drones and ground communication relays at time t, select an action A(t) = (m, Conn) according to the ε-greedy method, randomly select a drone with m nodes in the communication range as the current drone to be linked, if Conn = 0, randomly select another drone within the communication range of the current drone to establish a communication link, if Conn = 1, randomly select a ground communication relay to establish a communication link with the current drone, and after executing the action, obtain the updated state S(t+1);

[0033] S5.4. Calculate R(S(t), A(t)) according to formula (2);

[0034]

[0035] S5.5. Calculate Q(S(t), A(t)) according to formula (3);

[0036] Q(S(t),A(t))=R(S(t),A(t))+γ*max{Q(S(t+1),A(t+1))} (3)

[0037] S5.6. Update the Q table according to the calculation result of step S5.5;

[0038] S5.7. Update the current state S(t) to S(t+1);

[0039] S5.8. Loop through steps S5.3 to S5.7 until the number of iterations reaches the upper limit L, completing the construction of the air-ground collaborative network topology.

[0040] In step S6 of another embodiment of the present invention, for the aerial drone node, the number of wireless links connected to the drone at the current time t is determined. gs |+|E ss |, that is, the state S(t) at time t; find the action A(t) corresponding to the maximum Q value that can be generated by the state S(t) in the Q table, randomly select a current drone node from the drone set of communicable nodes that meet the conditions within the communication range, assuming it is drone U, and establish a wireless communication link between the current drone U and a ground communication relay, assuming that the ground communication relay is D; calculate the feedback value R(S(t), A(t)) that can be obtained by executing action A(t) according to formula (2), and calculate the Q value after executing action A(t) in state S(t) according to formula (3); update the Q table, and at the same time in E gs The wireless link established between the UAV U and the ground communication relay D is added, and the environmental state is transferred to S(t+1).

[0041] The present invention is used to solve the problem that the prior art does not consider the coordination of the air-ground layer and the dynamics of the aerial nodes in the process of network topology modeling, and does not consider the influence of network robustness on the stability of data transmission.

[0042] The present invention has the following beneficial effects:

[0043] The present invention centrally controls the aerial drones, ground command and control centers, and ground communication relay nodes in the air-ground collaborative network. Based on the communication range and communication capacity of the known drone nodes, the communicative drone nodes and ground communication relay nodes of each drone are determined according to the drone position. With the overall network connectivity and drone node degree balance as the goal, the reinforcement learning method is used to select appropriate wireless communication links for link building, and the topology modeling of the air-ground collaborative network is completed in a relatively short time, aiming to improve the robustness and anti-destruction of the air-ground collaborative network.

[0044] The present invention does not need to relearn the reinforcement learning model after a large position change of the aerial drone node occurs. By directly training and running within a smaller number of algorithm iterations, the network topology can be quickly reconstructed under the condition of dynamic movement of the drone node, aiming to improve the topology construction stability of the air-ground collaborative network in a highly dynamic environment.

[0045] To sum up, the present invention mainly considers the coordination at the air-ground level and the dynamics of aerial nodes, improves the dynamics and stability of the construction of the air-ground collaborative network topology, and at the same time improves the robustness and anti-destruction performance, which is suitable for air-ground collaborative network scenarios with high-speed movement of aerial drone nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the structure diagram of the air-ground collaborative network;

[0047] Figure 2 This is a schematic diagram of the drone's communication range;

[0048] Figure 3 This is a flowchart of the algorithm for building the topology robustness of air-ground collaborative networks based on reinforcement learning. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0050] The present invention proposes a method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning, which specifically includes the following steps:

[0051] S1. Centrally manage and control all network nodes in the air and on the ground.

[0052] See also Figure 1In step S1, the aerial nodes are all drones, and the ground nodes include the ground command center and the ground communication relay. The drones can establish wireless communication links with the drones and the ground communication relays.

[0053] S2. Build a complete ground network through wired links, assuming that the ground network topology has been determined in advance.

[0054] Currently, ground communication relays can establish wired communication links with other ground communication relays and ground command and control centers, and it is assumed that the ground network topology has been determined in advance.

[0055] For example, in the air-ground collaborative network, the physical network G is defined as {V g ,V s ,E gg ,E gs ,E ss}, where V g Represents the ground node set, denoted as V g = {V c ,V R}, where V c Represents the ground command and control center assembly, V R represents the ground communication relay set, V s Represents the set of aerial drone nodes; E gg Represents the ground link set, denoted as E gg ={E cr ,E rr}, where E cr Represents the set of wired links established between all ground command and control centers and ground communication relays, E rr Represents the set of wired links established between any two ground communication relays; E gs Represents the set of wireless communication links established between all ground communication relays and aerial drones; E ss represents the wireless communication link between the aerial drones. Therefore, V g 、V s 、E gg All are static collections, E gs 、E ss It is a dynamic collection, and the collection elements are updated in real time according to the wireless communication link connected to the drone. Assuming that each node on the ground is in a fixed position, a suitable wired communication link is established according to the position of each node to construct a fixed network topology on the ground.

[0056] S3. Determine the communication capacity and communication distance of the aerial drone nodes, and determine the set of all ground and aerial nodes within the communication range of each drone.

[0057] See also Figure 2, assuming that the relevant parameters of each drone are the same, the communication distance is expressed as R, that is, the drone can only establish wireless communication links with other nodes in a sphere with itself as the center and R as the radius. Since drone B and ground communication relay E are not within the communication range of drone A, drone A can only establish wireless communication links with drone C and ground communication relay D. The communication capacity is expressed as M, that is, each drone can establish wireless communication links with at most M drones and ground communication relays.

[0058] S4. Apply reinforcement learning methods to determine relevant parameters such as state set, action set, feedback function, etc.

[0059] Assume that the number of training iterations of reinforcement learning is L, and each training iteration will build at most one wireless link. The environment state of reinforcement learning is defined as the total number of wireless links established after the algorithm starts training, including two types of wireless links between drones and between drones and ground communication relays. Therefore, the state set is represented by S = {0, 1, 2, 3, ..., L}, and the state set size is L + 1;

[0060] The action of reinforcement learning is defined as selecting the current drone to establish a wireless communication link with another node within the communication range. First, a drone is selected according to the number of communicable nodes within the communication range of the current drone. Secondly, a wireless communication link is established with other drones or ground communication relays other than the drone within the communication range. Assuming that there is at least one drone and ground communication relay node within the communication range of each drone in the air, the sum of the number of other drones and ground communication relay nodes within the communication range of the drone can be a positive integer greater than or equal to 2. When m>M, m is represented as M. + , then the value range of m is the set {2,3,...,M,M +}, a total of M elements, and define the label at the same time:

[0061]

[0062] Therefore, the action is represented as A(t) = (m, Conn), and the action set size is M*2;

[0063] The purpose of the feedback function of reinforcement learning is to measure the connectivity and invulnerability of the network. The feedback function is expressed as:

[0064]

[0065] Where Sub represents the number of sub-networks directly connected to the current drone nodes, and N represents the total number of drone nodes, so Sub∈[1,N]; d max and d min Respectively represent the maximum and minimum node degrees of all drone nodes; α and β are weight coefficients.

[0066] And set the Q value update formula as:

[0067] Q(S(t),A(t))=R(S(t),A(t))+γ*max{Q(S(t+1),A(t+1))} (3)

[0068] Wherein, Q represents the total reward expected from taking an action in a given state, Q(S(t), A(t)) represents the Q value corresponding to the state S(t) and the action A(t) in the reinforcement learning Q table, R(S(t), A(t)) represents the feedback value obtained by selecting the action A(t) in the reinforcement learning state S(t), S(t) and A(t) represent the state and action of the reinforcement learning at time t, respectively, and γ is the greed factor, which is set to 0.8 in a specific embodiment of the present invention.

[0069] S5. Set the number of algorithm iterations, perform algorithm training in the time slots when the aerial drone nodes do not undergo significant position changes, and complete the construction of the network topology model.

[0070] See also Figure 3 , the reinforcement learning method is applied to establish a wireless communication link for the air-ground collaborative network, which specifically includes the following steps:

[0071] S5.1. Initialize the reinforcement learning Q table to an all-zero matrix with L+1 rows and M*2 columns, where the row elements represent all possible states and the column elements represent all possible actions.

[0072] S5.2, set the initial state S(0) = 0;

[0073] S5.3. Set the current state at time t to S(t), which is the total number of links between drones and drones and between drones and ground communication relays at time t. According to the ε-greedy method (Richard S.Sutton, Andrew G.Barto.Reinforcement learning: an introduction[J].IEEE Transactions on Neural Networks, 1998, 9(5): 1054-1054.), select an action A(t) = (n, Conn). Randomly select a drone with m nodes in the communication range as the current drone to be linked. If Conn = 0, randomly select another drone within the communication range of the current drone to establish a communication link. If Conn = 1, randomly select a ground communication relay to establish a communication link with the current drone. After executing the action, the updated state S(t+1) is obtained.

[0074] S5.4. Calculate R(S(t), A(t)) according to formula (2);

[0075] S5.5. Calculate Q(S(t), A(t)) according to formula (3);

[0076] S5.6. Update the Q table according to the calculation result of step S5.5;

[0077] S5.7. Update the current state S(t) to S(t+1);

[0078] S5.8. Loop through steps S5.3 to S5.7 until the number of iterations reaches the upper limit L, completing the construction of the air-ground collaborative network topology.

[0079] S6. When the position of the aerial drone node changes significantly, step S5 is repeated to re-establish the updated network topology model.

[0080] For example, for an aerial drone node, first determine the number of wireless links connected to the drone at the current time t |E gs |+|E ss |, that is, the state S(t) at time t; secondly, find the action A(t) corresponding to the maximum Q value that can be generated by the state S(t) in the Q table. Assuming A(t) = (3,1), randomly select a current drone node from the drone set with 3 communicable nodes within the communication range, assuming it is drone U, and establish a wireless communication link between the current drone U and a ground communication relay, assuming that the ground communication relay is D; then calculate the feedback value R(S(t), A(t)) that can be obtained by executing action A(t) according to formula (2), and calculate the Q value after executing action A(t) in state S(t) according to formula (3); finally, update the Q table and at the same time in E gs The wireless link established between the UAV U and the ground communication relay D is added, and the environmental state is transferred to S(t+1).

[0081] Repeat the above process until the number of reinforcement learning iterations reaches the upper limit L, and obtain the final air-ground collaborative network topology model. When the position of the aerial drone node changes significantly, repeat the above process and re-establish the updated network topology model.

[0082] When a large position change occurs to the drone node in step 6, the main consideration is the disconnection of the wireless communication link connected to the drone.

[0083] The present invention centrally controls the aerial drones, ground command and control centers and ground communication relay nodes in the air-ground collaborative network, determines the communicative drone nodes and communication relay nodes of each drone according to the drone position on the basis of the communication range and communication capacity of the known drone nodes; and takes the overall network connectivity and the balance of the drone node degree as the goal, uses the reinforcement learning method to select appropriate wireless communication links for link building, and completes the topology modeling of the air-ground collaborative network in a relatively short time, aiming to improve the robustness and anti-destruction of the air-ground collaborative network; the present invention does not need to relearn the reinforcement learning model after a large position change of the aerial drone node, and directly trains and runs within a small number of algorithm iterations, thereby completing the rapid reconstruction of the network topology under the dynamic movement of the drone node, aiming to improve the topology construction stability of the air-ground collaborative network in a high dynamic environment; the present invention mainly considers the coordination at the air-ground level and the dynamicity of the aerial nodes, improves the dynamicity and stability of the construction of the air-ground collaborative network topology, and at the same time improves the robustness and anti-destruction, and is suitable for air-ground collaborative network scenarios with high-speed movement of aerial drone nodes.

Claims

1. A method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning, characterized in that: The specific steps include: S1. Centrally manage and control all network nodes in the air and on the ground; The aerial nodes are all drones, and the ground nodes include the ground command center and ground communication relay. The drones can establish wireless communication links with each other and the ground communication relay. S2, build a complete ground network through wired links, assuming that the ground network topology has been determined in advance; The current ground communication relay can establish wired communication links with other ground communication relays and the ground command and control center, assuming that the ground network topology has been determined in advance; Assuming that each node on the ground is in a fixed position, appropriate wired communication links are established according to the position of each node to construct a fixed network topology on the ground; S3, determine the communication capacity and communication distance of the aerial drone nodes, and determine the set of all ground and aerial nodes within the communication range of each drone; S4. Apply reinforcement learning methods to determine relevant parameters such as state set, action set, and feedback function; S5. Set the number of algorithm iterations, perform algorithm training in the time slot when the aerial drone nodes do not undergo significant position changes, and complete the network topology model construction; S6. When the position of the aerial drone node changes significantly, step S5 is repeated to re-establish an updated network topology model; The above process is repeated until the number of reinforcement learning iterations reaches the upper limit L, and the final air-ground collaborative network topology model is obtained; subsequently, when the position of the aerial drone node changes significantly, the above process is repeated to re-establish the updated network topology model.

2. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 1, characterized in that: In step S2, in the air-ground collaborative network, a physical network G is defined as {V g ,V s ,E gg ,E gs ,E ss }, where V g Represents the ground node set, denoted as V g = {V c ,V r }, where V c Represents the ground command and control center assembly, V r represents the ground communication relay set, V s Represents the set of aerial drone nodes; E gg Represents the ground link set, denoted as E gg ={E cr ,E rr }, where E cr Represents the set of wired links established between all ground command and control centers and ground communication relays, E rr Represents the set of wired links established between any two ground communication relays; E gs Represents the set of wireless communication links established between all ground communication relays and aerial drones; E ss Represents the wireless communication link between aerial drones; Among them, V g 、V s 、E gg All are static collections, E gs 、E ss It is a dynamic collection, and the collection elements are updated in real time based on the wireless communication link connected to the drone.

3. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 1, characterized in that: In step S3, it is assumed that the relevant parameters of each drone are the same, and the communication distance is represented by R, that is, the drone can only establish a wireless communication link with other nodes in a sphere with itself as the center and R as the radius. Since drone B and ground communication relay E are not within the communication range of drone A, drone A can only establish a wireless communication link with drone C and ground communication relay D; the communication capacity is represented by M, that is, each drone can establish a wireless communication link with at most M drones and ground communication relays.

4. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 2, characterized in that: In step S4, assuming that the number of training iterations of reinforcement learning is L, each iterative training will build at most one wireless link. The environment state of reinforcement learning is defined as the total number of wireless links established after the algorithm starts training, including two types of wireless links between drones and between drones and ground communication relays. Therefore, the state set is represented by S = {0, 1, 2, 3, ..., L}, and the state set size is L + 1; The action of reinforcement learning is defined as selecting the current drone, establishing a wireless communication link with another node within the communication range, selecting a drone based on the number of communicable nodes within the communication range of the current drone, and establishing a wireless communication link with other drones or ground communication relays other than the drone within the communication range. Assuming that there is at least one drone and ground communication relay node within the communication range of each drone in the air, the total number of other drones and ground communication relay nodes within the communication range of the drone, m, can be a positive integer greater than or equal to 2. When m>M, m is represented as M. + , then the value range of m is the set {2,3,...,M,M + }, a total of M elements, and define the label at the same time: The action is represented as A(t)=(m,Conn), and the action set size is M*2; The feedback function is expressed as: Where S(t) represents the state of reinforcement learning at time t; Sub represents the number of subnetworks directly connected to the current drone nodes, N represents the total number of drone nodes, Sub∈[1,N]; d max and d min Respectively represent the maximum and minimum node degrees of all drone nodes; α and β are weight coefficients; The update formula for setting the Q value is: Q(S(t),A(t))=R(S(t),A(t))+γ*max{Q(S(t+1),A(t+1))} (3) Among them, Q represents the total reward expected from taking an action in a given state, Q(S(t), A(t)) represents the Q value corresponding to the state S(t) and action A(t) in the reinforcement learning Q table, R(S(t), A(t)) represents the feedback value obtained by selecting action A(t) in the reinforcement learning state S(t), and γ is the greed factor.

5. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 4, characterized in that: The greediness factor is set to 0.

8.

6. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 4, characterized in that: In step S5, a reinforcement learning method is applied to establish a wireless communication link for the air-ground collaborative network, which specifically includes the following steps: S5.

1. Initialize the reinforcement learning Q table to an all-zero matrix with L+1 rows and M*2 columns, where the row elements represent all possible states and the column elements represent all possible actions. S5.2, set the initial state S(0) = 0; S5.3, set the current state at time t to S(t), that is, the total number of links between drones and drones, and between drones and ground communication relays at time t, select an action A(t) = (m, Conn) according to the ε-greedy method, randomly select a drone with m nodes in the communication range as the current drone to be linked, if Conn = 0, randomly select another drone within the communication range of the current drone to establish a communication link, if Conn = 1, randomly select a ground communication relay to establish a communication link with the current drone, and after executing the action, obtain the updated state S(t+1); S5.

4. Calculate R(S(t), A(t)) according to formula (2); S5.

5. Calculate Q(S(t), A(t)) according to formula (3); Q(S(t),A(t))=R(S(t),A(t))+γ*max{Q(S(t+1),A(t+1))} (3) S5.

6. Update the Q table according to the calculation result of step S5.5; S5.

7. Update the current state S(t) to S(t+1); S5.

8. Loop through steps S5.3 to S5.7 until the number of iterations reaches the upper limit L, completing the construction of the air-ground collaborative network topology.

7. The method for modeling the topology robustness of an air-ground collaborative network based on reinforcement learning as claimed in claim 4, characterized in that: In step S6, for the aerial drone node, the number of wireless links connected to the drone at the current time t is determined |E gs |+|E ss |, that is, the state S(t) at time t; find the action A(t) corresponding to the maximum Q value that can be generated by the state S(t) in the Q table, randomly select a current drone node from the drone set of communicable nodes that meet the conditions within the communication range, assuming it is drone U, and establish a wireless communication link between the current drone U and a ground communication relay, assuming that the ground communication relay is D; According to formula (2), the feedback value R(S(t), A(t)) obtained by executing action A(t) is calculated. According to formula (3), the Q value after executing action A(t) in state S(t) is calculated. The Q table is updated and at the same time, gs The wireless link established between the UAV U and the ground communication relay D is added, and the environmental state is transferred to S(t+1).

Images