Intelligent agile networking method and device for large-scale emergency wireless communication network

Abstract

The application provides a large-scale emergency wireless communication network intelligent agile networking method and device. The method is based on orthogonal reward multi-agent deep deterministic policy gradient reinforcement learning (OTR-MADDPG) of topology perception and road network optimization, an agent learning environment Env containing the initial position distribution of the wireless communication network node, the environment constraint and the target constraint is designed; an improved action space based on road network search is designed for the MOV_BS agent, and the updating mode of the improved policy network pi based on the state input of the neighbor nodes in the connected branch is designed; an action space based on the access / rejection of the LEO satellite is designed for the LEO agent, and the updating mode of the policy network based on the MOV_BS coordinate-over-the-top satellite state pair is designed; a compound reward function is designed; the learning environment Env, the action generation mode of the agent, the updating mode of the policy network, the reward function R are integrated into the existing framework of MADDPG. The application effectively solves the problems of long topology construction time and poor node collaboration of the emergency communication network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wireless ad hoc networks, in particular to a large-scale emergency wireless communication network intelligent agile networking method and device, which improves the convergence time and practicality of the training process through road network optimization, improves the cooperation ability of the agent through the method of topology perception based on connected components, and solves the problem of uneven distribution of training rewards in the same environment through the method of orthogonal rewards. BACKGROUND

[0002] Multi-agent deep reinforcement learning plays an important role in the communication guarantee and post-disaster communication reconstruction scene of earthquake, forest fire, and large-scale flood disasters. The network nodes involved in emergency wireless communication are divided into mobile base station nodes (MOV_BS), unmanned aerial vehicle (UAV) air nodes, and low earth orbit satellite nodes (LEO). The network is a self-organizing task coordination network, which generates network topology according to task coordinates, follows according to task location migration, and adjusts link resources according to real-time distribution of tasks. Among them, MOV_BS is the first type of agent, UAV node is the extension of MOV_BS antenna, and LEO is the second type of agent, which is the auxiliary of ground network. The two types of agent scenes are coupled, and the exploration space is extremely large. Through the analysis of the situation of the disaster area and the modeling of the wireless channel environment, combined with wireless ad hoc network technology, the coverage of the disaster area and the guarantee route can be realized, and the interconnection with the rear command center can be realized, which can enhance the automatic networking capability in emergency tasks.

[0003] However, in the large-scale, high-uncertainty, and multi-node-type emergency communication scene, wireless ad hoc networks face many challenges and problems. Mobile communication relays are often easy to reach and deploy, and whether the actual wireless communication link is smooth is not only constrained by the performance parameters of the receiving and transmitting devices themselves, but also limited by the channel environment, including terrain, buildings, weather, and electromagnetic interference. The logical networking formed by relying on simple communication radius is only simulation feasible, but it is difficult to deploy, and even impossible to deploy, and the simulation results are not reliable. Secondly, in the highly uncertain environment, the network needs to have certain autonomous ability, and the whole or part of the network needs to have agile adjustment and link recovery ability in the face of increasing obstacles, expanding target coverage area, and node failure. Thirdly, the low earth orbit satellite constellation cooperating with the ground network needs to process information backhaul of multiple disaster areas, especially areas that the ground network cannot cover. The data packet distribution from the ground mobile base station is directly related to the ground emergency network topology, and the two network businesses are closely coupled but completely unrelated systems. In the face of complex and variable emergency scenes, the flexibility is poor.

[0004] Representative topology generation algorithms include hierarchical and staged topology generation algorithms, disjoint path topology generation algorithms, and topology generation algorithms based on homogenization of node degrees. Among them, the hierarchical and staged topology generation algorithm plans nodes layer by layer according to levels, judges whether the network is feasible through an evaluator, and centrally manages global nodes; the topology generation algorithm based on disjoint paths solves the backbone network topology by finding a maximum connected dominating set; the topology generation algorithm based on homogenization of node degrees adjusts the connection mode of the existing topology to make the node degree distribution of the backbone network relatively uniform, thereby improving the invulnerability.

[0005] MADDPG is a deterministic policy generation multi-agent deep reinforcement learning framework, and the networking process is a process from isolation to connectivity, which is more suitable for the architecture of centralized training and distributed execution under this framework. When training multiple agents, sharing observations is a common way to improve convergence speed and synergy. A representative method is the agent clustering method based on growing neural gas (GNG), which clusters a group of agents through the observation correlation of the agents. The observations of the group of agents form a joint observation, and a joint action is formed based on the joint observation. This causes a certain performance loss, but the agent synergy and convergence speed are greatly improved.

[0006] Two types or more of agents are trained and cooperated in the same reinforcement learning environment. Due to different optimization goals, there is a difficulty in credit assignment of the reward function. A representative method to alleviate this problem is the counterfactual baseline (COMA) technique, which solves the credit assignment problem by quantifying the contribution of agents.

[0007] Although topology network generation, multi-agent observation sharing, and multi-type agent credit assignment techniques have mature applications, the traditional solutions do not consider the constraints of the road network on mobile base stations in emergency scenarios, the constraints of the physical environment on communication quality, and the on-the-spot adjustment and cooperation of the network. The separation of rewards during training is not thorough enough.

[0008] Therefore, the present patent provides a large-scale emergency wireless communication network intelligent agile networking method and device, which constrains the exploration space of agents, solves the problem of severe dependence on simulation evaluation after topology generation, shares observations within connected branches, reduces convergence time, improves the noise problem of existing agents caused by shared observations, and solves the problem of mutual interference of rewards during cooperative training of different types of agents. SUMMARY

[0009] In view of this, the present application provides a large-scale emergency wireless communication network intelligent agile networking method and device, which provides a wireless ad hoc network topology generation and adjustment solution for large-scale emergency scenarios, solves the problem of simulation feasible deployment difficulty caused by insufficient coupling with the environment of the existing topology generation algorithm, improves the slow convergence time problem caused by the large node search area of the existing topology generation algorithm, solves the training instability and mutual dependence problem caused by local agent sharing observation of the existing agent coordination scheme, and solves the credit allocation problem caused by uneven reward function superposition in the existing multi-type agent coordination training process.

[0010] The present application provides a kind of Orthogonal Reward for Topology Aware and Road Optimized MADDPG (OTR-MADDPG) based on topology awareness and road network optimization. The method comprises the following steps:

[0011] S1, for the OTR-MADDPG, design self-defined agent learning environment Env based on the initial position distribution of the emergency wireless communication network node, channel environment, feasible region, target coverage area, LEO satellite orbit;

[0012] S2, in particular, for the MOV_BS agent, design improved action space based on road network search, in particular, design the update mode of improved policy network π based on the state input of neighbor node within connected branch;

[0013] S3, in particular, for the LEO agent, design the action space based on the access / rejection of LEO satellite, in particular, design the update mode of policy network Based on MOV_BS coordinate-overhead satellite state pair;

[0014] S4, design complex reward function , R1 is the reward of the MOV_BS agent, and the communication quality evaluation result between any two nodes of MOV_BS agent is positively correlated, R2 is the reward of the LEO agent, and the data amount processed per unit time is positively correlated, i is imaginary unit, And Reward factor of

[0015] S5, integrating the agent learning environment Env described in S1, the action generation method of the MOV_BS agent described in S2, the strategy network update method, the action generation method of the LEO agent described in S3, the strategy network update method, the compound reward function described in S4 into the existing framework of MADDPG.

[0016] 2. In some embodiments of the application, in S1, the agent learning environment Env, characterized in that the state information includes the agent state vector [coordinate, antenna height, maximum transmit power, transmit gain, receive gain, receive sensitivity, maximum speed, frequency band range, power, oil, neighbor set], task site state vector [coordinate, scale, neighbor set], agent action vector [MOV_BS agent coordinate, LEO agent task access].

[0017] 3. In some embodiments of the application, in S2, the MOV_BS agent action space design method, characterized in that,

[0018] The action space takes into account the characteristics of the agent on board, and is constrained to the highway network, greatly reducing the agent exploration time;

[0019] Initialize the number of MOV_BS agents to N, input a step size △d for the MOV_BS agent, and calculate the same step length node set based on the current coordinate of the agent and the highway network The agent selects a node from the set As the next action, let j represent the step number and i represent the execution round, then the joint action of all MOV_BS agents can be expressed as an N-dimensional vector: ,……, After episode_num steps, the action of all agents with respect to the step number can be expressed as: Then the single agent trajectory can be represented by ;

[0020] Introducing forbidden areas on the highway network simulates damaged road sections, and combining the multi-agent exploratory networking method makes the agent avoid the forbidden area, and the networking time is closer to the real networking time;

[0021] The introduction of the highway network and the forbidden area on the highway network increases the coupling of the agent with the real environment, which is reflected in that the topology construction process is clearly executable along the highway, and the topology construction time is part of the reward. After the topology is generated, there is no need for further manual assessment of feasibility and practicality.

[0022] 4. In some embodiments of the application, in S3, the action space of the LEO agent, characterized by,

[0023] The real-time coordinate distribution V of the MOV_BS agent obtained in S2 is input as the task coordinate of the LEO agent, and the LEO agent has the choice of accessing or rejecting the task generated by the MOV_BS node, i.e., a two-dimensional action space [1, 0], [0, 1], where 1 represents access and 0 represents rejection. The access time is obtained through the STK software interface.

[0024] 5. In some embodiments of the application, in S2, the policy network π, characterized by,

[0025] The action obtained by executing the improved action space searched by the road network obtains the real-time coordinate distribution V of the agent. Since the communication topology is constructed, the set of nodes with communicable links is defined as a connected branch. Crucially, the policy network π uses the joint observation formed by the observations of the nodes in a single connected branch when updating the parameters, rather than the local observation of a single agent, which greatly enhances the synergy of the communication agents. Each agent has a value network Q and a policy network π during training. In particular, the π network of the agent being updated inputs the observations of the nodes in the connected branch in which the agent is located when updating, and the value network Q still inputs the global observation. This is because the OTR-MADDPG of claim 1 itself outputs a communication network, and the agents in the connected branch can share observations and do not violate the CTDE principle. Through the dependency relationship between nodes, the synergy between agents is enhanced. At the same time, in order to prevent the policy network π from being too large in size and affecting the training time and being too dependent on other agents, the π network only inputs the observations of itself and its neighbor nodes when updating.

[0026] 6. In some embodiments of the application, in S4, the communication quality evaluation between any two nodes of the MOV_BS agent, characterized by,

[0027] The signal attenuation degree is calculated through the channel environment, and the link budget is calculated combining with the self-transmitting and receiving performance of the agent. The multipath, rain attenuation and diffraction caused by buildings are mainly considered. According to the ITU-R P530 recommendation, the meteorological environment of the affected area is sliced, and the meteorological attenuation propagation loss analysis and multipath fading between any two nodes are performed. According to the ITU-R P526 recommendation, the geographical environment between any two nodes is profiled according to the first Fresnel ellipsoid, and the electromagnetic wave diffraction propagation loss is analyzed. The free space loss , diffraction of large buildings , atmospheric absorption , thermal noise forest penetration loss obtain the loss , through the link budget formula:

[0028]

[0029]

[0030] obtain the communication quality evaluation result between two nodes, i.e. the received power , if , the link exists, otherwise it does not exist.

[0031] 7. In some embodiments of the application, in the S4, the complex reward function is designed, characterized in that,

[0032] According to the mobile base station structure and the wireless signal transceiver mechanism, the key attributes of the MOV_BS agent are determined, including the moving speed v, the antenna type Type, the antenna support frequency band f, the antenna height h, the antenna transmit power , the antenna transmit gain , the antenna receive gain , the receive sensitivity , the Env obtained by S1 and the agent attributes jointly constrain the link quality and the connectivity between any two nodes in the MOV_BS agent, and the connectivity of the entire network is extended to constrain the connectivity of the network topology;

[0033] According to the channel characteristics and on-board processing capability of the LEO node's satellite-ground transmission, the key attributes of the LEO agent reward are determined, including the LEO over-the-top time window[ , ], the remaining memory R_Remain, the task priority pri of the MOV_BS node and the UAV node, the resource consumption stor_cos, and the task time window[ , ];

[0034] The reward factors include:

[0035] a) Given the connectivity K, the number of neighbor nodes of a single agent approaches K positively or negatively, and a positive reward is given;

[0036] b) The number of connected branches branchs of the network decreases, and a positive reward is given;

[0037] c) The exploration area area of the network becomes larger, and a positive reward is given;

[0038] d) The number of task locations c accessed by the network increases, and a positive reward is given;

[0039] e) Given the communication quality qlt, the communication quality between neighbor nodes is positively or negatively close to qlt, and a positive reward is given;

[0040] f) A current maximum coverage size and optimal communication quality are achieved, and a positive reward is given when the value of |V| is smaller;

[0041] g) A current maximum coverage size and optimal communication quality are achieved, and a positive reward is given when the value of T or the total displacement X of the agent is smaller;

[0042] h) Single-satellite task access task priority pri;

[0043] i) Total task processing quantity num_processed;

[0044] j) Total consumption of on-board memory resources ;

[0045] Since the MOV_BS agent and the LEO agent are trained and cooperated in the same environment, if the reward function still uses real number operation, the reward of the MOV_BS agent will disturb the policy network update of the LEO agent, and vice versa, which leads to unstable training. The use of complex orthogonal operation can avoid this problem, and the reward function of the MOV_BS agent is , and the reward function of the LEO agent is .

[0046] On the other hand, the application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method according to any one of the above-mentioned methods.

[0047] The application has at least the following beneficial effects:

[0048] The application provides a large-scale emergency wireless communication network intelligent agile networking method and device. In view of the network failure problem caused by the damage of communication facilities in disaster scenes such as earthquakes and floods, the OTR-MADDPG algorithm is used to realize the cooperative networking of ground mobile base stations and the maximum task access of satellite nodes. The orthogonal reward mechanism is innovatively designed to separate the reward disturbance of the two types of agents, the action space is constrained by the highway network isochronous line to reduce the exploration dimension, and the observation shared policy network is constructed to enhance the cooperation ability. The technical features include: establishing a self-defined learning environment integrating geographical environment and electromagnetic propagation model; designing a mobile base station action space and satellite access decision mechanism based on road network optimization; constructing a complex reward function to handle the rewards of ground network agents and satellite network agents. The application solves or improves the problems of too large multi-agent exploration space, poor cooperation, and uneven credit distribution of multi-type agent reward functions in emergency scenarios through deep coupling of topology perception and resource optimization.

[0049] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings. The objects and other advantages of the present application can be realized and attained by means of the instrumentalities particularly pointed out in the specification and drawings.

[0050] Those skilled in the art will appreciate that the objects and advantages of the application can be realized and attained by means of the instrumentalities and combinations particularly pointed out in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application. In the drawings:

[0052] Figure 1 A schematic diagram of an emergency communication network networking algorithm based on a fusion geographic environment and an electromagnetic propagation model in an embodiment of the present application.

[0053] Figure 2 A schematic diagram of an orthogonal reward multi-agent deep deterministic policy gradient reinforcement learning (OTR-MADDPG) training process based on topology awareness and road network optimization in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the embodiments and drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application, but are not intended to limit the present application.

[0055] It should also be noted that, in order to avoid the present application being obscured by unnecessary details, only structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0056] It should be emphasized that the terms “comprises / comprising” when used in this specification, refer to presence of the stated features, elements, steps or components but do not preclude one or more additional features, elements, steps, or components.

[0057] It should also be noted that, unless otherwise specified, the term “connected” used herein can not only mean direct connection, but also indirect connection in the presence of an intermediate.

[0058] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, like reference numerals denote like or similar components, or like or similar steps.

[0059] To solve the problem of insufficient coverage of network facilities or destruction of basic network facilities and large-scale communication failure in emergency communication scenarios such as earthquakes, forest fires, and flood disasters, wireless networking methods mainly based on mobile base stations, unmanned aerial vehicles, and satellites become more flexible and mobile. The present application provides an intelligent and agile networking algorithm for large-scale emergency wireless communication networks, aiming to solve the planning and construction of invulnerability topology in emergency scenarios, improve the networking speed of the network and the node coordination under scenario mutation, make full use of the constraints of the road network and the decentralized real-time adjustment ability of the Agent in multi-agent deep reinforcement learning, and output a practical networking scheme that adapts to dynamic changes in the environment.

[0060] S1, for the OTR-MADDPG, a combined environment based on observations of MOV_BS nodes, UAV nodes, and LEO nodes is designed. A custom agent learning environment Env is constructed based on the initial position distribution of communication nodes, channel environment, feasible region, target coverage area, and LEO satellite orbit; the initial position distribution refers to the distribution of various emergency wireless base stations, which is generally defined at various fire units, the channel environment considers factors that exist in real emergency scenarios and have a greater impact on 30M-30GHz radio propagation, the dynamic environment includes terrain environment, weather environment, and vegetation coverage, among which the weather model considers multipath, and the remaining free space loss and thermal noise are given according to empirical values, the feasible region refers to the road network and its extended area, i.e., the places where the vehicle-mounted base station can reach, and the target coverage area refers to the coordinates of the disaster area;

[0061] S2, for the MOV_BS agent, an improved action space based on road network search and an improved policy network update method based on the state input of neighbor nodes within the connected branch are designed. The agent real-time coordinate distribution V obtained by the action of the improved action space based on road network search is executed; since it is the construction of communication topology, the set of nodes with communicable links is defined as a connected branch, and the key is that the policy network π uses the joint observation formed by the observations of the neighbor nodes of the agent within a single connected branch for parameter updating, rather than the local observation of a single agent, which greatly improves the coordination of communication agents;

[0062] S3, design the action space of access / rejection based on LEO satellites for the LEO agent, and the policy network updating method based on the STK over-the-top time window. The real-time coordinate distribution V of the MOV_BS agent obtained in S2 is input as the task coordinate of the LEO agent, i.e. LEO, which has the choice of access or rejection to the tasks generated on the ground, i.e. has a two-dimensional action space [1, 0], [0, 1], where 1 represents access and 0 represents rejection. The over-the-top time access is obtained through the STK software interface uiApplication.Personality2.GetObjectFromPath(f"Satellite / {satellite_name}").GetAccessToObject(place).

[0063] S4, design the compound reward function , for the reward of the MOV_BS agent, for the reward of the LEO agent, i is the imaginary unit. and The reward factors of and are obtained through the interaction of the agent with the Env obtained in S1.

[0064] S5, integrate the agent learning environment Env in S1, the action generation method and policy network updating method of the MOV_BS agent in S2, the action generation method and policy network updating method of the LEO agent in S3, and the compound reward function in S4 into the existing framework of MADDPG, and the core training process of OTR-MADDPG corresponds to Figure 2 , the pseudo code is as follows, and the underlined part is the design point:

[0065]

[0066] In step S1, the agent learning environment Env, the state information includes the following contents:

[0067] Agent state vector [coordinate, antenna height, maximum transmit power, transmit gain, receive gain, receive sensitivity, maximum speed, frequency band range, power, oil, neighbor set, processor remaining resources];

[0068] Task site state vector [coordinate, scale, neighbor set];

[0069] Agent action vector [MOV_BS agent coordinate, LEO agent task access or not];

[0070] The mobile base station structure and the wireless signal transceiver mechanism determine the key attributes of the MOV_BS agent, including the moving speed v, the antenna type Type, the antenna supported frequency band f, the antenna height h, the antenna transmit power , the antenna transmit gain , the antenna receive gain , and the receive sensitivity . The Env obtained by S1 and the agent attributes jointly constrain the link quality and the connectivity between the two agents, and the connectivity of the entire network is extended to constrain the connectivity of the network topology;

[0071] The channel characteristics of the LEO node's satellite-ground transmission and the on-board processing capability determine the key attributes of the LEO agent reward, including the LEO over-the-top time window , , the memory remaining R_Remain, the task priority pri of the MOV_BS node and the UAV node, the resource consumption stor_cos, and the task time window , .

[0072] In step S2, the action space generation method based on the improved road network search includes the following steps:

[0073] The action space generation stage, the mobile base station as the MOV_BS agent, assuming that the number of MOV_BS agents is N, for the MOV_BS agent, input a step size △d, based on the current coordinates of the agent and the road network, a set of nodes with the same step size is calculated .

[0074] The single-step action discretization expression stage, taking the current coordinates as the starting point and the step size △d as the radius, the step size is divided into actions two-dimensional vectors, since only one action can be selected, a actions-dimensional one-hot vector [0, 0, 1, …, 0] can be used to represent the current action action, and the coordinates corresponding to the action are .

[0075] The single-step action generation stage, from find the node closest to p , as the action to be executed next.

[0076] The multi-agent trajectory generation stage, where j represents the step number and i represents the execution round, then the joint action of all MOV_BS agents can be expressed as an N-dimensional vector: ,……, ; until the networking is completed, a total of episode_num steps, the actions of all agents with respect to the step number can be expressed as: , the travel trajectory of a single agent can be represented as .

[0077] The no-entry zone mechanism introduces a no-entry zone in the highway network, simulates damaged road sections, and combines the exploratory networking mode of multiple agents to make the agents avoid the no-entry zone, so that the networking time is closer to the real networking time.

[0078] The action space based on the improved road network search increases the coupling of the agent and the real environment, which is embodied in that the topology is clear and can be executed along the road, and after the topology is generated, the feasibility does not need to be judged manually.

[0079] In step S2, the improved connected component topology perception technology comprises the following steps:

[0080] The connected component stage is judged, the union set of the neighbor set of the agent is taken, and if , then share all observations in .

[0081] The observation input stage, each agent has a value network Q and a policy network π during training, and in particular, the π network of the updated agent inputs all the observations of all the nodes in the connected component where the agent is located during updating, and the value network Q still inputs the global observation.

[0082] This is because the OTR-MADDPG itself outputs a communication network, and the agents in the connected component can share observations and do not violate the CTDE principle, and the dependence between nodes enhances the cooperation between agents. At the same time, in order to prevent the scale of the policy network π from being too large to affect the training time and over-relying on other agents, the π network only inputs the observations of itself and the neighbor nodes during updating.

[0083] In step S3, the low-orbit satellite access strategy based on multi-agent deep reinforcement learning comprises the following steps:

[0084] Multi-source data fusion modeling, generating three-dimensional spatiotemporal features of task targets based on a satellite orbit dynamics model, including randomly distributed geographical positions, task duration, and timeliness constraints. Through a satellite-ground visibility analysis model, calculate the dynamic observable time window of the task target and the satellite constellation, and construct the satellite-ground association topology relationship. Align and standardize the heterogeneous spatiotemporal data, and extract the feature matrix containing the task trigger time, satellite transit period, observation elevation angle, and other core parameters.

[0085] The cooperative decision-making model trains and designs a distributed policy network architecture, each agent corresponds to a specific satellite node, and achieves inter-satellite cooperative perception through shared observation features. A priority experience replay mechanism is used to dynamically adjust the training sample weights of key decision events such as task allocation and resource preemption.

[0086] A competitive reward function is introduced to optimize the multi-objective balance of satellite payload utilization (maximize task completion) and energy efficiency (minimize attitude maneuver energy consumption).

[0087] On-board strategy verification and optimization, deployment of a lightweight strategy inference module, real-time verification in a satellite on-orbit simulation environment, and detection of the impact of calculation delay on decision-making.

[0088] A rolling horizon optimization mechanism is established to dynamically update the parameter weights of the policy network according to the actual inter-satellite link state.

[0089] Evaluate the robustness of the strategy through Monte Carlo perturbation testing, and compensate for the failure mode of the strategy under extreme conditions (such as single satellite failure and sudden task surge).

[0090] In step S4, the orthogonal reward credit allocation method comprises the following steps:

[0091] Determine the reward factor stage, give the connectivity K, the number of neighbor nodes of a single agent approaches K positively or negatively, give positive reward; the number of connected branches of the network branchs decreases, gives positive reward; the exploration area of the network area becomes larger, gives positive reward; the number of task access points of the network c increases, gives positive reward; given the communication quality qlt, the communication quality between neighbor nodes approaches qlt positively or negatively, gives positive reward; achieve the current maximum coverage scale and optimal communication quality, and the value of |V| is smaller, give positive reward; achieve the current maximum coverage scale and optimal communication quality, and the use time T or the total displacement X of the agent is smaller, give positive reward; the task priority pri of a single satellite task access; the total number of task processing num_processed; the total consumption of on-board memory resources ;

[0092] Reward function design stage, since there are MOV_BS agents and LEO agents training and cooperating in the same environment, if the reward function still uses real number operation, it will cause the reward of different types of agents to disturb the strategy network update of other agents, leading to unstable training, using complex orthogonal operation can avoid this problem, the reward function of the MOV_BS agent , the reward function of the LEO agent . Complex reward function .

[0093] In step S4, the calculation of the reward factor of network connectivity is obtained by link budget between any two points in combination with factors such as terrain, weather, vegetation coverage, etc., including the following steps:

[0094] Considering weather attenuation and multipath fading, the weather environment of the disaster area is sliced according to ITU-R P530 recommendations, and the weather attenuation propagation loss analysis and multipath fading between any two nodes are carried out.

[0095] The diffraction loss caused by large buildings or obstacles is analyzed according to ITU-R P526 recommendations, the geographical environment between any two nodes is profiled according to the first Fresnel ellipsoid, and the electromagnetic wave diffraction propagation loss is analyzed.

[0096] Through free space loss , large building diffraction , atmospheric absorption , basic thermal noise , dense forest penetration loss Get loss , combined with the link budget formula:

[0097]

[0098]

[0099] Get the communication quality evaluation result between two nodes, that is, the received power If , the link exists, otherwise it does not exist.

[0100] The application also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the emergency wireless communication network intelligent agile networking based on OTR-MADDPG.

[0101] Corresponding to the above method, the application also provides a device, which includes a computer device, the computer device including a processor and a memory, the memory having computer instructions stored therein, and the processor being configured to execute the computer instructions stored in the memory, and the device implements the steps of the method as described above when the computer instructions are executed by the processor.

[0102] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the aforementioned edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0103] In summary, the present application provides a large-scale emergency wireless communication network intelligent agile networking method and device. The method is designed based on improved highway network exploration action space design, a strategy network updating method based on shared observation of agents within a communication network connected branch, and a multi-type agent reward accumulation strategy based on orthogonal rewards, to meet the demand for agile networking when the network is inherently insufficiently covered or the basic network infrastructure is destroyed and large-scale communication is disabled in an emergency scenario. The method provides an effective solution to large-scale wireless network topology coordination problems under environmental mutations. Considering that the observation of a single agent cannot effectively learn the environmental characteristics in a complex communication scenario, leading to slow training, a strategy network updating technology based on shared observation of agents within a communication network connected branch is proposed. In the face of existing methods for solving the credit allocation of reward functions for simultaneous training of multiple types of agents, which fail to fully separate the mutual disturbance of rewards of different agents, an orthogonal reward method is introduced to fully separate the rewards of different agents.

[0104] Those of ordinary skill in the art should understand that the example components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. The exact implementation depends on the specific application and design constraints imposed on the overall system. Those of skill can use various approaches to implement the described functionality depending on the specific application, but such implementation should not be considered to be beyond the scope of the present application. When implemented in hardware, the hardware can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are the program or code segments to perform a specific task. The program or code segments can be stored in a machine readable medium, or transmitted by a data signal carried in a carrier wave over a transmission medium or communication link.

[0105] It is to be expressly understood that the invention is not limited to the specific configurations and process described above and illustrated in the accompanying drawings. For the sake of clarity, detailed descriptions of known methods are omitted. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and various changes, modifications and additions can be made thereto by one of ordinary skill in the art without departing from the spirit of the present invention, and the order of the steps can be changed.

[0106] In the present invention, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0107] The above description is merely illustrative of the application, and is not intended to limit the scope of the application. Various modifications and changes can be made by one of ordinary skill in the art without departing from the spirit and scope of the application. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the application should be included in the scope of the application.

Claims

1. A large-scale emergency wireless communication network intelligent agile networking method, characterized in that, The method comprises the following steps: According to the communication guarantee and post-disaster communication reconstruction scene of earthquake, forest fire, large-scale flood disaster, the large-scale emergency wireless communication network node is divided into movable base station node (MOV_BS), unmanned aerial vehicle (UAV) air node and low earth orbit satellite node (LEO), and the network is a self-organizing task coordination network, the network topology is generated according to task coordinates, the network topology is followed according to task position migration, and the link resource is adjusted according to task real-time distribution, wherein the MOV_BS is a ground intelligent agent, the UAV node is an extension of the MOV_BS antenna, and the LEO is a satellite intelligent agent and an auxiliary of the ground network; an orthogonal reward multi-agent deep deterministic policy gradient reinforcement learning (OTR-MADDPG) based on topology awareness and road optimization is designed; S1, for the OTR-MADDPG, a self-defined intelligent agent learning environment Env based on the initial position distribution of the emergency wireless communication network node, a channel environment, a feasible region, a target coverage area and a LEO satellite orbit is designed; the environment state information includes an intelligent agent state vector [coordinates, antenna height, maximum transmission power, transmission gain, reception gain, reception sensitivity, maximum speed, frequency band range, power, oil, neighbor set], a task site state vector [coordinates, scale, neighbor set] and an intelligent agent action vector [MOV_BS intelligent agent coordinates, LEO intelligent agent task access]; S2, design an improved action space based on road network search for the MOV_BS agent, and perform the improved action space based on road network search to obtain an action of the agent, so that the real-time coordinate distribution of the agent is obtained , design an update method of the improved strategy network π based on the state input of the neighbor nodes in the connected branch; S3, design the action space of access / rejection based on LEO satellites for the LEO agent, and design the policy network based on MOV_BS coordinate-over-the-top satellite state pairs updating mode S4, design a reward function , reward for the MOV_BS agent, the communication quality evaluation result between any two nodes of the MOV_BS agent is positively related, reward for the LEO agent, positively related to the amount of data processed per unit time, i is an imaginary unit, and the reward factor is obtained through the interaction of the agent with S1 and the Env. S5, the intelligent agent learning environment Env in S1, the action generation method and the policy network updating method of the MOV_BS intelligent agent in S2, the action generation method and the policy network updating method of the LEO intelligent agent in S3, and the compound reward function in S4 are integrated into the existing framework of MADDPG.

2. The intelligent agile networking method of large-scale emergency wireless communication network according to claim 1, the design method of the action space of the MOV_BS agent S2, characterized in that, The action space considers the characteristics of the intelligent agent vehicle and is constrained to the road network; Initialize the number of MOV_BS agents as N, for each MOV_BS agent, input a step size △d, based on the current coordinates of the agent and the road network, calculate the same step length node set , the agent selects a node from it as the action for the next step, j denotes the number of steps, i denotes the execution round, then the joint action of all MOV_BS agents can be expressed as an N-dimensional vector: ; To the networking is completed, total episode_num step, all agent's action about step number can be expressed as: Then the single agent's travel trajectory can be expressed as . 3.The intelligent agile networking method of large-scale emergency wireless communication network of claim 1, wherein the action space of the LEO agent of S3, characterized in that, The MOV_BS agent real-time coordinate distribution obtained from S2 As the task coordinate input of LEO agent, LEO has the choice of access or rejection to the task generated by MOV_BS node, that is, it has a two-dimensional action space [1,0], [0,1], where 1 represents access, and 0 represents rejection. The over-the-top time access is obtained through the STK software interface.

4. The large-scale emergency wireless communication network intelligent agile networking method according to claim 1, wherein the policy network π in S2 is characterized by, The node set with a communicable link is defined as a connected branch, and the joint observation formed by the observation of the nodes in a single connected branch is used instead of the local observation of a single intelligent agent during the parameter updating of the policy network π; each intelligent agent has a value network Q and a policy network π during training, and the π network of the updated intelligent agent inputs the observation of the nodes in the connected branch where the intelligent agent is located during updating, and the value network Q still inputs the global observation.

5. The intelligent agile networking method of large-scale emergency wireless communication network according to claim 1, wherein the communication quality evaluation between any two nodes of the MOV_BS intelligent agent of S4 is characterized in that, The signal attenuation degree is calculated through the channel environment, the link budget is combined with the self-transmission and reception performance of the agent, the diffraction loss caused by multipath fading, rain attenuation, buildings or obstacles is mainly considered, the meteorological environment of the disaster area is sliced, the meteorological attenuation propagation loss analysis and multipath fading between any two nodes are carried out; the geographical environment between any two nodes is profiled according to the first Fresnel ellipsoid, and the electromagnetic wave diffraction propagation loss is analyzed; the free space loss , diffraction of large buildings , atmospheric absorption , basic thermal noise , dense forest penetration loss get loss , define antenna transmit power , antenna transmit gain , antenna receive gain , receive sensitivity , through the link budget formula: The communication quality between two nodes is obtained, i.e. the received power If then the link exists, otherwise it does not.

6. The intelligent agile networking method of large-scale emergency wireless communication network according to claim 1, the reward function design of S4 is characterized in that, The key attributes of the MOV_BS agent are determined according to the mobile base station structure and the wireless signal transceiver mechanism, including the moving speed v , the antenna type Type , the antenna supported frequency band f , the antenna height h , and the receiving sensitivity . The Env obtained by S1 and the agent attributes jointly constrain the link quality and the connectivity between any two nodes in the MOV_BS agent, and the connectivity of the entire network is extended to constrain the connectivity of the network topology. The key attributes that constrain the LEO agent's reward are determined according to the channel characteristics of LEO node's satellite-ground transmission and the on-board processing capability, including LEO over-the-top time window , memory remaining R_Remain , task priority of MOV_BS node and UAV node pri , resource consumption stor_cos、 task time window ; The reward factor comprises: a) for a given connectivity K, a single intelligent agent neighbor node number approaches K positively or negatively, and a positive reward is given; b) number of connected components of the network branchs reduce, give positive reinforcement; c) Exploration area of the network area grows, giving positive reinforcement; d) the number of task sites accessed by the network c increased, positive reinforcement is given; e) given communication quality qlt , the communication quality between neighbor nodes is approaching positive or negative qlt , give positive reward; f) achieving the current maximum coverage size and optimal communication quality, and A positive reward is given when the value is smaller. g) Achieving current maximum coverage size and optimal communication quality, and time of use T or total displacement of the agent X Smaller, give positive reward; h) Single-satellite mission access mission priority pri ; i) total number of task processes num_processed ; j) total on-board memory resource consumption ; The reward function of the MOV_BS intelligent agent , the reward function of the LEO agent .

Images