Intelligent agile networking method and device for large emergency wireless communication network

Abstract

The invention provides an intelligent agile networking method and device for a large emergency wireless communication network. The method comprises the following steps: designing an agent learning environment Env containing wireless communication network node initial position distribution, environment constraint and target constraint based on orthogonal reward multi-agent depth deterministic policy gradient reinforcement learning (OTR-MADDPG) of topology perception and road network optimization; designing an improved action space based on road network search for the MOVBS intelligent agent, and designing an updating mode of an improved strategy network pi based on state input of neighbor nodes in a connected branch; designing an access / rejection action space based on an LEO satellite for the LEO intelligent agent, and designing an updating mode of a strategy network # imgabs0 # based on an MOVBS coordinate-overhead satellite state pair; a reward function # imgabs1 # is designed; a learning environment Env, an action generation mode of an intelligent agent, a strategy network updating mode and a reward function R are integrated into an existing framework of MADDPG. According to the invention, the problems of long topology construction time and poor node collaboration of the emergency communication network are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of wireless ad hoc networks, and in particular to a method and device for intelligent and agile networking of large-scale emergency wireless communication networks. The method improves the convergence time and practicality of the training process by optimizing the road network, improves the collaborative ability of intelligent agents by topology perception based on connected branches, and solves the problem of uneven credit distribution of training rewards for multiple intelligent agents in the same environment by using orthogonal rewards. Background Art

[0002] Multi-agent deep reinforcement learning and wireless ad hoc networking (MANET) technology play a vital role in communication support and post-disaster communication reconstruction for earthquakes, forest fires, and large-scale floods. The network nodes involved in emergency wireless communications are divided into mobile base stations (MOV_BS), unmanned aerial vehicles (UAV) aerial nodes, and low-orbit satellite (LEO) nodes. This network is a self-organizing task-coordination network that generates network topology based on task coordinates, follows tasks as they migrate, and adjusts link resources based on real-time task distribution. The MOV_BS is the first type of agent, while the UAV node serves as an extension of the MOV_BS antenna. The LEO is the second type of agent, assisting the ground network. The coupling of these two types of agents creates a vast exploration space. By analyzing the situation in the affected area and modeling the wireless channel environment, combined with MANET technology, coverage of the affected area and support routes can be achieved, as well as interconnection with rear-area command posts. This enhances the automated networking capabilities for emergency missions.

[0003] However, in emergency communication scenarios characterized by large scale, high uncertainty, and a multitude of node types, wireless ad hoc networks face numerous challenges and issues. Mobile communication relays must be easily accessible and deployable, and the smoothness of actual wireless communication links is constrained not only by the performance parameters of the transceiver equipment itself but also by the channel environment, including terrain, buildings, weather, and electromagnetic interference. Logical networking based on a simple communication radius may be feasible for simulation, but deployment is difficult or even impossible, and the simulation results are unreliable. Furthermore, networking in highly uncertain environments requires nodes to possess a degree of autonomy, and the network, whether overall or in parts, must possess agile adaptation and link recovery capabilities to cope with increasing obstacles, expanding target coverage areas, and node failures. Third, low-orbit satellite constellations, working in conjunction with terrestrial networks, must handle information backhaul from multiple disaster-stricken areas, especially those difficult for terrestrial networks to cover. The distribution and frequency of data packets from terrestrial mobile base stations are directly related to the topology of the terrestrial emergency network. Consequently, the two networks operate in a tightly coupled yet completely independent system, resulting in limited flexibility in the face of complex and ever-changing emergency scenarios.

[0004] Representative topology generation algorithms include the hierarchical and phased topology generation algorithm, the disjoint-path topology generation algorithm, and the homogenized node degree-based topology generation algorithm. The hierarchical and phased topology generation algorithm plans nodes layer by layer, uses an evaluator to determine network feasibility, and centrally manages global nodes. The disjoint-path topology generation algorithm solves the backbone network topology by finding a maximal connected dominating set. The homogenized node degree-based topology generation algorithm adjusts the connectivity of the existing topology to achieve a relatively uniform node degree distribution in the backbone network, improving its survivability.

[0005] MADDPG is a multi-agent deep reinforcement learning framework for deterministic policy generation. The networking process is a transition from isolation to connectivity, making it more suitable for the centralized training and distributed execution architecture within this framework. When training multiple agents, shared observations are a common method for improving convergence speed and collaboration. A representative approach is the agent clustering method based on Growing Neural Gas (GNG). This clustering method clusters the observation correlations of the agents to form a group. The observations of this group form joint observations, and joint actions are generated based on these joint observations. This results in a certain performance loss, but significantly improves agent collaboration and convergence speed.

[0006] When two or more types of agents are trained and collaborated in the same reinforcement learning environment, there are difficulties in assigning credit to the reward function due to different optimization objectives. A representative mitigation method is the Counterfactual Baseline (COMA) technique, which solves the credit assignment problem by quantifying the contributions of the agents.

[0007] Although topological network generation, multi-agent observation sharing, and multi-type agent credit allocation technologies have mature applications, traditional solutions do not consider the constraints of the road network on mobile base stations in emergency scenarios, the constraints of the physical environment on communication quality, the network's on-the-spot adjustment and coordination, and the separation of rewards during training is not thorough enough.

[0008] Therefore, this patent provides a method and device for intelligent and agile networking of large-scale emergency wireless communication networks, which constrains the intelligent agent's exploration space, solves the problem of heavy reliance on simulation evaluation after previous topology generation, shares observations within connected branches, reduces convergence time, improves the noise problem caused by shared observations of existing intelligent agents, and solves the problem of mutual interference of rewards during collaborative training of different types of intelligent agents. Summary of the Invention

[0009] In view of this, the present invention provides an intelligent and agile networking method and device for a large-scale emergency wireless communication network, providing a solution for wireless self-organizing network topology generation and adjustment for large-scale emergency scenarios, solving the problem of difficulty in simulation feasible deployment caused by insufficient coupling with the environment in existing topology generation algorithms, improving the problem of slow convergence time caused by large node search areas in existing topology generation algorithms, solving the problems of training instability and mutual dependence caused by shared observations of local agents in existing intelligent agent collaboration schemes, and solving the credit distribution problem caused by uneven superposition of reward functions in the existing collaborative training process of multiple types of intelligent agents.

[0010] The present invention provides an orthogonal reward for topology-aware and road-optimized multi-agent deep deterministic policy gradient reinforcement learning (OTR-MADDPG). The method comprises the following steps: S1. For the OTR-MADDPG, design a custom agent learning environment Env based on the initial position distribution of the emergency wireless communication network nodes, the channel environment, the feasible domain, the target coverage area, and the LEO satellite orbit; S2. Specifically, an improved action space based on road network search is designed for the MOV_BS agent, and specifically, an improved update method of the policy network π based on the state input of neighbor nodes in the connected branch is designed; S3. Specifically, design an action space for the LEO agent based on the access / rejection of LEO satellites, specifically, design a policy network based on the MOV_BS coordinate-overhead satellite state pair. Update method; S4. Designing a complex reward function , is the reward of the MOV_BS agent, R1 is positively correlated with the communication quality evaluation result between any two nodes of the MOV_BS agent, is the reward of the LEO agent, R2 is positively correlated with the amount of data processed per unit time, i is an imaginary unit, and The reward factor is obtained by the interaction between the agent and the Env obtained by S1; S5. Integrate the agent learning environment Env described in S1, the action generation method and policy network update method of the MOV_BS agent described in S2, the action generation method and policy network update method of the LEO agent described in S3, and the complex reward function described in S4 into the existing framework of MADDPG.

[0011] 2. In some embodiments of the present invention, in said S1, said agent learning environment Env is characterized in that the state information includes the agent state vector [coordinates, antenna height, maximum transmission power, transmission gain, reception gain, reception sensitivity, maximum speed, frequency band range, power, fuel volume, neighbor set], the task location state vector [coordinates, scale, neighbor set], and the agent action vector [MOV_BS agent coordinates, whether the LEO agent task is connected].

[0012] 3. In some embodiments of the present invention, in S2, the MOV_BS agent action space design method is characterized by: The action space is constrained to the road network considering the characteristics of the agent on board, which greatly reduces the agent's exploration time; Initialize the number of MOV_BS agents to N, input a step size △d for the MOV_BS agent, and calculate the set of nodes with the same step size based on the current coordinates of the agent and the road network. , the agent selects a node from As the next action to be executed, j represents the step number and i represents the execution round number, then the joint action of all MOV_BS agents can be expressed as an N-dimensional vector: [ ,……, ]; until the network is completed, there are episode_num steps in total. The actions of all agents can be expressed in terms of the number of steps as follows: , then the trajectory of a single agent can be expressed as express; Introducing no-traffic zones in the highway network, simulating damaged road sections, and combining multi-agent exploratory networking methods to enable agents to avoid no-traffic zones, making networking time closer to real-world networking time. The introduction of road networks and no-go zones on them increases the coupling between the agent and the real environment. This is reflected in the fact that the topology construction process is explicitly executable along roads, and the topology construction time is included as part of the reward. After the topology is generated, no further human evaluation of feasibility and practicality is required.

[0013] 4. In some embodiments of the present invention, in S3, the action space of the LEO agent is characterized by: The real-time coordinate distribution V of the MOV_BS agent obtained by S2 is used as the task coordinate input of the LEO agent. LEO has the choice of accepting or rejecting the task generated by the MOV_BS node, that is, it has a two-dimensional action space [1,0], [0,1], where 1 represents acceptance and 0 represents rejection. The overhead time access is obtained through the STK software interface.

[0014] 5. In some embodiments of the present invention, in S2, the policy network π is characterized in that: The action obtained by executing the improved action space searched by the road network is used to obtain the real-time coordinate distribution V of the agent. Since it is a construction of a communication topology, the set of nodes with communicative links is defined as a connected branch. Crucially, when updating the parameters of the policy network π, the joint observation formed by the observation of the nodes in a single connected branch is used, rather than the local observation of a single agent, which produces a large gain for the collaboration of the communicating agents. Each agent has a value network Q and a policy network π during training. In particular, when updating the π network of the updated agent, the input is the observation of the nodes in the connected branch where the agent is located, and the value network Q still inputs the global observation. This is because the OTR-MADDPG described in claim 1 itself outputs a communication network. The agents in the connected branch are interoperable and can share observations, which does not violate the CTDE principle. The collaboration between agents is enhanced by the dependency between nodes. At the same time, in order to prevent the input scale of the policy network π from being too large, affecting the training time and over-reliance on other agents, the π network only inputs the observations of itself and neighboring nodes when updating.

[0015] 6. In some embodiments of the present invention, in said S4, the communication quality evaluation between any two nodes of the MOV_BS agent is characterized by: The signal attenuation degree is calculated by channel environment, and the link budget is performed in combination with the intelligent agent's own transceiver performance, focusing on multipath, rain attenuation and diffraction caused by buildings. According to ITU-R P530 recommendation, the meteorological environment of the disaster area is sliced, and meteorological attenuation propagation loss and multipath fading are analyzed between any two nodes; according to ITU-R P526 recommendation, the geographical environment between any two nodes is profiled according to the first Fresnel ellipsoid, and electromagnetic wave diffraction propagation loss is analyzed; through free space loss , diffraction from large buildings , atmospheric absorption , basic thermal noise , dense forest penetration loss Get loss , through the link budget formula:

[0016]

[0017] Get the communication quality evaluation result between the two nodes, that is, the received power ,like , then the link exists, otherwise it does not exist.

[0018] 7. In some embodiments of the present invention, in S4, the complex reward function design is characterized by: According to the mobile base station structure and wireless signal transmission and reception mechanism, the key attributes of the MOV_BS agent are determined, including mobile speed v, antenna type Type, antenna supported frequency band f, antenna height h, antenna transmission power , Antenna transmission gain , Antenna receiving gain , receiving sensitivity , the Env obtained by S1 and the agent attributes jointly constrain the link quality and connectivity between any two nodes in the MOV_BS agent, and extending to the entire network, it constrains the connectivity of the network topology; The key attributes constraining the LEO agent's reward are determined based on the channel characteristics of the LEO node's satellite-to-ground transmission and the on-board processing capabilities, including the LEO overhead time window [ , ], memory remaining R_Remain, task priority pri of MOV_BS node and UAV node, resource consumption stor_cos, task time window[ , ]; The reward factors include: a) Given a connectivity K, if the number of neighbor nodes of a single agent approaches K positively or negatively, a positive reward is given; b) The number of connected branches of the network decreases, and positive rewards are given; c) The network's exploration area becomes larger, giving positive rewards; d) The number of task locations c connected to the network increases, and positive rewards are given; e) Given a communication quality qlt, if the communication quality between neighboring nodes is positively or negatively close to qlt, a positive reward will be given; f) A positive reward will be given if the maximum coverage scale and optimal communication quality are achieved and the |V| value is smaller; g) Achieving the current maximum coverage scale and optimal communication quality, with a smaller usage time T or total agent displacement X, will be rewarded positively; h) Single satellite mission access task priority pri; i) Total number of tasks processed num_processed; j) Total memory resource consumption on board ; Since the MOV_BS agent and the LEO agent are trained and collaborated in the same environment, if the reward function still uses real number operations, the reward of the MOV_BS agent will cause disturbances to the update of the LEO agent's policy network, and vice versa, resulting in unstable training. The use of complex orthogonal operations can avoid this problem. The reward function of the MOV_BS agent is , the reward function of the LEO agent .

[0019] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the methods mentioned above when executed by a processor.

[0020] The beneficial effects of the present invention are at least: The present invention provides a method and device for intelligent and agile networking of a large-scale emergency wireless communication network. In response to the problem of network disability caused by damage to communication facilities in disaster scenarios such as earthquakes and floods, the OTR-MADDPG algorithm is proposed to realize the collaborative networking of ground mobile base stations and maximize the task access of satellite nodes. The reward disturbances of the two types of agents are separated by an innovative design of an orthogonal reward mechanism, the action space is constrained by the highway network isochrones to reduce the exploration dimension, and an observation sharing strategy network is constructed within the connected branches to enhance the collaborative ability. Technical features include: establishing a customized learning environment that integrates the geographical environment and electromagnetic propagation model; designing a mobile base station action space and a satellite access decision mechanism based on road network optimization; and constructing a complex reward function to simultaneously process the rewards of ground network agents and satellite network agents. The present invention solves or improves the problems of excessive exploration space of multiple agents, poor coordination, and uneven credit distribution of reward functions for multiple types of agents in emergency scenarios through the deep coupling of topology perception and resource optimization.

[0021] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0022] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings: Figure 1This is a schematic diagram of the design of an emergency communication network networking algorithm based on the integration of geographical environment and electromagnetic propagation model in one embodiment of the present invention.

[0024] Figure 2 Schematic diagram of the training process of orthogonal reward multi-agent deep deterministic policy gradient reinforcement learning (OTR-MADDPG) based on topology perception and road network optimization in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0026] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0027] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0028] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0030] To address the inherent lack of network coverage or destruction of basic network infrastructure, as well as widespread communication failures, in emergency communication scenarios such as earthquakes, forest fires, and floods, wireless networking approaches based on mobile base stations, drones, and satellites have been made more flexible and maneuverable. This paper provides an intelligent and agile networking algorithm for large-scale emergency wireless communication networks. This algorithm aims to address the planning and construction of destruction-resistant topologies in emergency scenarios, improve network construction speed, and enhance node coordination under sudden changes in scenarios. It leverages the constraints of the highway network and the decentralized, real-time adjustment capabilities of agents in multi-agent deep reinforcement learning to produce practical networking solutions that adapt to dynamically changing environments.

[0031] S1. For the OTR-MADDPG, a combined environment based on observations of MOV_BS nodes, UAV nodes, and LEO nodes is designed. A custom agent learning environment Env is constructed based on the initial position distribution of communication nodes, channel environment, feasible domain, target coverage area, and LEO satellite orbit; the initial position distribution refers to the distribution locations of various emergency wireless base stations, which are generally defined in various fire units. The channel environment takes into account the existence of real emergency scenarios and factors that have a greater impact on 30M-30GHz radio propagation. The dynamic environment includes terrain environment, meteorological environment, and vegetation coverage. Multipath is considered in the meteorological model, and the remaining free space loss and thermal noise are given according to empirical values. The feasible domain refers to the road network and its extended area, that is, the place that the vehicle-mounted base station can reach. The target coverage area refers to the coordinates of the disaster-stricken area. S2. Design an improved action space based on network search and an improved policy network update method for the MOV_BS agent based on neighbor node state input within a connected branch. Execute the actions obtained from the improved action space based on network search to obtain the agent's real-time coordinate distribution V. Because this is a communication topology, a connected branch is defined as a set of nodes with communicative links. Crucially, when updating the policy network π parameters, it uses joint observations formed by the agent's neighbor node observations within a single connected branch, rather than local observations of a single agent. This significantly improves the collaboration of the communicating agents. S3. Design an action space for the LEO agent based on the access / rejection of the LEO satellite and a strategic network update method based on the STK overpass time window. Use the real-time coordinate distribution V of the MOV_BS agent obtained in S2 as the task coordinate input of the LEO agent, i.e., LEO. LEO has the option of accessing or rejecting tasks generated on the ground, i.e., it has a two-dimensional action space [1,0], [0,1], where 1 represents access and 0 represents rejection. Overpass time access is obtained through the STK software interface uiApplication.Personality2.GetObjectFromPath(f"Satellite / {satellite_name}").GetAccessToObject(place).

[0032] S4. Designing a complex reward function , is the reward of the MOV_BS agent, is the reward of the LEO agent, and i is an imaginary unit. and The reward factor is obtained by the interaction between the agent and the Env obtained by S1.

[0033] S5. Integrate the agent learning environment Env described in S1, the action generation method and policy network update method of the MOV_BS agent described in S2, the action generation method and policy network update method of the LEO agent described in S3, and the complex reward function described in S4 into the existing framework of MADDPG. The core training process of OTR-MADDPG corresponds to Figure 2 , the pseudo code is as follows, the underlined part is the design points:

[0034] In step S1, the agent learning environment Env, the state information includes the following: Agent state vector [coordinates, antenna height, maximum transmit power, transmit gain, receive gain, receive sensitivity, maximum speed, frequency band, battery level, fuel level, neighbor set, remaining processor resources]; Task location state vector [coordinates, scale, neighbor set]; Agent action vector [MOV_BS agent coordinates, whether the LEO agent task is connected]; The structure of the mobile base station and the wireless signal transmission and reception mechanism determine the key attributes of the MOV_BS agent, including the mobile speed v, antenna type Type, antenna supported frequency band f, antenna height h, antenna transmission power , Antenna transmission gain , Antenna receiving gain , receiving sensitivity The Env obtained by S1 and the agent attributes jointly constrain the link quality and connectivity between the two agents, and extending to the entire network, it constrains the connectivity of the network topology; The channel characteristics of LEO node satellite-to-ground transmission and the on-board processing capabilities determine the key attributes constraining the LEO agent's reward, including the LEO overhead time window [ , ], memory remaining R_Remain, task priority pri of MOV_BS node and UAV node, resource consumption stor_cos, task time window[ , ].

[0035] In step S2, the method for generating an action space based on an improved road network search comprises the following steps: In the action space generation phase, the mobile base station is used as the MOV_BS agent. Assume that its number is N. For the MOV_BS agent, a step size △d is input. Based on the current coordinates of the agent and the road network, a set of nodes with the same step size is calculated. .

[0036] Single-step action discrete expression stage, based on the current coordinate As the starting point, the step size △d is the radius, and it is divided into two-dimensional vectors of actions. Since only one action can be selected, a one-hot vector [0,0,1,…,0] of actions dimension can be used to represent the current action action. The coordinates corresponding to action are .

[0037] Single-step action generation phase, from Find the node closest to p , , as the next action to be performed.

[0038] In the multi-agent trajectory generation phase, j represents the number of steps and i represents the number of execution rounds. The joint action of all MOV_BS agents can be expressed as an N-dimensional vector: [ ,……, ]; until the network is completed, there are episode_num steps in total. The actions of all agents can be expressed in terms of the number of steps as follows: , then the trajectory of a single agent can be expressed as express.

[0039] The no-drive zone mechanism introduces no-drive zones on the highway network, simulates damaged road sections, and combines multi-agent exploratory networking to enable agents to avoid no-drive zones, making networking time closer to real networking time; The action space based on the improved road network search increases the coupling between the intelligent agent and the real environment, which is reflected in the fact that the topology is clearly executable along the road. After the topology is generated, there is no need for manual feasibility judgment.

[0040] In step S2, the improved connected branch topology awareness technology includes the following steps: Determine the connected branch stage and take the union of the agent's neighbor sets. If ,but shared All observations within .

[0041] In the observation input stage, each agent has a value network Q and a policy network π during training. In particular, when the π network of the updated agent is updated, the input is the observation of all nodes in the connected branch where the agent is located, and the value network Q still inputs the global observation.

[0042] This is because OTR-MADDPG itself outputs a communication network. Agents within connected branches are interconnected and can share observations, thus maintaining the CTDE principle. Dependencies between nodes enhance inter-agent collaboration. Furthermore, to prevent excessive input size from affecting training time and over-dependence on other agents, the π network only inputs observations from itself and its neighbors during updates.

[0043] In step S3, the low-orbit satellite access strategy based on multi-agent deep reinforcement learning includes the following steps: Multi-source data fusion modeling and satellite orbit dynamics models are used to generate the three-dimensional spatiotemporal characteristics of mission objectives, including randomly distributed geographic locations, mission duration, and timeliness constraints. A satellite-ground visibility analysis model is used to calculate the dynamic observable time window between the mission objectives and the satellite constellation, and to construct a satellite-ground topological relationship. Feature alignment and standardization are performed on heterogeneous spatiotemporal data to extract a feature matrix containing key parameters such as mission trigger time, satellite transit period, and observation elevation angle.

[0044] Collaborative decision-making model training and distributed strategy network architecture design are implemented. Each agent corresponds to a specific satellite node, and inter-satellite collaborative perception is achieved by sharing observation features. A priority experience replay mechanism is used to dynamically adjust the training sample weights for key decision-making events such as task allocation and resource preemption.

[0045] A competitive reward function is introduced to simultaneously optimize the multi-objective balance between satellite payload utilization (maximizing the number of missions completed) and energy efficiency (minimizing the energy consumption of attitude maneuvers).

[0046] On-board strategy verification and optimization, deployment of lightweight strategy inference modules, real-time verification in the satellite on-orbit simulation environment, and detection of the impact of computational delays on decision-making.

[0047] A rolling time domain optimization mechanism is established to dynamically update the parameter weights of the strategy network according to the actual intersatellite link status.

[0048] The robustness of the strategy is evaluated through Monte Carlo perturbation testing, and compensation training is performed for strategy failure modes under extreme working conditions (such as single satellite failure and sudden mission surge).

[0049] In step S4, the orthogonal reward credit allocation method includes the following steps: Determine the reward factor stage, given the connectivity K, the number of neighbor nodes of a single agent is positively close to or negatively close to K, and a positive reward is given; the number of connected branches of the network branchess decreases, and a positive reward is given; the exploration area of the network becomes larger, and a positive reward is given; the number of task locations c accessed in the network increases, and a positive reward is given; given the communication quality qlt, the communication quality between neighbor nodes is positively or negatively close to qlt, and a positive reward is given; achieving the current maximum coverage scale and optimal communication quality, and the |V| value is smaller, a positive reward is given; achieving the current maximum coverage scale and optimal communication quality, and the usage time T or the total displacement X of the agent is smaller, a positive reward is given; single satellite mission access task priority pri; total number of task processing num_processed; total on-board memory resource consumption ; During the reward function design phase, since the MOV_BS agent and the LEO agent are trained and collaborated in the same environment, if the reward function still uses real number operations, the rewards of different types of agents will cause disturbances to the update of the strategy network of other agents, resulting in unstable training. The use of complex orthogonal operations can avoid this problem. The reward function of the MOV_BS agent is , the reward function of the LEO agent Complex reward function .

[0050] In step S4, the calculation of the network connectivity in the reward factor is obtained by combining factors such as terrain, weather, and vegetation cover, and is obtained through the link budget between any two points, including the following steps: Taking into account meteorological attenuation and multipath fading, the meteorological environment of the disaster area is sliced according to Recommendation ITU-R P530, and meteorological attenuation propagation loss and multipath fading are analyzed between any two nodes; Diffraction loss caused by large buildings or obstacles: Based on Recommendation ITU-R P526, the geographical environment between any two nodes is profiled using the first Fresnel ellipsoid to analyze electromagnetic wave diffraction propagation loss. Through free space loss , diffraction from large buildings , atmospheric absorption , basic thermal noise , dense forest penetration loss Get loss , combined with the link budget formula:

[0051]

[0052] Get the communication quality evaluation result between the two nodes, that is, the received power ,like , then the link exists, otherwise it does not exist.

[0053] The present invention also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the program implements the steps of intelligent and agile networking of an emergency wireless communication network based on OTR-MADDPG.

[0054] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0055] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0056] In summary, the present invention provides a large-scale emergency wireless communication network intelligent and agile networking method and device. According to the needs of agile networking in emergency scenarios when the network's inherent coverage is insufficient or the basic network facilities are destroyed and large-scale communications are disabled, the method designs an improved highway network exploration action space design, a strategy network update method based on shared observations of agents within connected branches of the communication network, and a multi-type agent reward accumulation strategy based on orthogonal rewards; it provides an effective solution to the problem of large-scale wireless network topology coordination under sudden environmental changes; considering that the observation of a single agent is difficult to effectively learn the environmental characteristics in complex communication scenarios, which leads to slow training, a strategy network update technology based on shared observations of agents within connected branches of the communication network is proposed; in view of the problem that the existing reward function credit allocation method for solving the simultaneous training of multiple types of agents fails to fully separate the mutual disturbance of rewards of different agents, an orthogonal reward method is introduced to fully separate the rewards of different agents.

[0057] It should be understood by those skilled in the art that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether to implement the system in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried in a carrier wave.

[0058] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0059] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0060] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A large-scale emergency wireless communication network intelligent and agile networking method, characterized in that: The method comprises the following steps: According to the communication guarantee and post-disaster communication reconstruction scenarios for earthquakes, forest fires, and large-scale floods, the large-scale emergency wireless communication network nodes are divided into mobile base station nodes (MOV_BS), unmanned aerial vehicle (UAV) aerial nodes, and low-orbit satellite nodes (LEO). The network is a self-organizing task collaboration network that generates network topology based on task coordinates, follows task location migration, and adjusts link resources based on real-time task distribution. Among them, MOV_BS is the intelligent agent on the ground side, UAV nodes serve as extensions of MOV_BS antennas, and LEO is a satellite intelligent agent that assists the ground network. The exploration space of the ground intelligent agent is extremely large, and there is task collaboration between the ground intelligent agent and the satellite intelligent agent. In order to overcome the problems of uneven credit distribution, slow convergence, and poor collaboration ability, an orthogonal reward multi-agent deep deterministic policy gradient reinforcement learning (Orthogonal Reward for Topology Aware and Road Optimized MADDPG, OTR-MADDPG) based on topology awareness and road network optimization is designed. Uneven credit distribution is addressed through orthogonal rewards, slow convergence is addressed through network optimization, and poor collaboration is addressed through topology awareness. S1. For the OTR-MADDPG, design a custom agent learning environment Env based on the initial position distribution of the emergency wireless communication network nodes, the channel environment, the feasible domain, the target coverage area, and the LEO satellite orbit; S2. Specifically, an improved action space based on road network search is designed for the MOV_BS agent, and specifically, an improved update method of the policy network π based on the state input of neighbor nodes in the connected branch is designed; S3. Specifically, design an action space for the LEO agent based on the access / rejection of LEO satellites, specifically, design a policy network based on the MOV_BS coordinate-overhead satellite state pair. Update method; S4. Designing a complex reward function , is the reward of the MOV_BS agent, R1 is positively correlated with the communication quality evaluation result between any two nodes of the MOV_BS agent, is the reward of the LEO agent, R2 is positively correlated with the amount of data processed per unit time, i is an imaginary unit, and The reward factor is obtained by the interaction between the agent and the Env obtained by S1; S5. Integrate the agent learning environment Env described in S1, the action generation method and policy network update method of the MOV_BS agent described in S2, the action generation method and policy network update method of the LEO agent described in S3, and the complex reward function described in S4 into the existing framework of MADDPG.

2. The agent learning environment Env according to S1 in claim 1, characterized in that The state information includes the agent state vector [coordinates, antenna height, maximum transmit power, transmit gain, receive gain, receive sensitivity, maximum speed, frequency band, power, fuel, neighbor set], the task location state vector [coordinates, scale, neighbor set], and the agent action vector [MOV_BS agent coordinates, whether the LEO agent is connected to the task].

3. The method for designing the action space of the MOV_BS agent according to S2 of claim 1, characterized in that: The action space is constrained to the road network considering the characteristics of the agent on board, which greatly reduces the agent's exploration time; Initialize the number of MOV_BS agents to N, input a step size △d for the MOV_BS agent, and calculate the set of nodes with the same step size based on the current coordinates of the agent and the road network. , the agent selects a node from As the next action to be executed, j represents the step number and i represents the execution round number, then the joint action of all MOV_BS agents can be expressed as an N-dimensional vector: [ ,……, ]; until the network is completed, there are episode_num steps in total. The actions of all agents can be expressed in terms of the number of steps as follows: , then the trajectory of a single agent can be expressed as express; Introducing no-traffic zones in the highway network, simulating damaged road sections, and combining multi-agent exploratory networking methods to enable agents to avoid no-traffic zones, making networking time closer to real-world networking time. The introduction of road networks and no-go zones on them increases the coupling between the agent and the real environment. This is reflected in the fact that the topology construction process is explicitly executable along roads, and the topology construction time is included as part of the reward. After the topology is generated, no further human evaluation of feasibility and practicality is required.

4. The action space of the LEO agent according to S3 of claim 1, characterized in that The real-time coordinate distribution V of the MOV_BS agent obtained by S2 is used as the task coordinate input of the LEO agent. LEO has the choice of accepting or rejecting the task generated by the MOV_BS node, that is, it has a two-dimensional action space [1,0], [0,1], where 1 represents acceptance and 0 represents rejection. The overhead time access is obtained through the STK software interface.

5. The policy network π according to S2 in claim 1, characterized in that The action obtained by executing the improved action space searched by the road network is used to obtain the real-time coordinate distribution V of the agent. Since it is a construction of a communication topology, the set of nodes with communicative links is defined as a connected branch. Crucially, when updating the parameters of the policy network π, the joint observation formed by the observation of the nodes in a single connected branch is used, rather than the local observation of a single agent, which produces a large gain for the collaboration of the communicating agents. Each agent has a value network Q and a policy network π during training. In particular, when updating the π network of the updated agent, the input is the observation of the nodes in the connected branch where the agent is located, and the value network Q still inputs the global observation. This is because the OTR-MADDPG described in claim 1 itself outputs a communication network. The agents in the connected branch are interoperable and can share observations, which does not violate the CTDE principle. The collaboration between agents is enhanced by the dependency between nodes. At the same time, in order to prevent the input scale of the policy network π from being too large, affecting the training time and over-reliance on other agents, the π network only inputs the observations of itself and neighboring nodes when updating.

6. The communication quality evaluation between any two nodes of the MOV_BS agent according to S4 in claim 1, characterized in that: The signal attenuation degree is calculated through the channel environment, and the link budget is performed in combination with the intelligent agent's own transceiver performance, focusing on multipath fading, rain attenuation, and diffraction loss caused by buildings or obstacles. According to the ITU-R P530 recommendation, the meteorological environment of the disaster area is sliced, and meteorological attenuation propagation loss and multipath fading are analyzed between any two nodes; according to the ITU-R P526 recommendation, the geographical environment between any two nodes is profiled according to the first Fresnel ellipsoid, and electromagnetic wave diffraction propagation loss is analyzed; through free space loss , diffraction from large buildings , atmospheric absorption , basic thermal noise , dense forest penetration loss Get loss , through the link budget formula: Get the communication quality evaluation result between the two nodes, that is, the received power ,like , then the link exists, otherwise it does not exist.

7. The complex reward function design according to S4 of claim 1, characterized in that: According to the mobile base station structure and wireless signal receiving and transmitting mechanism, the key attributes of the MOV_BS agent are determined, including the moving speed v, antenna type Type, antenna supported frequency band f, antenna height h, and antenna transmission power. , Antenna transmission gain , Antenna receiving gain , receiving sensitivity , the Env obtained by S1 and the agent attributes jointly constrain the link quality and connectivity between any two nodes in the MOV_BS agent, and extending to the entire network, it constrains the connectivity of the network topology; The key attributes constraining the LEO agent's reward are determined based on the channel characteristics of the LEO node's satellite-to-ground transmission and the on-board processing capabilities, including the LEO overhead time window [ , ], memory remaining R_Remain, task priority pri of MOV_BS node and UAV node, resource consumption stor_cos, task time window[ , ]; The reward factors include: a) Given a connectivity K, a single agent is rewarded if the number of neighbor nodes approaches K positively or negatively. b) The number of connected branches of the network decreases, and positive rewards are given; c) The network's exploration area becomes larger, giving positive rewards; d) The number of task locations c connected to the network increases, and positive rewards are given; e) Given a communication quality qlt, if the communication quality between neighboring nodes approaches qlt positively or negatively, a positive reward will be given; f) Achieving the current maximum coverage scale and optimal communication quality, with a smaller |V| value, will be rewarded positively; g) Achieve the current maximum coverage scale and optimal communication quality, and the usage time T or the total displacement X of the agent is smaller, and give positive rewards; h) Single satellite mission access task priority pri; i) Total number of tasks processed num_processed; j) Total memory resource consumption on the satellite ; Since the MOV_BS agent and the LEO agent are trained and collaborated in the same environment, if the reward function still uses real number operations, the reward of the MOV_BS agent will cause disturbances to the update of the LEO agent's policy network, and vice versa, resulting in unstable training. The use of complex orthogonal operations can avoid this problem. The reward function of the MOV_BS agent is , the reward function of the LEO agent .

Images