Emergency communication node element deployment method and system based on reinforcement learning

Abstract

The present disclosure belongs to the technical field of emergency communication node element deployment, and specifically provides an emergency communication node element deployment method and system based on reinforcement learning, which comprises the following steps: based on deep reinforcement learning, a full convolutional neural network with a coding and decoding architecture is used as a value network, and an agent state is designed according to different environmental factors and the type of elements currently requested to be deployed; a neural network model is trained; a road network map is constructed according to an emergency area, input into the trained neural network model, and an element deployment scheme is output. The present disclosure is based on a new algorithm of deep Q learning. In the algorithm, a full convolutional neural network is used as a value network, and the network is trained using data accumulated in an experience replay area. Compared with manual deployment, experimental results show that the proposed method can better complete the emergency communication node element planning task and greatly improve the deployment efficiency of the emergency communication node.

Description

Technical Field

[0001] The present disclosure relates to the technical field of emergency communication node element deployment, and in particular to a reinforcement learning-based emergency communication node element deployment method and system. Background Art

[0002] An emergency communications network generally refers to a specialized communications network constructed to ensure emergency response and essential communications during major natural disasters or sudden emergencies, leveraging various communication resources. An emergency communications node typically consists of seven components: an on-site command center, switches, satellite vehicles, microwave transmitters, optical transceivers, mobile communications vehicles, and shortwave vehicles. Satellite, microwave, and optical cable, three high-bandwidth communication methods, serve as the transmission network, providing communication services between the disaster area and the command center. Shortwave and mobile communication networks serve as the command network, providing wireless access to users at the disaster site and transmitting real-time information back to the on-site command center. With the increasing frequency of natural disasters, a stable and reliable emergency communications network is essential to ensure that disaster areas do not become "information islands." Current emergency communications networks are multi-dimensional, integrated, and heterogeneous networks constructed using a variety of emergency communications methods. This allows the on-site command center to establish a high-bandwidth, reliable transmission link with the command center via the transmission network, while also establishing extensive command network coverage at the disaster site.

[0003] To ensure efficient operation of emergency communication nodes, the deployment of various elements within the hub often requires comprehensive consideration of multiple factors, including terrain, communication range, disaster direction, and mutual interference between elements. This often prevents emergency communication commanders from quickly planning a reasonable configuration plan for emergency communication network elements to guide the rapid establishment of the emergency communication network.

[0004] Emergency communication nodes contain multiple communication elements. The placement of each element directly determines the overall communication effectiveness of the hub. Manual deployment often requires extensive knowledge and a long configuration time. This makes it difficult for commanders to design high-quality deployment plans in a short period of time.

[0005] There has been considerable research on traditional communication network planning, primarily using heuristic algorithms such as minimum spanning trees, simulated annealing algorithms, and genetic algorithms to optimize network topology. However, for complex communication network planning problems, a single heuristic algorithm can become trapped in local optimal solutions and fail to achieve ideal planning results. In their paper, Yin Changsheng et al. proposed migrating concepts such as networks and nodes to chessboards and pieces, and using Alpha Zero's deep reinforcement learning algorithm to plan emergency communication networks. However, these methods are highly abstract, optimizing the network's topology and rarely considering the impact of external factors such as terrain on the network. Furthermore, the neural networks used in deep reinforcement algorithms are mostly convolutional neural network (CNN) architectures, with fully connected output layers, which greatly limits the range of operational options available to the agent. Summary of the Invention

[0006] The present disclosure aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for deploying emergency communication node elements based on reinforcement learning.

[0007] In a first aspect, the present disclosure provides a method for deploying emergency communication node elements based on reinforcement learning, comprising:

[0008] S1, based on deep reinforcement learning, uses a fully convolutional neural network with an encoding / decoding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0009] S2, training neural network model;

[0010] S3, builds a road network map based on the emergency area, inputs it into the trained neural network model, and outputs the element deployment plan.

[0011] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0012] When using deep reinforcement learning, the state of the intelligent agent is related to the type of element currently requested for deployment. During deployment, the initial element deployed is the on-site command center, followed by the switch. The remaining five elements, namely the satellite vehicle, microwave transmitter, optical terminal, mobile communication vehicle, and shortwave vehicle, are deployed in random order.

[0013] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0014] When using deep reinforcement learning, the state of the intelligent agent is related to the geographical environment. During deployment, the elements of the emergency communication network are large vehicles, so the locations where the elements are opened are also on both sides of the road. The reward R1 is set to 1 for the road area T, and no reward is set in other areas.

[0015] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0016] When using deep reinforcement learning, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the emergency communication network elements, each element is kept as far away from the disaster center as possible, and the disaster impact range is set to R2;

[0017]

[0018] Among them, the distance from the disaster to the element is d.

[0019] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0020] When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows:

[0021]

[0022] Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

[0023] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0024] For optical terminals, the location of the optical cable connection point is crucial. To reduce transmission attenuation, reduce the complexity of optical cable wiring, and facilitate maintenance and management, the optical terminal is usually installed near the optical cable connection point. The reward function R4 is designed as follows:

[0025]

[0026] Among them, the distance from the optical terminal to the optical cable connection point is d.

[0027] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0028] When using deep reinforcement learning, the agent's state is related to the microwave direction. During deployment, the propagation loss L is calculated based on the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location using the electromagnetic wave free space attenuation formula, and this is used as the microwave direction reward R5.

[0029]

[0030] Among them, G t ,G r ,C,f are the microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency respectively.

[0031] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0032] When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows:

[0033]

[0034] Among them, IoU is the communication coverage of the element A com and the affected area A dis The intersection and union ratio.

[0035] Preferably, the agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including:

[0036]

[0037] Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

[0038] Preferably, the S2 specifically includes:

[0039] S21, randomly select H {s i , a i , r i , s i+1}, used to train neural networks;

[0040] S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and deployment-capable areas are extracted to calculate the value; the action space A contains (n×n) actions;

[0041] S23, use the target network to calculate s i+1 The best action a in the statemax =argmaxQ w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the action a at this point t The value of the action space A', the maximum value of all actions q max Constructing set Q′ t+1 ;

[0042] S24, using bilinear interpolation, Q' t+1 The size of becomes back to (n×n);

[0043] S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w is the value network, Q w- For the target network, we get the estimated next step value Q′ and use the temporal difference algorithm to calculate the target value matrix (R+γQ′);

[0044] Q′=Q w -(s t+1 , argmaxQ w (s t+1 , a))

[0045] S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters;

[0046]

[0047] Among them, R j For each reward item, s t ,a t ,r t are the state, action and reward at time t, and γ is the discount factor.

[0048] The present invention also provides an emergency communication node element deployment system based on reinforcement learning, which is used to implement the emergency communication node element deployment method based on reinforcement learning, including:

[0049] The modeling module is configured based on deep reinforcement learning and uses a fully convolutional neural network with an encoding and decoding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0050] a training module configured to train a neural network model;

[0051] The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the feature deployment plan.

[0052] Beneficial effects:

[0053] The problem of deploying emergency communication network elements is formulated as a Markov decision process, where the deployment of each element depends solely on its current state. Based on a high-resolution road network map and through extensive exploration and learning of the environment by an intelligent agent, a deep reinforcement learning algorithm automatically discovers the complex relationships between the environment and emergency communication network elements, as well as between each element. Based on this, it generates optimized deployment locations for these elements, thereby improving the reliability of the emergency communication network element configuration plan. This algorithm uses a fully convolutional neural network as the value network and trains the network using data accumulated in the experience replay area. Compared to manual deployment, experimental results demonstrate that the proposed method effectively completes the task of planning emergency communication node elements and significantly improves the deployment efficiency of emergency communication nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A diagram showing the overall structure of a model for the reinforcement learning-based emergency communication node element deployment method provided in an embodiment of the present disclosure;

[0055] Figure 2 A graph of average rewards and losses for algorithm training provided by embodiments of the present disclosure;

[0056] Figure 3 An example diagram of a deployment solution provided for an embodiment of the present disclosure;

[0057] Figure 4 A schematic diagram of the emergency communication node topology structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the present disclosure is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning as understood by a person of ordinary skill in the art to which the present disclosure pertains. The terms "first", "second", and similar terms used in the present disclosure do not denote any order, quantity, or importance, but are used to distinguish different components. Similarly, the terms "one", "a", or "the" and similar terms do not denote a quantity of at least one, but are used to indicate the existence of at least one. The terms "comprise", "comprising", and similar terms mean that the elements or objects listed after the terms encompass the elements or objects listed and equivalents thereof, and do not exclude other elements or objects. The terms "connected" or "coupled" and similar terms do not limit to physical or mechanical connections or couplings, but can include electrical connections or couplings, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are used only to describe relative positions, and when the absolute positions of the objects described are changed, the relative positions can also be changed accordingly.

[0060] In the various drawings, like reference numerals are used to refer to like elements throughout. For the sake of clarity, not every component is labeled in every drawing. In addition, some components can not be shown in the drawings.

[0061] Many specific details of the present disclosure are described below in order to provide a thorough understanding of the present disclosure. However, as will be readily understood by one skilled in the art, the present disclosure can be practiced without these specific details.

[0062] With the rapid development of artificial intelligence, it is possible to provide automatic deployment for emergency communication network elements through deep reinforcement learning technology. By using reinforcement learning, intelligent deployment of emergency communication node elements can effectively enhance the rapid construction capability of information communication guarantee in emergency situations, alleviate the contradiction between long time consumption and efficient deployment of emergency communication network elements, and conform to the trend of intelligent decision-making, which has strong practical significance for improving the rapid response capability of information communication guarantee.

[0063] Based on this, the embodiment of the present application provides an emergency communication node element deployment method based on reinforcement learning, which comprises:

[0064] S1, based on deep reinforcement learning, a fully convolutional neural network with a coding and decoding architecture is used as a value network, and an agent state is designed according to different environmental factors and the type of elements currently requested to be deployed;

[0065] S2, training a neural network model;

[0066] S3, constructing a road network map according to an emergency area, inputting the road network map into the trained neural network model, and outputting an element deployment scheme.

[0067] It should be noted that the technical solution of the present disclosure does not limit the order of the above steps, that is, the execution order of the steps can be arranged arbitrarily.

[0068] like Figure 1 As shown in Figure 1, reinforcement learning is based on the Double DQN algorithm. The architecture of the model is shown in Figure 1, which mainly consists of two parts: the value network and the intelligent agent. The value network uses a lightweight fully convolutional neural network (FCN) as the backbone network to estimate the Q value of different locations, that is, to treat the complex Q value table as data and use a parameterized function to fit the data. The fully convolutional neural network extracts features from the input environmental feature map and the distance interval map, and finally generates a Q value matrix to estimate the Q value of each location in the area where the communication hub is expected to be opened. The intelligent agent uses a greedy strategy to select action a according to the Q value of different states. t By performing action a t , obtain environmental rewards r t , and the state will change to s t+1 The agent will (s t ,a t ,r t ,s t+1 ) matrix is ​​stored in the experience replay area (Replay Buffer).

[0069] (1) The state in reinforcement learning refers to the set of information that the agent can observe at any given moment. Referring to this concept, the state of the problem is designed. First, the road network map is divided into (n×n) areas. In solving the problem of intelligent configuration of emergency communication network elements based on deep reinforcement learning, the state of the agent is not only related to the type of element currently requested for placement, but also to factors such as the geographical environment, the communication range between elements, and the direction of the disaster. The state matrices of the expression factors are superimposed to form a multi-channel state matrix, which is used as the input of the neural network.

[0070] Further solutions, in S1, the agent state is designed according to different environmental factors and the type of elements currently requested for deployment, including:

[0071] When using deep reinforcement learning, the state of the intelligent agent is related to the type of element currently requested for deployment. During deployment, the initial element deployed is the on-site command center, followed by the switch. The remaining five elements, namely the satellite vehicle, microwave transmitter, optical terminal, mobile communication vehicle, and shortwave vehicle, are deployed in random order.

[0072] (2) Action design. In this problem, the choice of the element to its deployment location is considered as action a t. Elements can be deployed in (n×n) units, so the action space A should also contain (n×n) actions. The intelligent experience will be based on the current state s of the element t , the Q value matrix is ​​calculated by the value network. The intelligent experience will use the greedy strategy to choose the location of the next element based on the obtained Q value matrix. t The set of locations selected for all node element deployments is {a1, a2, ..., a t}, which is the emergency communication node element deployment plan.

[0073] (3) Reward design. In practice, factors such as different geographical environments (such as terrain features) and distances between elements will affect the efficiency of the emergency communication network and the channel transmission quality of different communication means. In order to use reinforcement learning methods to solve the deployment problem of emergency communication network elements, it is necessary to first model the environment and design a reasonable reward function. In this problem, the rewards given by the environment to different elements can mainly come from several aspects: geographical environment, disaster direction, communication distance between elements, and interval distance. The specific calculation method of each reward is shown below. For different elements, the proportion of each type of reward is different.

[0074] Further solutions are designed to design the agent state based on different environmental factors and the type of elements currently requested for deployment, including:

[0075]

[0076] Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

[0077] (1) Geographical environment. When using deep reinforcement learning, the state of the agent is related to the geographical environment. During deployment, the elements of the emergency communication network are large vehicles, so the locations of the elements are also on both sides of the road. The reward R1 is set to 1 for the road area T, and no reward is set in other areas.

[0078] (2) Disaster direction. When using deep reinforcement learning, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the elements of the emergency communication network, each element is kept as far away from the disaster center as possible, and the disaster impact range is set to R2;

[0079]

[0080] Among them, the distance from the disaster to the element is d.

[0081] (3) Communication distance between elements. When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows:

[0082]

[0083] Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

[0084] (4) Cable connection location. For optical terminals, the location of the cable connection point is very critical. To reduce transmission attenuation, reduce the complexity of optical cable wiring, and facilitate maintenance and management, the optical terminal is usually installed near the cable connection point. The reward function R4 is designed as follows:

[0085]

[0086] Among them, the distance from the optical terminal to the optical cable connection point is d.

[0087] (5) Microwave direction. When using deep reinforcement learning, the state of the agent is related to the microwave direction. When deployed, the propagation loss L is calculated based on the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location using the electromagnetic wave free space attenuation formula, and this is used as the microwave direction reward R5;

[0088]

[0089] Among them, G t ,G r ,C,f are the microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency respectively.

[0090] (6) Coverage. When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows:

[0091]

[0092] Among them, IoU is the communication coverage of the element A com and the affected area A dis The intersection and union ratio.

[0093] A further approach is to train a neural network model based on the Double Deep Q Network (DDQN), using a fully convolutional neural network with an encoding / decoding architecture as the value network. The algorithm flow is as follows:

[0094] S21, randomly select H {s i , a i , r i , s i+1}, used to train neural networks;

[0095] S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and deployment-capable areas are extracted to calculate the value; the action space A contains (n×n) actions;

[0096] S23, use the target network to calculate s i+1 The best action a in the state max =argmaxQ w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the action a at this point t The value of the action space A', the maximum value of all actions q max Construct set Q′ t+1 ;

[0097] S24, using bilinear interpolation, Q' t+1 The size of becomes back to (n×n);

[0098] S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w is the value network, Q w- For the target network, we get the estimated next step value Q′ and use the temporal difference algorithm to calculate the target value matrix (R+γQ′);

[0099] Q′=Q w -(s t+1 , argmaxQ w (s t+1 , a))

[0100] S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters;

[0101]

[0102] Among them, R j For each reward item, s t ,a t ,r t are the state, action and reward at time t, and γ is the discount factor.

[0103] Table 1 Glossary

[0104]

[0105] Finally, the above model was simulated. In this simulation, elements were deployed in a planned deployment area of ​​1.2 km x 1.2 km. The planned deployment area was divided into 256 × 256 pixels, with each pixel representing a distance of approximately 6 meters. Seven emergency communication network elements were deployed in this environment. I set a learning rate of 0.0001, an experience replay size of 10,000, a training batch size of 8, and a target network update interval of 20. This algorithm was trained for 1,000 epochs. All simulations were performed on an NVIDIA RTX 3090Ti GPU, using Ubuntu 22.04.4, Python 3.10, and the Pytorch 2.3 deep learning framework.

[0106] To find the optimal deployment model, we used two reinforcement learning algorithms (DQN and DDQN) and two value network frameworks, VGG and Resnet. We compared the performance of the four methods. Their different combinations helped us select the best one for the deployment task. Figure 2 In Figure 2, we compare the rewards of the four combinations in each training episode. It is easy to see that DDQN outperforms DQN, and the value network using VGG is more powerful than the value network using Resnet.

[0107] The four combinations were then compared with manual methods to verify the feasibility of the algorithm. A greedy strategy was used to simulate the manual method for deploying each element. The average reward for each method was estimated using 100 deployment rounds. Table 2 shows that the ResNet-based DDQN method outperforms other methods, including the manual deployment method.

[0108] Table 2 Average rewards of different algorithms

[0109]

[0110] Deployment scenario examples include Figure 3 Road areas are shown in black and other landforms are shown in white. The specific meanings of the seven icons in the figure are shown in Figure 4 .

[0111] Several innovative improvements of the present disclosure are as follows:

[0112] (1) The impact of various environmental factors on different elements within the hub was fully considered, quantified, and a reward function for evaluation was established.

[0113] (2) A deep reinforcement learning model based on a fully convolutional neural network (FCN) is proposed. This model uses a fully convolutional neural network as its value network, effectively expanding the action space of the intelligent agent.

[0114] (3) A new loss function is designed to enable the neural network to better simulate the value matrix.

[0115] An embodiment of the present invention further provides a reinforcement learning-based emergency communication node element deployment system, which is used to implement a reinforcement learning-based emergency communication node element deployment method, including:

[0116] The modeling module is configured based on deep reinforcement learning and uses a fully convolutional neural network with an encoding and decoding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0117] a training module configured to train a neural network model;

[0118] The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the feature deployment plan.

[0119] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for deploying emergency communication node elements based on reinforcement learning, characterized in that: include: S1, based on deep reinforcement learning, uses a fully convolutional neural network with an encoding / decoding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment; S2, training neural network model; Wherein, the S2 includes: S21, randomly select H {s i ,a i ,r i ,s i+1 }, used to train neural networks; S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and deployment-capable areas are extracted to calculate the value; the action space A contains (n×n) actions; S23, use the target network to calculate s i+1 The best action a in the state max =argmax Q w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the action a at this point t The value of the action space A', the maximum value of all actions q max Constructing set Q′ t+1 ; S24, using bilinear interpolation, Q′ t+1 The size of becomes back to (n×n); S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w is the value network, Q w- For the target network, we get the estimated next step value Q′ and use the temporal difference algorithm to calculate the target value matrix (R+γQ′); Q′=Q w -(s t+1 ,argmaxQ w (s t+1 ,a)) S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters; Among them, R j For each reward item, s t ,a t ,r t are the state, action and reward at time t, respectively, and γ is the discount factor; S3, builds a road network map based on the emergency area, inputs it into the trained neural network model, and outputs the element deployment plan.

2. The method for deploying emergency communication node elements based on reinforcement learning according to claim 1, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: When using deep reinforcement learning, the state of the intelligent agent is related to the type of element currently requested for deployment. During deployment, the initial element deployed is the on-site command center, followed by the switch. The remaining five elements, namely the satellite vehicle, microwave transmitter, optical terminal, mobile communication vehicle, and shortwave vehicle, are deployed in random order.

3. The method for deploying emergency communication node elements based on reinforcement learning according to claim 2, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: When using deep reinforcement learning, the state of the agent is related to the geographical environment. During deployment, the elements of the emergency communication network are large vehicles, so the locations of the elements are also located on both sides of the road. The reward R1 is set to 1 in the road area T, and no reward is set in other areas. When using deep reinforcement learning, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the emergency communication network elements, each element is kept as far away from the disaster center as possible, and the disaster impact range is set to R2; Among them, the distance from the disaster to the element is d.

4. The method for deploying emergency communication node elements based on reinforcement learning according to claim 3, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows: Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

5. The method for deploying emergency communication node elements based on reinforcement learning according to claim 4, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: For the optical terminal, the reward function R4 is designed as follows: Among them, the distance from the optical terminal to the optical cable connection point is d.

6. The method for deploying emergency communication node elements based on reinforcement learning according to claim 5, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: When using deep reinforcement learning, the agent's state is related to the microwave direction. During deployment, the propagation loss L is calculated based on the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location using the electromagnetic wave free space attenuation formula, and this is used as the microwave direction reward R5. Among them, G t ,G r ,C,f are the microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency respectively.

7. The method for deploying emergency communication node elements based on reinforcement learning according to claim 6, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows: Among them, IoU is the communication coverage of the element A com and the affected area A dis The intersection and union ratio.

8. The method for deploying emergency communication node elements based on reinforcement learning according to claim 7, characterized in that: The agent state design in S1 is performed according to different environmental factors and the type of elements currently requested for deployment, specifically including: Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

9. Emergency communication node element deployment system based on reinforcement learning, characterized by: The system is used to implement the reinforcement learning-based emergency communication node element deployment method according to any one of claims 1 to 8, comprising: The modeling module is configured based on deep reinforcement learning and uses a fully convolutional neural network with an encoding and decoding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment; a training module configured to train a neural network model; The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the feature deployment plan.

Images