Emergency communication node element deployment method and system based on reinforcement learning

Abstract

The invention belongs to the technical field of emergency communication node element deployment, and particularly provides an emergency communication node element deployment method and system based on reinforcement learning, and the method comprises the steps: employing a full convolutional neural network with a coding and decoding architecture as a value network based on deep reinforcement learning, performing agent state design according to different environmental factors and the element types requested to be deployed currently; training a neural network model; and constructing a road network map according to the emergency area, inputting the road network map to the trained neural network model, and outputting an element deployment scheme. The invention discloses a brand new algorithm based on deep Q learning. According to the algorithm, a full convolutional neural network is used as a value network, and the network is trained by using data accumulated in an experience playback area. Compared with manual deployment, experimental results show that the provided method can well complete emergency communication node element planning tasks, and the deployment efficiency of the emergency communication nodes is greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of emergency communication node element deployment, and in particular to an emergency communication node element deployment method and system based on reinforcement learning. Background Art

[0002] Emergency communication network generally refers to a special communication network built to ensure emergency response and necessary communication by comprehensively utilizing various communication resources when a major natural disaster or emergency occurs. An emergency communication node is usually composed of seven elements, including: on-site command center, switch, satellite vehicle, microwave transmitter, optical terminal, mobile communication vehicle and shortwave vehicle. Among them, satellite, microwave and optical cable, three large-bandwidth communication means, are used as transmission networks to provide communication services for the disaster area and the rear command center; shortwave network and mobile communication network can be used as command networks to provide wireless access services for users in the disaster relief site area and transmit front-line information back to the on-site command center in real time. With the increasing frequency of natural disasters, in order to ensure that the disaster area does not become an "information island", a stable and reliable emergency communication network must be built. The current emergency communication network is a multi-dimensional, integrated, heterogeneous network constructed by a variety of emergency communication means. This allows the on-site command center to establish a large-bandwidth, reliable transmission link with the rear command center through the transmission network, and to establish a large-scale command network coverage at the disaster relief site.

[0003] In order to ensure the efficient operation of emergency communication nodes, the deployment of various elements in the hub often needs to comprehensively consider multiple factors such as terrain, communication range, disaster direction, and mutual interference of elements. This results in emergency communication commanders often being unable to plan a reasonable configuration plan for emergency communication network elements in a short period of time to guide the rapid opening of the emergency communication network.

[0004] Emergency communication nodes contain a variety of communication elements. The deployment location of each element will directly determine the overall communication effect of the hub. Manual deployment often requires more knowledge and a longer configuration time. This makes it impossible for commanders to design high-quality deployment plans in a short period of time.

[0005] There have been quite a lot of traditional research on communication network planning, mainly using heuristic algorithms such as minimum spanning tree, simulated annealing algorithm and genetic algorithm to optimize the network topology. However, for complex communication network planning problems, a single heuristic algorithm will fall into the local optimal solution and cannot achieve the ideal planning effect. In the paper, Yin Changsheng et al. proposed to migrate concepts such as networks and nodes to chessboards and chess pieces, and use Alpha Zero's deep reinforcement learning algorithm to plan emergency communication networks. However, the above are all highly abstract communication networks, optimizing the topological structure of the network, and rarely considering the impact of external factors such as terrain on the network. At the same time, the neural networks used in deep reinforcement algorithms are mostly convolutional neural network CNN architectures, and the output layer is a fully connected layer, which greatly limits the operating space available to the intelligent agent. Summary of the invention

[0006] The present disclosure aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for deploying emergency communication node elements based on reinforcement learning.

[0007] In a first aspect, the present disclosure provides a method for deploying emergency communication node elements based on reinforcement learning, including:

[0008] S1, based on deep reinforcement learning, uses a fully convolutional neural network with a coding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0009] S2, training neural network model;

[0010] S3, constructs a road network map based on the emergency area, inputs it into the trained neural network model, and outputs the element deployment plan.

[0011] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0012] When using deep reinforcement learning, the state of the agent is related to the type of element currently requested for deployment. During deployment, the initial deployment element is the on-site command center, followed by the switch. The remaining five elements, satellite vehicles, microwave transmitters, optical terminals, mobile communication vehicles, and shortwave vehicles, are deployed in random order.

[0013] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0014] When using deep reinforcement learning, the state of the intelligent agent is related to the geographical environment. During deployment, the elements of the emergency communication network are large vehicles, so the locations where the elements are opened are also on both sides of the road. The reward R1 is set to 1 for the road area T, and no reward is set in other areas.

[0015] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0016] When deep reinforcement learning is used, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the elements of the emergency communication network, each element is kept as far away from the disaster center as possible, and the disaster impact range is set to R2;

[0017]

[0018] Among them, the distance from the disaster to the element is d.

[0019] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0020] When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows:

[0021]

[0022] Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

[0023] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0024] For optical terminals, the location of the optical cable connection point is very critical. In order to reduce transmission attenuation, reduce the complexity of optical cable wiring, and facilitate maintenance and management, the optical terminal is usually installed near the optical cable connection point. The design reward function R4 is as follows:

[0025]

[0026] Among them, the distance from the optical terminal to the optical cable connection point is d.

[0027] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0028] When using deep reinforcement learning, the state of the agent is related to the direction of the microwave. When deployed, the electromagnetic wave free space attenuation formula is used to calculate the propagation loss L according to the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location, and it is used as the microwave direction reward R5;

[0029]

[0030] Among them, G t ,G r ,C, and f are microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency, respectively.

[0031] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0032] When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows:

[0033]

[0034] Among them, IoU is the communication coverage of the element A com Affected area A dis The intersection and ratio of .

[0035] Preferably, the agent state design in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including:

[0036]

[0037] Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

[0038] Preferably, the S2 specifically includes:

[0039] S21, randomly select H {s i , a i , r i ,s i+1}, used to train neural networks;

[0040] S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and areas with deployment capabilities are extracted to calculate the value; the action space A contains (n×n) actions;

[0041] S23, use the target network to calculate s i+1 The best action in the state amax = argmaxQ w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the point action a t The value of the action space A' is the maximum value of all actions in the action space A'. max Constructing the set Q′ t+1 ;

[0042] S24, using bilinear interpolation, Q′ t+1 The size of becomes back to (n×n);

[0043] S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w is the value network, Q w- For the target network, get the estimated next step value Q′, and use the temporal difference algorithm to calculate the target value matrix (R+γQ′);

[0044] Q′=Q w -(s t+1 , argmaxQ w (s t+1 , a))

[0045] S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters;

[0046]

[0047] Among them, R j For each award item, s t ,a t ,r t are the state, action and reward at time t respectively, and γ is the discount factor.

[0048] The present invention also provides an emergency communication node element deployment system based on reinforcement learning, and the system is used to implement the emergency communication node element deployment method based on reinforcement learning, including:

[0049] The modeling module is configured based on deep reinforcement learning, using a fully convolutional neural network with a codec architecture as the value network, and designs the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0050] a training module configured to train a neural network model;

[0051] The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the element deployment plan.

[0052] Beneficial effects:

[0053] The problem of the location deployment of emergency communication network elements is regarded as a Markov decision process. The deployment of each element of the emergency communication network depends only on the current state. Based on the high-resolution road network map, through the agent's extensive exploration and learning of the environment, the deep reinforcement learning algorithm can automatically discover the complex relationship between the opening environment and the emergency communication network elements, and between the emergency communication network elements, and generate optimized emergency communication network element deployment locations based on this, thereby improving the reliability of the emergency communication network element configuration scheme. The algorithm uses a fully convolutional neural network as a value network and uses the data accumulated in the experience playback area to train the network. Compared with manual deployment, the experimental results show that the proposed method can better complete the emergency communication node element planning task and greatly improve the deployment efficiency of emergency communication nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The overall structure diagram of the model of the emergency communication node element deployment method based on reinforcement learning provided in the embodiment of the present disclosure;

[0055] Figure 2 Average reward and loss graph for algorithm training provided by embodiments of the present disclosure;

[0056] Figure 3 An example diagram of a deployment solution provided for an embodiment of the present disclosure;

[0057] Figure 4 A schematic diagram of the emergency communication node topology structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure are not any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" are not limited to quantity, but are at least one. "Including" or "including" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used for relative positional relationships. When the absolute position of the described target changes, the relative positional relationship may also change accordingly.

[0060] In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not all drawn to scale. In addition, some well-known parts may not be shown in the drawings.

[0061] Many specific details of the present disclosure are described below, such as component structures, materials, dimensions, processing techniques, and technologies, in order to more clearly understand the present disclosure. However, as those skilled in the art will appreciate, the present disclosure may be implemented without following these specific details.

[0062] With the rapid development of artificial intelligence, deep reinforcement learning technology has provided new possibilities for the automatic deployment of emergency communication network elements. Using reinforcement learning methods to intelligently deploy emergency communication node elements can effectively enhance the ability to quickly build information and communication security in emergencies, alleviate the contradiction between the time-consuming configuration of emergency communication network elements and efficient opening, conform to the general trend of intelligent decision-making, and have strong practical significance for improving the rapid response capability of information and communication security.

[0063] Based on this, an embodiment of the present invention provides an emergency communication node element deployment method based on reinforcement learning, including:

[0064] S1, based on deep reinforcement learning, uses a fully convolutional neural network with a coding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0065] S2, training neural network model;

[0066] S3, constructs a road network map based on the emergency area, inputs it into the trained neural network model, and outputs the element deployment plan.

[0067] It should be noted that the technical solution of the present disclosure does not limit the sequence of the above steps, that is, the execution order of the steps can be arranged arbitrarily.

[0068] like Figure 1 As shown in Figure 1, reinforcement learning is based on the Double DQN algorithm. The architecture of the model is shown in Figure 2, which mainly includes two parts: the value network and the intelligent agent. The value network uses a lightweight fully convolutional neural network (FCN) as the backbone network to estimate the Q values ​​of different locations, that is, to treat the complex Q value table as data and use a parameterized function to fit the data. The fully convolutional neural network extracts features from the input environmental feature map and the distance interval map, and finally generates a Q value matrix to estimate the Q value of each location in the area where the communication hub is expected to be opened. The intelligent agent uses a greedy strategy to select action a according to the Q value of different states. t By performing action a t , get environmental return r t , and the state will change to s t+1 The agent will (s t ,a t ,r t ,s t+1 ) matrix is ​​stored in the experience replay area (Replay Buffer).

[0069] (1) The state in reinforcement learning refers to the set of information that the agent can observe at any given moment. With reference to this concept, the state of this problem is designed. First, the road network map is divided into (n×n) regions. In solving the problem of intelligent configuration of emergency communication network elements based on deep reinforcement learning, the state of the agent is not only related to the type of element currently requested to be placed, but also to factors such as the geographical environment, the communication range between elements, and the direction of the disaster. The state matrices of the expression factors are superimposed to form a multi-channel state matrix, which is used as the input of the neural network.

[0070] Further solutions, in S1, the agent state is designed according to different environmental factors and the type of elements currently requested for deployment, including:

[0071] When using deep reinforcement learning, the state of the agent is related to the type of element currently requested for deployment. During deployment, the initial deployment element is the on-site command center, followed by the switch. The remaining five elements, satellite vehicles, microwave transmitters, optical terminals, mobile communication vehicles, and shortwave vehicles, are deployed in random order.

[0072] (2) Action design. In this problem, the choice of the element for its deployment location is regarded as action a. t. Elements can be deployed in (n×n) units, so the action space A should also contain (n×n) actions. The agent will be based on the current state s of the element t , the Q value matrix is ​​calculated by the value network. The intelligent body will use the greedy strategy to choose the opening position of the next element based on the obtained Q value matrix. t The set of locations selected for all node element deployments is {a1, a2, …, a t}, which is the deployment plan for emergency communication node elements.

[0073] (3) Reward design. In practice, various factors such as different geographical environments (such as terrain features) and distances between elements will affect the efficiency of the establishment of emergency communication networks and the channel transmission quality of different communication methods. In order to use reinforcement learning methods to solve the deployment problem of emergency communication network elements, it is necessary to first model the environment and design a reasonable reward function. In this problem, the rewards given by the environment to different elements can mainly come from several aspects: geographical environment, disaster direction, communication distance between elements, and interval distance. The specific calculation method of each reward is as follows. For different elements, the proportion of each type of reward is different.

[0074] Further solutions are designed according to different environmental factors and the type of elements currently requested for deployment, including:

[0075]

[0076] Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

[0077] (1) Geographical environment. When using deep reinforcement learning, the state of the agent is related to the geographical environment. When deployed, the elements of the emergency communication network are large vehicles, so the locations where the elements are opened are also on both sides of the road. The reward R1 is set to 1 for the road area T, and no reward is set in other areas.

[0078] (2) Disaster direction. When using deep reinforcement learning, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the elements of the emergency communication network, each element is as far away from the disaster center as possible, and the disaster impact range is set to R2;

[0079]

[0080] Among them, the distance from the disaster to the element is d.

[0081] (3) Communication distance between elements. When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows:

[0082]

[0083] Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

[0084] (4) The location of the optical cable connection point. For optical terminals, the location of the optical cable connection point is very critical. In order to reduce transmission attenuation, reduce the complexity of optical cable wiring, and facilitate maintenance and management, the optical terminal is usually installed near the optical cable connection point. The design reward function R4 is as follows:

[0085]

[0086] Among them, the distance from the optical terminal to the optical cable connection point is d.

[0087] (5) Microwave direction. When using deep reinforcement learning, the state of the agent is related to the microwave direction. When deployed, the propagation loss L is calculated based on the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location using the electromagnetic wave free space attenuation formula, and it is used as the microwave direction reward R5;

[0088]

[0089] Among them, G t ,G r ,C, and f are microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency, respectively.

[0090] (6) Coverage. When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows:

[0091]

[0092] Among them, IoU is the communication coverage of the element A com Affected area A dis The intersection and ratio of .

[0093] As a further solution, the algorithm for training the neural network model is based on Double Deep Q Network (DDQN), which uses a fully convolutional neural network with a coding architecture as the value network. The algorithm flow is as follows:

[0094] S21, randomly select H {s i , a i , r i ,s i+1}, used to train neural networks;

[0095] S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and areas with deployment capabilities are extracted to calculate the value; the action space A contains (n×n) actions;

[0096] S23, use the target network to calculate s i+1 The best action in the state a max = argmaxQ w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the point action a t The value of the action space A' is the maximum value of all actions in the action space A'. max Constructing the set Q′ t+1 ;

[0097] S24, using bilinear interpolation, Q′ t+1 The size of becomes back to (n×n);

[0098] S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w is the value network, Q w- For the target network, get the estimated next step value Q′, and use the temporal difference algorithm to calculate the target value matrix (R+γQ′);

[0099] Q′=Q w -(s t+1 , argmaxQ w (s t+1 , a))

[0100] S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters;

[0101]

[0102] Among them, R j For each award item, s t ,a t ,r t are the state, action and reward at time t respectively, and γ is the discount factor.

[0103] Table 1 Glossary

[0104]

[0105] Finally, the above model is simulated. In this simulation, the elements are deployed in the expected opening area of ​​1.2km*1.2km. The expected opening area is divided into 256pixels×256pixels, and the side length of each pixel represents a distance of about 6m. Seven emergency communication network elements need to be deployed in this environment. I set the learning rate to 0.0001, the experience playback area size to 10000, the training batch size to 8, and the target network update interval to 20. 1000 rounds of training were performed based on this algorithm. All simulations were performed on an NVIDIA RTX 3090Ti GPU, using the Ubuntu22.04.4 system, using Python3.10 and Pytorch2.3 deep learning frameworks.

[0106] To find the best deployment model, two reinforcement learning algorithms (DQN and DDQN) and two value network frameworks, VGG and Resnet, were used. The performance of the four methods was compared. Their different combinations were helpful in selecting the best one for the deployment task. Figure 2 In Figure 2, the rewards of the four combinations in each training episode are compared. It is easy to see that DDQN outperforms DQN, and the value network using VGG is more powerful than the value network using Resnet.

[0107] Then, the four combinations are compared with the manual method to verify the feasibility of the algorithm. The greedy strategy is used to simulate the manual method to deploy each element. The average reward of each method is estimated using 100 rounds of deployment. From Table 2, the Resnet-based DDQN method outperforms other methods, including: Manual deployment method.

[0108] Table 2 Average rewards for different algorithms

[0109]

[0110] Deployment scenario examples include Figure 3 Road areas are shown in black and other landforms are shown in white. The specific meanings of the seven icons in the figure are shown in Figure 4 .

[0111] Several innovative improvements of the present disclosure are as follows:

[0112] (1) The impact of various environmental factors on different elements within the hub was fully considered, quantified, and a reward function for evaluation was established.

[0113] (2) A deep reinforcement learning model based on a fully convolutional neural network (FCN) is proposed. This model uses a fully convolutional neural network as its value network, which effectively increases the action space of the intelligent agent.

[0114] (3) A new loss function is designed to enable the neural network to better simulate the value matrix.

[0115] The embodiment of the present invention further provides an emergency communication node element deployment system based on reinforcement learning, and the system is used to implement an emergency communication node element deployment method based on reinforcement learning, including:

[0116] The modeling module is configured based on deep reinforcement learning, using a fully convolutional neural network with a codec architecture as the value network, and designs the agent state according to different environmental factors and the type of elements currently requested for deployment;

[0117] a training module configured to train a neural network model;

[0118] The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the element deployment plan.

[0119] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for deploying emergency communication node elements based on reinforcement learning, characterized in that: include: S1, based on deep reinforcement learning, uses a fully convolutional neural network with a coding architecture as the value network to design the agent state according to different environmental factors and the type of elements currently requested for deployment; S2, training neural network model; S3, constructs a road network map based on the emergency area, inputs it into the trained neural network model, and outputs the element deployment plan.

2. The method for deploying emergency communication node elements based on reinforcement learning according to claim 1, characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: When using deep reinforcement learning, the state of the agent is related to the type of element currently requested for deployment. During deployment, the initial deployment element is the on-site command center, followed by the switch. The remaining five elements, satellite vehicles, microwave transmitters, optical terminals, mobile communication vehicles, and shortwave vehicles, are deployed in random order.

3. The method for deploying emergency communication node elements based on reinforcement learning according to claim 1, characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: When using deep reinforcement learning, the state of the agent is related to the geographical environment. When deployed, the elements of the emergency communication network are large vehicles, so the location of the elements is also on both sides of the road. The reward R1 is set to 1 for the road area T, and no reward is set in other areas. When deep reinforcement learning is used, the state of the agent is related to the direction of the disaster. During deployment, in order to reduce the negative impact of the disaster on the elements of the emergency communication network, each element is kept as far away from the disaster center as possible, and the disaster impact range is set to R2; Among them, the distance from the disaster to the element is d.

4. The method for deploying emergency communication node elements based on reinforcement learning according to claim 3 is characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: When using deep reinforcement learning, the state of the agent is related to the communication distance between elements. When deployed, the reward function R3 is as follows: Among them, the distance between two elements is d, and the communication distance interval between elements is [d max ,d min ]; the standard deviation is c.

5. The method for deploying emergency communication node elements based on reinforcement learning according to claim 4 is characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: For the optical transceiver, the reward function R4 is designed as follows: Among them, the distance from the optical terminal to the optical cable connection point is d.

6. The method for deploying emergency communication node elements based on reinforcement learning according to claim 5, characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: When using deep reinforcement learning, the state of the agent is related to the direction of the microwave. When deployed, the electromagnetic wave free space attenuation formula is used to calculate the propagation loss L according to the transmission distance between the microwave ground receiving station and the microwave transmitter deployment location, and it is used as the microwave direction reward R5; Among them, G t ,G r ,C, and f are microwave transmitting and receiving antenna gains, the speed of light, and the carrier frequency, respectively.

7. The method for deploying emergency communication node elements based on reinforcement learning according to claim 6, characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: When using deep reinforcement learning, the state of the agent is related to the coverage. When deployed, the coverage reward R6 is as follows: Among them, IoU is the communication coverage of the element A com Affected area A dis The intersection and ratio of .

8. The method for deploying emergency communication node elements based on reinforcement learning according to claim 7, characterized in that: The state design of the intelligent body in S1 is performed according to different environmental factors and the element type currently requested to be deployed, specifically including: Among them, R j For each reward item, β j It is the correction coefficient corresponding to each reward item.

9. The method for deploying emergency communication node elements based on reinforcement learning according to claim 8, characterized in that: The S2 specifically includes: S21, randomly select H {s i , a i , r i ,s i+1 }, used to train neural networks; S22, samples the action space A with a step size of 2. After sampling, the action space will have n 2 / 4 actions, and then the roads and areas with deployment capabilities are extracted to calculate the value; the action space A contains (n×n) actions; S23, use the target network to calculate s i+1 The best action in the state a max = argmaxQ w (s i+1 , a), and calculate the value of the action under the target network [Q w -(s i+1 , a max )], as the point action a t The value of the action space A' is the maximum value of all actions in the action space A'. max Construct set Q′ t+1 ; S24, using bilinear interpolation, Q′ t+1 The size of becomes back to (n×n); S25, take the action a with the maximum value in the action space A′ t+1max , handed over to the target network for calculation, the target network parameter w - , the value network parameters are w, Q w For the value network, For the target network, get the estimated next step value Q′, and use the temporal difference algorithm to calculate the target value matrix (R+γQ′); S26, quantify the model error by comparing the estimated value matrix Q with the target value matrix (R+γQ′), design the model loss function loss, and use the stochastic gradient descent method to update the value network parameters; Among them, R j For each award item, s t ,a t ,r t are the state, action and reward at time t respectively, and γ is the discount factor.

10. Emergency communication node element deployment system based on reinforcement learning, characterized in that: The system is used to implement the emergency communication node element deployment method based on reinforcement learning as described in any one of claims 1 to 9, comprising: The modeling module is configured based on deep reinforcement learning, using a fully convolutional neural network with a codec architecture as the value network, and designs the agent state according to different environmental factors and the type of elements currently requested for deployment; a training module configured to train a neural network model; The output module is configured to construct a road network map based on the emergency area, input it into the trained neural network model, and output the element deployment plan.

Images