A Joint Optimization Method for Frame Time and Bandwidth Allocation in Emergency Communication Networks

Abstract

This invention discloses a joint optimization method for framing time and bandwidth allocation in emergency communication networks. Under time-varying channel conditions, the method constructs the state space, action space, and reward function of the emergency communication network; it also constructs a dual-depth Q-network, including an online network and a target network; it stores experience samples generated by the interaction between the agent and the emergency communication network environment in an experience replay pool; for each experience sample, based on an experience enhancement extraction mechanism, it calculates the sampling priority and sampling probability, and assigns importance sampling weights; it randomly selects a preset number of experience samples to train the agent, which outputs the optimal action based on the real-time state of the emergency communication network, thus completing the link resource allocation of the emergency communication network. This invention, through real-time interaction between the agent and the environment, autonomously learns the optimal combination strategy for framing time and bandwidth allocation, thereby significantly improving link bandwidth utilization and synergistically reducing data transmission latency under time-varying channel conditions.

Description

Technical Field

[0001] This invention relates to the field of information engineering, and specifically to a method for joint optimization of framing time and bandwidth allocation in emergency communication networks. Background Technology

[0002] In emergency scenarios such as natural disasters, ground communication infrastructure often suffers large-scale damage. Air Base Stations (AirBS), as mobile communication relay nodes, are considered a crucial component in building emergency communication networks due to their flexible deployment and rapid coverage capabilities. In such scenarios, wireless channel conditions are highly time-varying and non-stationary, while the computing resources and energy of airborne platforms are typically limited, posing a significant challenge to the dynamic optimization and allocation of communication resources.

[0003] In emergency communication networks, data needs to be encapsulated into frames before forwarding. Frame generation time and bandwidth allocation are two core parameters affecting system performance, and they are closely coupled: frame generation time directly affects the signaling overhead ratio; too short a time leads to a lower proportion of effective data in the frame, wasting bandwidth, while too long a time increases packet queuing latency. Bandwidth allocation determines the maximum transmission capacity of the channel and also determines the lower limit of frame generation time; exceeding this lower limit increases packet queuing latency. However, existing research mostly optimizes these two parameters in isolation, which has significant limitations. Regarding frame generation time optimization, although some studies have applied algorithms such as near-end policy optimization (PPO) and deep deterministic policy gradient (DDPG) for adjustment, they have not considered the impact of bandwidth changes, resulting in insufficient bandwidth utilization and constrained latency performance. In terms of bandwidth allocation, methods such as Deep Q Network (DQN) and Dueling DQN are widely used, but their strategies are not considered in conjunction with the framing process, making it difficult to coordinate the contradiction between latency and bandwidth utilization, and thus difficult to achieve optimal overall system performance under dynamic channel conditions.

[0004] Deep reinforcement learning, especially DQN and its variants, has provided a solution to this type of sequential decision-making problem. However, DQN suffers from overestimation of Q-values, and its learning efficiency and convergence performance are constrained by the experience replay mechanism of uniform sampling. Furthermore, traditional heuristic optimization approaches, such as the Artificial Fish Swarm Algorithm (AFSA), struggle to optimize frame generation time and bandwidth allocation in real-time within dynamic networks, resulting in insufficient adaptability to emergency communication scenarios. Therefore, it is necessary to propose a joint optimization algorithm for frame generation time and bandwidth allocation that can efficiently explore the environment and exhibits good adaptability in emergency communication scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a joint optimization method for framing time and bandwidth allocation in emergency communication networks. This method aims to solve the bottleneck link resource allocation problem in emergency communication networks under time-varying channel conditions. Addressing the system performance bottleneck caused by the separate optimization of framing time and bandwidth allocation in existing technologies, and the problems of inaccurate Q-value estimation, low learning efficiency, and slow convergence speed in standard deep reinforcement learning algorithms when solving this joint optimization problem, this invention proposes a joint optimization method based on a Double Deep Q-Network (DDQN) and an experience-enhanced extraction mechanism driven by both temporal difference (TD) error and real-time reward. Through real-time interaction between the agent and the environment, the agent autonomously learns the optimal combination strategy for framing time and bandwidth allocation, thereby significantly improving link bandwidth utilization and synergistically reducing data transmission latency under time-varying channel conditions.

[0006] To achieve the above functions, this invention designs a joint optimization method for framing time and bandwidth allocation in emergency communication networks. Under time-varying channel conditions, the agent, based on a dual-deep Q-network and an experience replay pool, executes the following steps S1-S6 to complete the link resource allocation of the emergency communication network:

[0007] Step S1: Construct the state space of the current emergency communication network, including the signal-to-interference-plus-noise ratio, latency, and data transmission rate of the current channel;

[0008] Step S2: Construct the action space of the current emergency communication network, including discrete framing time and discrete bandwidth;

[0009] Step S3: Construct a reward function based on the actual bandwidth utilization of the link;

[0010] Step S4: Construct a dual-depth Q-network, which includes an online network and a target network. The online network takes into account the current state of the emergency communication network and is used for action selection and real-time training updates. The target network is used to calculate the target Q-value and performs periodic soft updates based on the parameters of the online network.

[0011] Step S5: The agent interacts with the emergency communication network environment and stores the generated experience samples in the experience replay pool; for each experience sample, a joint driving function based on the time-series difference error and the current real-time reward is constructed to calculate the sampling priority, sampling probability, and assign importance sampling weights.

[0012] Step S6: When the number of experience samples in the experience replay pool reaches a preset threshold, a preset number of experience samples are randomly selected according to the sampling priority to train the agent, calculate the weighted loss, update the online network parameters through gradient descent, and softly update the target network; after training, the agent outputs the optimal action according to the current state of the emergency communication network to complete the allocation of link resources of the emergency communication network.

[0013] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0014] This invention designs a joint optimization method for framing time and bandwidth allocation in emergency communication networks. By employing an algorithm based on DDQN for joint optimization of framing time and bandwidth allocation, it achieves collaborative optimization of these two strongly coupled parameters within a unified reinforcement learning framework. This overcomes the system performance bottleneck caused by separate optimization. Furthermore, based on an experience enhancement extraction mechanism driven by both temporal differential error and real-time reward, the training process converges rapidly, thereby improving the practicality of the method in large-scale experience scenarios. In addition, this invention is lightweight and can be directly deployed on UAV relay nodes with limited computing resources, providing an efficient and adaptive resource management solution for dynamic and harsh emergency communication environments. It is also applicable to other dynamic wireless networks such as UAV swarms and the Internet of Things. Attached Figure Description

[0015] Figure 1 This is an emergency communication network architecture diagram provided according to an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of a joint optimization method for framing time and bandwidth allocation in an emergency communication network according to an embodiment of the present invention.

[0017] Figure 3 This is a comparison chart of end-to-end latency and bandwidth utilization of different algorithms provided in different traffic intensity scenarios according to embodiments of the present invention;

[0018] Figure 4 This is a comparison chart of end-to-end latency and bandwidth utilization of different algorithms provided in resource-constrained scenarios according to embodiments of the present invention;

[0019] Figure 5 This is a comparison chart of end-to-end latency and bandwidth utilization of different algorithms provided in the embodiments of the present invention under topology change scenarios;

[0020] Figure 6 This is a comparison chart of end-to-end latency, bandwidth utilization, and packet loss rate in an ablation experiment provided according to an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0022] This invention provides a joint optimization method for framing time and bandwidth allocation in emergency communication networks. Under time-varying channel conditions, the agent utilizes a dual-deep Q-network and an experience-enhanced extraction mechanism driven by both temporal differential (TD) error and real-time reward. Figure 1 Perform the following steps S1-S6 to complete the allocation of link resources for the emergency communication network:

[0023] Step S1: Construct the state space of the current emergency communication network, including the signal-to-interference-plus-noise ratio, latency, and data transmission rate of the current channel;

[0024] The state space of the current emergency communication network is as follows:

[0025] ;

[0026] in, This indicates the state of the emergency communication network at time t. The signal-to-interference-plus-noise ratio (SIR) of the channel at time t is used to characterize the channel quality. This represents the time delay at time t. This represents the data transmission rate at time t.

[0027] Step S2: Construct the action space of the current emergency communication network, including discrete framing time and discrete bandwidth;

[0028] The action space is defined as a discrete two-dimensional vector. The action space of the current emergency communication network is as follows:

[0029] ;

[0030] in, This represents the actions of the emergency communication network at time t. This represents the framing time at time t. This represents the bandwidth at time t. Action pairs are constructed using combinations of discrete framing time options and discrete bandwidth allocation options. This indicates that one action pair is selected from the options to be executed.

[0031] In this embodiment, the emergency communication network architecture diagram is referenced. Figure 1It employs a multi-hop wireless mesh topology composed of interconnected air base stations (AirBS). The routing protocol within the network is OSPF, with bottleneck links having a bandwidth of 10Mbps and a latency of 5ms, while access links have a bandwidth of 100Mbps. Each routing node (air base station) acts as a data repeater, equipped with a distributed intelligent agent capable of autonomously optimizing framing time and bandwidth allocation. Specifically, each routing node is responsible for... The system provides data relay services for traffic from distributed transmitting nodes (TN, such as ground sensor equipment, user mobile phones, or other communication devices) and upstream neighboring nodes, forwarding their data through bottleneck links to the next-hop node, and ultimately reaching the rescue command center or the corresponding rescue team. By monitoring the real-time link status, the agent determines the framing time at that node and the bandwidth allocated to the forwarding link.

[0032] Consider four different types of application service streams generated by the sending nodes, representing typical data types in emergency communication: Video streams have high constant or variable bitrates, require large bandwidth, have some tolerance for latency but require continuity, and are of high priority; Image services have large and relatively fixed data packets with random transmission intervals, have some tolerance for latency and do not require continuity, and are of high priority; Voice streams have small data packets and constant rates, but are sensitive to latency and jitter, and are of high priority; Data streams include sensor readings, text information, etc., are usually bursty services, are least sensitive to latency but require high reliability, and are of low priority. Each node is configured with different types and numbers of applications, and its sending rate is determined by the type of application it runs.

[0033] A single routing node manages two First-In-First-Out (FIFO) queues, buffering incoming data packets and primarily performing framing and forwarding operations. When a data packet arrives, data in the higher-priority queue is always processed first. Framing operations can be triggered by two conditions, and execution is performed if either condition is met: (A) Time-triggered condition: the time interval since the last framing operation is greater than the framing time given by the agent; (B) Length-triggered condition: the length of the accumulated data in the queue exceeds the Ethernet standard Maximum Transmission Unit (MTU) of 1500 bytes (framing operations in the lower-priority queue are only triggered by length).

[0034] Step S3: Construct a reward function based on the actual bandwidth utilization of the link;

[0035] The reward function is as follows:

[0036] ;

[0037] in, This represents the reward at time t. This represents the effective data transfer rate excluding padding bytes. Its value is related not only to bandwidth allocation but also to framing time (as framing time decreases, the proportion of effective data in each frame decreases). This represents channel capacity, calculated using Shannon's formula, and its value is determined by bandwidth allocation; the specific calculations are as follows:

[0038] ;

[0039] ;

[0040] in, The time interval for a single statistical analysis. This represents the number of valid bits transmitted within a single statistical period, excluding padding bytes. The bandwidth is allocated to the link based on the link status and past experience. The signal-to-interference-plus-noise ratio (SIR) of a channel measures the strength of the effective signal relative to the sum of background noise and interference from other sources. Channel capacity represents the absolute highest rate achievable under specific channel conditions for error-free transmission. After obtaining the link limit rate, it needs to be converted into a decision about the frame rate applied to the framing process.

[0041] ;

[0042] in, It is a system efficiency factor used to characterize the effective data transmission rate after considering additional overheads such as protocol overhead, retransmission overhead, and contention overhead. The number of bytes in one frame; The frame rate at which data frames are sent refers to the forwarding rate at which forwarding nodes forward data frames based on the allocated bandwidth.

[0043] Step S4: Construct a dual-depth Q-network, which includes an online network and a target network. The online network takes into account the current state of the emergency communication network and is used for action selection and real-time training updates. The target network is used to calculate the target Q-value and performs periodic soft updates based on the parameters of the online network.

[0044] The specific steps of step S4 are as follows:

[0045] Step S4.1: The online network selects the action corresponding to the maximum estimate of the Q-value of the next state as the optimal action, as shown in the following formula:

[0046] ;

[0047] in, This represents the actions of the emergency communication network at time t. Indicates online network, This indicates the state of the emergency communication network at time t. For any combination of actions in the action space, argmax Indicates the maximum value parameter;

[0048] The online network parameters are updated in real time, as shown in the following formula:

[0049] ;

[0050] in, For learning rate, The target Q-value is calculated for the target network. The current Q value calculated for the online network; Indicates online network parameters, For parameters The gradient is used to minimize the mean square error between the predicted and target values ​​using gradient descent, thereby updating the weights of the online network.

[0051] Step S4.2: The target network calculates the target Q-value, which is calculated as follows:

[0052] ;

[0053] in, For the first Rewards for each experience sample As a discount factor, Indicates online network, Indicates the target network. For the first The next state of an empirical sample; Indicates the target network parameters. argmax This represents the maximum value parameter. Compared to frequently updated online networks, using a more stable target network helps prevent Q-value oscillations and divergence, making the learning process smoother. The soft update of the target network is shown in the following formula:

[0054] ;

[0055] in, Update the parameters for the target network.

[0056] Step S5: The agent interacts with the emergency communication network environment and stores the generated experience samples in the experience replay pool; for each experience sample, a joint driving function based on the time-series difference error and the current real-time reward is constructed to calculate the sampling priority, sampling probability, and assign importance sampling weights.

[0057] The specific steps of step S5 are as follows:

[0058] Step S5.1: The agent interacts with the emergency communication network environment to generate experience samples in the form of quadruples. ,in, This indicates the state of the emergency communication network at time t. This represents the actions of the emergency communication network at time t. This represents the reward at time t. The state of the emergency communication network at time t+1 is represented; the experience samples are stored in the experience replay pool in the form of a SumTree.

[0059] Step S5.2: For each experience sample in the experience replay pool, calculate the sampling priority based on the temporal difference (TD) error magnitude and the reward, as follows:

[0060] ;

[0061] in, This indicates the sampling priority of the i-th empirical sample. Let represent the reward for the i-th experience sample. This represents the state of the emergency communication network corresponding to the i-th experience sample. This represents the action of the emergency communication network corresponding to the i-th experience sample. It is a constant used to prevent the sampling priority from being zero;

[0062] Step S5.3: During sampling, non-uniform sampling is performed according to sampling priority. Based on the sampling priority, the sampling probability of each empirical sample is calculated as follows:

[0063] ;

[0064] in, This represents the sampling probability of the i-th empirical sample. The hyperparameter for controlling priority is N, where N is the capacity of the empirical replay pool. To assign priority to the k-th experience sample among all samples in the experience replay pool, based on each sample's... Assign sampling priority based on the proportion in the total;

[0065] Step S5.4: To correct for sampling bias, an importance sampling weight is introduced for each empirical sample, transforming the loss function into a weighted mean square error, as shown in the following formula:

[0066] ;

[0067] in, The importance sampling weight of the i-th empirical sample is represented. The hyperparameters for controlling the correction intensity.

[0068] Step S6: When the number of experience samples in the experience replay pool reaches a preset threshold, a preset number of experience samples are randomly selected according to the sampling priority to train the agent, calculate the weighted loss, update the online network parameters through gradient descent, and softly update the target network; after training, the agent outputs the optimal action according to the current state of the emergency communication network to complete the allocation of link resources of the emergency communication network.

[0069] The training process of the agent in step S6 is as follows:

[0070] Step S6.1: The agent first observes the current status of the emergency communication network. Online networks The -greedy policy selects actions; based on the actions, it sets new framing times and allocates new bandwidth to the link;

[0071] Step S6.2: After executing the action, the emergency communication network calculates the reward value according to the reward function and reports the status of the emergency communication network at the next moment. Construct experience samples and store them in the experience replay pool;

[0072] Step S6.3: When updating the agent, if the number of experience samples in the experience replay pool is greater than the batch size, then a preset number of experience samples are sampled according to the sampling priority. The target Q value and the current Q value of the experience samples are calculated, and the loss is calculated based on the loss function, as shown in the following formula:

[0073] ;

[0074] in, Represents the loss function. Indicates the sample batch size. Indicates the first The importance sampling weight of each empirical sample This represents the target Q-value calculated by the target network. This represents the current Q-value of the online network computation;

[0075] Step S6.4: Perform gradient descent using the Adam optimizer to update the online network parameters. and soft update the target network parameters. Furthermore, during each sampling, the sampling priority of each empirical sample is recalculated, and the priority of each empirical sample in SumTree is updated.

[0076] Step S6.5: Repeat the training until the average reward value within a preset number of consecutive evaluation periods does not exceed the preset threshold; after training is completed, save the final online network parameters.

[0077] The following is an application embodiment of the present invention, and the main parameter settings are shown in Table 1:

[0078] Table 1. Main Parameter Settings

[0079]

[0080] This invention deploys a Joint Frame Time and Bandwidth Allocation Optimization (JFBA) method for emergency communication networks designed in this invention on the AirBS side, implemented based on the PyTorch deep learning framework. Each time slot... The execution order is: monitor the environment → select actions → execute actions → store experience → sample and learn → update network parameters. The specific implementation process has the following conventions:

[0081] (1) State perception and preprocessing: The agent collects the original state in each decision time slot (fixed at 0.1 seconds). , The signal-to-interference-plus-noise ratio (SIR) of the channel is obtained directly from the physical layer; This is obtained by calculating the time difference between the first data packet being enqueued and being successfully acknowledged, reflecting the real-time latency of the system. It is obtained by dividing the amount of effective data successfully transmitted in the previous time slot by the time slot length.

[0082] To achieve discrete control of framing time and bandwidth allocation, a finite joint action space needs to be constructed and reasonably discretized. Framing time minimum value Considering the physical limits, the minimum frame formation time is the minimum value. Considering the maximum frame formation time that business requirements can tolerate; bandwidth allocation minimum value Considering the minimum rate required by business needs, the maximum rate Considering the total bandwidth constraints and physical realizability limitations of the actual link, and considering that the action space can be discretized using both uniform and non-uniform discretization, if the environment is sensitive to a certain interval of the action, the number of discrete points in that interval can be increased (i.e., non-uniform discretization). If the environment's sensitivity to the action does not change significantly, uniform discretization can be used. The total dimension of the action space is the Cartesian product of two discrete action sets.

[0083] (2) Specific architecture and forward propagation of DDQN network: such as Figure 2 As shown, the online network and the target network have the exact same multi-layer neural network structure: the number of neurons in the input layer is [number of states]. The dimensions are 3D; Hidden layer 1 is a fully connected layer with a dimension of 64, using ReLU activation function, followed by a normalization layer to accelerate training convergence; Hidden layer 2 is also a fully connected layer, but with a dimension of 128, also using ReLU activation function, followed by a normalization layer; Output layer: Fully connected layer, with a dimension of joint action space. The dimension size is (25), and linear activation is used to output the Q-value estimate for each joint action. During forward propagation, the state... By sequentially passing through the above levels, a 25-dimensional Q-value vector is finally obtained, which is used to determine the optimal action combination.

[0084] (3) Experience enhancement extraction mechanism driven by TD error and real-time reward: In order to achieve efficient sampling based on priority, a SumTree data structure is used to store data and sample. The time complexity of sampling an experience sample is only a fraction of the time required by this method. N is the capacity of the experience replay pool, which is extremely efficient.

[0085] After each interaction with the environment, the agent stores the experience samples in quadruplets. The sample is stored in the experience replay pool, and its initial priority is set to the highest priority in the current experience replay pool to ensure that each new experience sample is sampled at least once. When the number of samples in the experience replay pool is greater than the batch size (256), a small batch of experience samples is sampled from SumTree according to priority for agent training.

[0086] (4) Model Training and Update: The training phase begins with initialization, copying the online network parameters to the target network and clearing the experience replay pool. During training, the agent first observes the current state of the emergency communication network. Online networks -Greedy strategy selects actions (exploring probabilities in the early stages of training). Initially, the network randomly selects actions to explore the environment; as training progresses, it gradually tends to choose actions deemed optimal by the online network. Subsequently, a new framing time threshold is set based on the action value, and new bandwidth is allocated to the forwarding links. After an action is executed, the environment calculates a reward value according to the reward function and reports the next state. The experience sample is then stored in the experience replay pool.

[0087] During model updates, if the number of experience samples in the experience replay pool is greater than the batch size (256), a small batch of experience samples is sampled from the SumTree according to sampling priority. The target Q-value and the current Q-value of the experience samples are calculated, and the loss is calculated based on the loss function.

[0088] Using the Adam optimizer (learning rate) Perform gradient descent to update the online network parameters. and soft update the target network parameters. This ensures the stability of the target Q-value during training. During each sampling, the sampling priority of each empirical sample is recalculated, and its priority in the SumTree is updated.

[0089] (5) Model Convergence and Deployment: Repeat the training steps until the average reward remains stable over multiple consecutive evaluation periods. After training, save the final online network parameters. In actual deployment, simply load the trained model, and the agent can perform forward propagation based on the real-time state, outputting the optimal joint action to achieve adaptive control. If the system performance changes beyond a certain threshold, model training can be restarted.

[0090] Figure 3 To compare the performance of the proposed method with existing algorithms such as DQN, PPO, DuelingDQN, and AFAS under different traffic intensities, this invention demonstrates superior performance. Regarding end-to-end latency, the proposed method consistently maintains the lowest latency across the entire range of varying traffic intensity, exhibiting a particularly significant advantage under high-load scenarios. Specifically, when the network load is 80 Mbps, the proposed method reduces latency by approximately 35.4% compared to the DQN algorithm and by up to 58.2% compared to the AFSA algorithm. While the AFSA algorithm achieves acceptable latency under light load conditions (20 Mbps), its performance deteriorates rapidly with increasing traffic volume, primarily due to its inability to effectively suppress queue accumulation in mesh topologies. In terms of bandwidth utilization, the proposed method also maintains a consistent advantage, maintaining a bandwidth utilization rate above 92% even under near-saturation conditions. This advantage is even more pronounced under heavy load conditions: the bandwidth utilization improvement of the method of this invention compared to Dueling DQN, PPO, and AFSA algorithms is no less than 11.5%, 18.3%, and 24.6%, respectively. Compared to these methods (which often converge to local optima, leading to a mismatch between framing time and bandwidth allocation), the method of this invention can dynamically coordinate framing time according to the allocated bandwidth, thereby effectively reducing padding overhead. In summary, the method of this invention significantly outperforms heuristic methods and standard deep reinforcement learning (DRL) algorithms in terms of congestion control capabilities. Even under overload conditions, the method of this invention can still guarantee a latency of less than 100ms and achieve near-optimal bandwidth utilization (95.36%), thus strongly verifying its robustness under the inherent dynamic and bursty service modes of emergency communication networks.

[0091] like Figure 4The figure shows a performance comparison between the method proposed in this invention and the DQN algorithm, PPO algorithm, DuelingDQN algorithm and AFSA algorithm in a resource-constrained scenario, wherein resources are constrained by adjusting the proportion of available bandwidth. Figure 4 The latency results shown demonstrate that the method of this invention exhibits a significant advantage across the entire available bandwidth range, and the performance gap between algorithms widens significantly as resources become increasingly scarce. For example, under the most stringent constraints (available bandwidth ratio of only 20%), the method of this invention is the only algorithm capable of maintaining latency below the 100ms threshold. Compared to its closest competitor, DuelingDQN (204.35ms), the method of this invention reduces latency by 53.39%; compared to the worst-performing DQN algorithm (1535.24ms) in this scenario, the latency reduction is as high as 93.79%. In addition to its low latency advantage, the method of this invention also demonstrates excellent bandwidth utilization. Even when the network approaches saturation (available bandwidth ratio of 40%), the method of this invention can still maintain bandwidth utilization stably above 97%. Under the more extreme condition of 20%, this advantage is further manifested as a significant improvement in bandwidth utilization: the bandwidth utilization of the method described in this invention is improved by 7.99%, 9.80%, and 22.90% compared to the Dueling DQN, PPO, and DQN algorithms, respectively. Overall, the simulation results above collectively verify that in resource-constrained environments, the method described in this invention can effectively alleviate and, to some extent, "decouple" the inherent trade-off between end-to-end latency and bandwidth utilization.

[0092] like Figure 5The diagram shows a performance comparison between the proposed method and the DQN, PPO, Dueling DQN, and AFSA algorithms in a topology-changing scenario, where the network topology is altered by randomly destroying several links in the core network. As the network topology becomes fragmented, service flows are forced to converge onto more concentrated and longer transmission paths, inevitably increasing network latency for all algorithms. However, the proposed method still exhibits outstanding performance. In the most severe scenario of three links being damaged, the latency performance of traditional algorithms deteriorates significantly: the DQN algorithm suffers from an extremely poor latency of 1030.27ms due to its poor learning efficiency and inability to correctly predict the optimal action; similarly, the AFSA and PPO algorithms reach latencies of 380.15ms and 320.46ms respectively, both exceeding QoS constraint thresholds. In stark contrast, the proposed method stabilizes the latency at 94.34ms, remaining the only algorithm meeting the latency requirement of less than 100ms even under severely damaged infrastructure conditions. Regarding bandwidth utilization efficiency, as links are severed, the remaining paths in the network gradually evolve into critical bottlenecks. When three links fail, the bandwidth utilization rate of the method in this invention reaches 97.50%, indicating that it can fully utilize the remaining network capacity without causing system collapse. Compared with the DQN (55.10%) and PPO (76.80%) algorithms, this result represents significant improvements of 42.4% and 20.7%, respectively. This robustness essentially benefits from an experience-enhanced extraction mechanism driven by both temporal difference (TD) error and real-time reward: topological mutations trigger a sharp increase in TD error, and the experience-enhanced extraction mechanism enables the agent to quickly learn from these critical transitions, thereby adjusting its policy more agilely and achieving adaptive optimization compared to baseline methods.

[0093] like Figure 6The figure shows the system performance after removing the experience-enhanced extraction mechanism driven by both TD error and real-time reward, or decoupling optimization of a single action. Compared to various ablation variants, the complete method of this invention achieves the best overall performance across all evaluation metrics, and quantitative analysis clearly reveals the specific contributions of each module. When the dynamic bandwidth allocation mechanism is disabled, the end-to-end latency surges to 210.56 ms, and the packet loss rate rises to 8.43%. This performance degradation highlights the crucial role of flexible bandwidth provisioning in suppressing buffer overflows and maintaining basic connectivity under high load conditions. Conversely, when the framing time is fixed, although the bandwidth utilization remains relatively high (93.86%), the latency rises significantly to 132.45 ms. This result verifies the necessity of adaptive framing time in reducing packet encapsulation latency and optimizing the trade-off between signaling overhead and transmission efficiency. Furthermore, removing the experience-enhanced extraction mechanism (uniform sampling, w / o PER) results in a 73.5% increase in latency, from 49.39 ms to 85.68 ms, while the packet loss rate also increases to 6.81%. This degradation phenomenon demonstrates that empirical enhancement extraction mechanisms are crucial for capturing sparse but critical sample features (characterized by large TD errors), thereby significantly accelerating convergence and improving robustness in dynamic environments. In summary, the superior performance of the method in this invention relies on the collaborative integration of multiple components, rather than the isolated operation of any single module.

[0094] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for jointly optimizing framing time and bandwidth allocation in an emergency communication network, characterized in that, Under time-varying channel conditions, the agent, based on a dual-deep Q-network and an experience replay pool, executes the following steps S1-S6 to complete the allocation of link resources for the emergency communication network: Step S1: Construct the state space of the current emergency communication network, including the signal-to-interference-plus-noise ratio, latency, and data transmission rate of the current channel; Step S2: Construct the action space of the current emergency communication network, including discrete framing time and discrete bandwidth; Step S3: Construct a reward function based on the actual bandwidth utilization of the link; The reward function constructed in step S3 is as follows: ； in, This represents the reward at time t. This indicates the effective data transfer rate excluding padding bytes. The channel capacity is represented and calculated as follows: ； ； in, For a single statistical time interval, This represents the number of valid bits transmitted within a single statistical period, excluding padding bytes. The allocated bandwidth; The signal-to-interference-plus-noise ratio (SINR) of the channel; Step S4: Construct a dual-depth Q-network, which includes an online network and a target network. The online network takes into account the current state of the emergency communication network and is used for action selection and real-time training updates. The target network is used to calculate the target Q-value and performs periodic soft updates based on the parameters of the online network. Step S5: The agent interacts with the emergency communication network environment and stores the generated experience samples in the experience replay pool; for each experience sample, a joint driving function based on the time-series difference error and the current real-time reward is constructed to calculate the sampling priority, sampling probability, and assign importance sampling weights. Step S6: When the number of experience samples in the experience replay pool reaches a preset threshold, a preset number of experience samples are randomly selected according to the sampling priority to train the agent, calculate the weighted loss, update the online network parameters through gradient descent, and softly update the target network; after training, the agent outputs the optimal action according to the current state of the emergency communication network to complete the allocation of link resources of the emergency communication network.

2. The method for joint optimization of framing time and bandwidth allocation in an emergency communication network according to claim 1, characterized in that, The state space of the current emergency communication network constructed in step S1 is as follows: ； in, This indicates the state of the emergency communication network at time t. This represents the signal-to-interference-plus-noise ratio (SIR) of the channel at time t. This represents the time delay at time t. This represents the data transmission rate at time t.

3. The method for joint optimization of framing time and bandwidth allocation in an emergency communication network according to claim 2, characterized in that, The action space of the current emergency communication network constructed in step S2 is as follows: ； in, This represents the actions of the emergency communication network at time t. This represents the framing time at time t. This represents the bandwidth at time t.

4. The method for joint optimization of framing time and bandwidth allocation in an emergency communication network according to claim 3, characterized in that, The specific steps of step S4 are as follows: Step S4.1: The online network selects the action corresponding to the maximum estimate of the Q-value of the next state as the optimal action, as shown in the following formula: ； in, This represents the actions of the emergency communication network at time t. Indicates online network, This indicates the state of the emergency communication network at time t. For any combination of actions in the action space, argmax Indicates the maximum value parameter; Real-time updates of online network parameters are shown in the following formula: ； in, For learning rate, The target Q-value is calculated for the target network. The current Q value calculated for the online network; Indicates online network parameters, For information about online network parameters The gradient is used to minimize the mean square error between the predicted and target values ​​using gradient descent, thereby updating the weights of the online network. Step S4.2: The target network calculates the target Q-value. The target Q-value is calculated as follows: ； in, For the first Rewards for each experience sample As a discount factor, Indicates online network, Indicates the target network. For the first The next state of an empirical sample. argmax Indicates the maximum value parameter; Perform a soft update on the target network, as shown in the following formula: ； in, Update parameters for the target network. This represents the target network parameters.

5. The method for joint optimization of framing time and bandwidth allocation in an emergency communication network according to claim 4, characterized in that, The specific steps of step S5 are as follows: Step S5.1: The intelligent agent interacts with the emergency communication network environment to generate experience samples. ,in, This indicates the state of the emergency communication network at time t. This represents the actions of the emergency communication network at time t. This represents the reward at time t. This indicates the state of the emergency communication network at time t+1; experience samples are stored in the experience replay pool. Step S5.2: For each experience sample in the experience replay pool, the sampling priority is calculated based on the experience enhancement extraction mechanism driven by both temporal difference error and real-time reward, as follows: ； in, Indicates the first Sampling priority of an empirical sample Indicates the first Rewards for each experience sample Indicates the first The status of the emergency communication network corresponding to each experience sample. Indicates the first The actions of the emergency communication network corresponding to each experience sample It is a constant; Step S5.3: Based on the sampling priority, calculate the sampling probability of each empirical sample as follows: ； in, Indicates the first The sampling probability of an empirical sample. The hyperparameter for controlling priority is N, where N is the capacity of the empirical replay pool. The first of all experience samples in the experience replay pool Priority of each empirical sample; Step S5.4: For each empirical sample, calculate the importance sampling weight, as shown in the following formula: ； in, Indicates the first The importance sampling weight of each empirical sample The hyperparameters used to control the correction intensity.

6. The method for joint optimization of framing time and bandwidth allocation in an emergency communication network according to claim 5, characterized in that, The training process of the agent in step S6 is as follows: Step S6.1: The agent first observes the current status of the emergency communication network. Online networks The -greedy policy selects actions; based on the actions, it sets new framing times and allocates new bandwidth to the link; Step S6.2: After executing the action, the emergency communication network calculates the reward value according to the reward function and reports the status of the emergency communication network at the next moment. Construct experience samples and store them in the experience replay pool; Step S6.3: When updating the agent, if the number of experience samples in the experience replay pool is greater than the sampling batch size, then according to the sampling priority, a preset number of experience samples are sampled, the target Q value and the current Q value of the experience samples are calculated, and the loss is calculated based on the loss function, as shown in the following formula: ； in, Represents the loss function. Indicates the sample batch size. The importance sampling weight of the i-th empirical sample is represented. This represents the target Q-value calculated by the target network. This represents the current Q-value of the online network computation; Step S6.4: Perform gradient descent using the Adam optimizer to update the online network parameters. and soft update the target network parameters. Furthermore, the sampling priority of each empirical sample is recalculated for each sampling. Step S6.5: Repeat the training until the average reward value within a preset number of consecutive evaluation periods does not exceed the preset threshold; after training is completed, save the final online network parameters.

Images