An emergency management communication multi-network aggregation transmission optimization method and system based on reinforcement learning
By optimizing emergency management communication using a reinforcement learning-based approach, the problem of communication inconvenience in emergency management scenarios using existing multipath transmission schemes is solved, and an efficient data distribution strategy and improved transmission performance are achieved.
Patent Information
- Application Number
- CN202611186196.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-06
- Publication Date
- 2026-11-06
AI Technical Summary
Existing multipath transmission schemes are unable to detect the priority differences of service types in emergency management scenarios, and cannot dynamically optimize link bandwidth, latency and load changes, resulting in communication inconvenience.
By employing a reinforcement learning-based approach, the real-time status of heterogeneous communication links and the service attribute information of data streams are obtained. The data distribution strategy is optimized using a deep deterministic policy gradient algorithm, and the model is updated in conjunction with transmission quality feedback, thereby achieving priority scheduling of high-priority data packets and link weight allocation.
It enables adaptive scheduling of dynamic link changes and service priorities in emergency management scenarios, improving transmission performance, ensuring high throughput, low latency and load balancing, and adapting to the needs of different emergency tasks.
Smart Images
Figure CN123055648A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication transmission optimization technology, and in particular to a method and system for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning. Background Technology
[0002] In emergency management scenarios such as natural disaster relief, security for large-scale events, and remote medical monitoring, multiple types of business data typically need to be transmitted simultaneously on-site, such as command voice, real-time video, sensor monitoring data, and documents. Different services have significantly different requirements for network bandwidth, latency, and reliability: voice and real-time control commands are extremely sensitive to latency, high-definition video requires continuous high bandwidth, while file transmission focuses more on overall throughput. To ensure communication reliability, emergency sites are generally equipped with multiple heterogeneous network links such as satellite communication, 4G / 5G public networks, self-organizing networks, and dedicated core networks, hoping to improve overall transmission capacity and robustness through multi-network aggregation.
[0003] However, in existing multipath transmission schemes, such as Multipath TCP (MPTCP), although data streams can be split and transmitted across multiple paths, their default schedulers (such as minimum RTT priority) rely solely on one-way latency or packet loss rate for simple path selection. This makes it difficult to perceive the priority differences of service types and to perform fine-grained optimization for dynamic changes in bandwidth, latency, and load of each link. Static scheduling strategies such as weighted round-robin require manual pre-configuration of weights, making it difficult to adapt to rapid fluctuations in link quality. Furthermore, while some dynamic scheduling methods based on heuristic rules (such as allocating traffic according to the proportion of available bandwidth on the link) have made some improvements, they still struggle to simultaneously meet the low latency requirements of high-priority flows and the fairness of low-priority flows when faced with complex and ever-changing network conditions and the coexistence of multiple priority services. All of these factors cause numerous inconveniences to communication in emergency management scenarios. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning, in order to solve the problem that in the existing technology, communication methods in emergency management scenarios mostly adopt multi-path transmission schemes, and most existing multi-path transmission schemes have many limitations, causing many inconveniences to communication in emergency management scenarios.
[0005] To achieve the above objectives, this invention proposes a multi-network aggregation transmission optimization method for emergency management communication based on reinforcement learning, comprising the following steps: Acquire real-time status information of the heterogeneous communication link between the sending and receiving ends, as well as the service attribute information of the data stream to be transmitted; The real-time status information and the business attribute information are used as inputs to the reinforcement learning decision model, which then outputs a data distribution strategy for the heterogeneous communication link based on the current environmental state. Based on the data distribution strategy and in combination with the priority level of each data packet in the data stream to be transmitted, the high-priority data packets are preferentially scheduled to the link with the lowest round-trip latency for transmission. The transmitting end receives transmission quality feedback information from the receiving end, calculates an incentive signal based on the transmission quality feedback information, and feeds the incentive signal back to the reinforcement learning decision model to drive the reinforcement learning decision model to update the policy, thereby optimizing the subsequent data distribution strategy.
[0006] Preferably, the real-time status information includes at least one of the following parameters: available bandwidth, round-trip time, packet loss rate, and current sending queue length; The service attribute information includes at least one of the following parameters: the service priority level of the data packet, the size of the data packet, and the data stream's lifetime.
[0007] Preferably, the step of using the real-time state information and the service attribute information as input to the reinforcement learning decision model, and having the reinforcement learning decision model output a data distribution strategy for the heterogeneous communication link based on the current environment state, specifically involves: encoding the real-time state information and the service attribute information into a state vector as input to the reinforcement learning decision model; the reinforcement learning decision model outputting a transmission weight vector corresponding to the heterogeneous communication link; allocating transmission quotas to the corresponding heterogeneous communication link based on the transmission weight vector; and prioritizing the scheduling of high-priority data packets within the transmission quota of the heterogeneous communication link for transmission through the link with the lowest current transmission latency. The transmission weight vectors in the multiple heterogeneous communication links are all non-negative real numbers, and the sum of the transmission weight vectors corresponding to the multiple heterogeneous communication links is 1; The reinforcement learning model employs a deep deterministic policy gradient algorithm, comprising a policy network and a value network. The policy network takes a state vector as input and outputs deterministic path weights through a Softmax layer. The value network evaluates the Q-value corresponding to the state-action sequence to assist in the training of the reinforcement learning decision model.
[0008] Preferably, the transmitting end receives transmission quality feedback information from the receiving end, and the calculation of the incentive signal based on the transmission quality feedback information specifically involves: the incentive signal being a composite reward value R, which is obtained by weighted combination of the normalized throughput component Throughput_norm, the delay component AvgDelay_norm, the priority violation component VioCount_norm, and the load balancing component Balance_norm; The throughput component is: Throughput is the total throughput of successful transmissions, and T_max is the sum of the available bandwidth of each path; The time delay component is: AvgDelay is the average delay of all currently transmitted data streams from the sender to the receiver, and D_max is the maximum tolerable delay of the high-priority data packets. The priority violation component is: VioCount is the number of times the high-priority data packet exceeds the maximum delay. The total number of high-priority data packets sent during this period; The load balancing component is: , This represents the standard deviation of bandwidth utilization for each link within this period. To determine the theoretical maximum standard deviation of bandwidth utilization for each link within this period, we can take the standard deviation when each path's capacity is half fully loaded and half unloaded. The compound reward value R is: Wherein, α, β, γ, and δ are preset adjustable weight coefficients.
[0009] Preferably, the reinforcement learning decision model adopts a deployment method that combines offline pre-training and online fine-tuning, specifically: In the offline stage, the policy network and the value network are pre-trained to convergence in a simulation environment; in the online stage, the policy network outputs the data distribution policy in a deterministic manner without introducing random exploration noise. At the same time, the transfer samples generated during the online operation are continuously stored in the experience replay pool, and the policy network and the value network are updated online with a learning rate lower than that in the offline stage. Gradient pruning is performed during the update process to constrain the magnitude of parameter changes.
[0010] Preferably, the step of feeding back the stimulus signal to the reinforcement learning decision model to drive the reinforcement learning decision model to perform policy updates in order to optimize the subsequent data distribution strategy specifically involves: storing the current real-time state information, the data distribution strategy, the stimulus signal, and the state information of the next cycle as transfer samples in the experience replay pool, and sampling a small batch of transfer samples from the experience replay pool to update the parameters of the reinforcement learning decision model.
[0011] Preferably, when the receiving end aggregates the data packets from each of the heterogeneous communication links, it maintains a reordering buffer for each data stream and sets a dynamic timeout timer. The timeout threshold of the dynamic timeout timer is dynamically set according to the maximum round-trip delay value among all currently active heterogeneous communication links.
[0012] In another aspect of the present invention, a multi-network aggregation transmission optimization system for emergency management communication based on reinforcement learning is proposed, comprising: A multi-network interface module, configured to support access to at least two heterogeneous communication networks; The network status awareness module is configured to collect real-time status information of each heterogeneous communication link; The multi-path aggregation proxy module is deployed on the sending end and configured to receive data streams from upper-layer applications, identify the priority level of data packets, and perform data packet distribution operations according to the data distribution strategy output by the reinforcement learning decision engine. A multipath aggregation module, deployed at the receiving end, is configured to aggregate and reassemble the data packets arriving from each link and generate transmission quality feedback information. The reinforcement learning decision engine includes a reinforcement learning decision model, configured to output the data distribution strategy based on the real-time state information provided by the network state awareness module, and to calculate the incentive signal based on the transmission quality feedback information to update itself.
[0013] Preferably, the multi-path aggregation proxy module is implemented based on a user-space protocol stack and identifies the service priority level of the data packet by distinguishing between service tags or application layer identifiers; The multipath aggregation module maintains a reordering buffer and a dynamic timeout timer to reorder out-of-order data packets and determine packet loss for data packets that have not arrived within the timeout period.
[0014] Preferably, the heterogeneous communication network includes at least two of the following: satellite communication network, 4G / 5G cellular network, and dedicated core network; The transmitting end and the receiving end establish multiple parallel transmission paths through multiple heterogeneous communication links to achieve multi-network aggregated transmission.
[0015] Compared with existing technologies, this invention has at least the following advantages: It introduces deep reinforcement learning into multi-network aggregation transmission scheduling for emergency management, enabling simultaneous perception of dynamic changes in links and service priority characteristics. This achieves superior overall transmission performance compared to static scheduling and traditional heuristic scheduling. Furthermore, the constructed composite reward function achieves adaptive trade-offs among multiple mutually constraining objectives such as high throughput, low latency, priority guarantee, and load balancing, with adjustable coefficients to adapt to the differentiated needs of various emergency tasks. The reinforcement learning decision model employs the DDPG algorithm and an experience replay pool, utilizing a deployment method combining offline pre-training and online fine-tuning to enable the system to continuously adapt to environmental changes. Moreover, for upper-layer applications, the system transparently implements multi-path aggregation between the sending and receiving ends, shielding the underlying multi-path details and facilitating integration with existing emergency communication equipment, thus providing numerous conveniences for communication in emergency management scenarios. Attached Figure Description
[0016] Figure 1 This is a schematic diagram illustrating the steps of a multi-network aggregation transmission optimization method for emergency management communication based on reinforcement learning, as described in this invention. Figure 2 This is a schematic diagram of a multi-network aggregation transmission optimization system module for emergency management communication based on reinforcement learning, according to the present invention. Figure 3 This is a schematic diagram of the working and training process of the reinforcement learning decision model in this invention. Detailed Implementation
[0017] The specific embodiments of the present invention will be described in detail below, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.
[0018] Example 1 like Figure 1 As shown in the figure, this embodiment proposes a multi-network aggregation transmission optimization method for emergency management communication based on reinforcement learning, including the following steps: It acquires real-time status information of the heterogeneous communication link between the sending and receiving ends, as well as the service attribute information of the data stream to be transmitted.
[0019] The real-time status information and the business attribute information are used as inputs to the reinforcement learning decision model, which then outputs a data distribution strategy for the heterogeneous communication link based on the current environmental state.
[0020] Based on the data distribution strategy and the priority level of each data packet in the data stream to be transmitted, the high-priority data packets are preferentially scheduled to the link with the lowest round-trip latency for transmission.
[0021] The transmitting end receives transmission quality feedback information from the receiving end, calculates an incentive signal based on the transmission quality feedback information, and feeds the incentive signal back to the reinforcement learning decision model to drive the reinforcement learning decision model to update the policy, thereby optimizing the subsequent data distribution strategy.
[0022] In this embodiment, the real-time status information includes at least one of the following parameters: available bandwidth, round-trip time, packet loss rate, and current sending queue length.
[0023] The service attribute information includes at least one of the following parameters: the service priority level of the data packet, the size of the data packet, and the data stream's lifetime.
[0024] In this embodiment, the step of using the real-time state information and the service attribute information as input to the reinforcement learning decision model, and having the reinforcement learning decision model output a data distribution strategy for the heterogeneous communication link based on the current environment state, specifically involves: encoding the real-time state information and the service attribute information into a state vector as input to the reinforcement learning decision model; the reinforcement learning decision model outputting a transmission weight vector corresponding to the heterogeneous communication link; allocating transmission quotas to the corresponding heterogeneous communication link based on the transmission weight vector; and prioritizing the scheduling of high-priority data packets within the transmission quota of the heterogeneous communication link for transmission through the link with the lowest current transmission latency.
[0025] The transmission weight vectors in the multiple heterogeneous communication links are all non-negative real numbers, and the sum of the transmission weight vectors corresponding to the multiple heterogeneous communication links is 1.
[0026] The reinforcement learning model employs a deep deterministic policy gradient algorithm, comprising a policy network and a value network. The policy network takes a state vector as input and outputs deterministic path weights through a Softmax layer. The value network evaluates the Q-value corresponding to the state-action sequence to assist in the training of the reinforcement learning decision model.
[0027] In this embodiment, the transmitting end receives transmission quality feedback information from the receiving end, and the incentive signal is calculated based on the transmission quality feedback information as follows: the incentive signal is a composite reward value R, which is obtained by weighted combination calculation of the normalized throughput component Throughput_norm, delay component AvgDelay_norm, priority violation component VioCount_norm, and load balancing component Balance_norm.
[0028] The throughput component is: Throughput is the total throughput of successful transmissions, and T_max is the sum of the available bandwidth of each path.
[0029] The time delay component is: AvgDelay is the average delay of all currently transmitted data streams from the sender to the receiver, and D_max is the maximum tolerable delay for high-priority data packets.
[0030] The priority violation component is: VioCount is the number of times the high-priority data packet exceeds the maximum delay. This represents the total number of high-priority data packets sent during this period.
[0031] The load balancing component is: , This represents the standard deviation of bandwidth utilization for each link within this period. To determine the theoretical maximum standard deviation of bandwidth utilization for each link within this period, we can take the standard deviation when each path's capacity is half full and half empty.
[0032] The compound reward value R is: Wherein, α, β, γ, and δ are preset adjustable weight coefficients.
[0033] In this embodiment, the reinforcement learning decision model adopts a deployment method combining offline pre-training and online fine-tuning. Specifically: In the offline phase, the policy network and the value network are pre-trained to convergence in a simulation environment. In the online phase, the policy network outputs the data distribution policy in a deterministic manner, without introducing random exploration noise. Simultaneously, the transition samples generated during online operation are continuously stored in the experience replay pool, and the policy network and the value network are updated online with a learning rate lower than that in the offline phase. Gradient pruning is performed during the update process to constrain the magnitude of parameter changes.
[0034] In this embodiment, feeding back the stimulus signal to the reinforcement learning decision model and driving the reinforcement learning decision model to update the policy in order to optimize the subsequent data distribution strategy specifically involves: storing the current real-time state information, the data distribution strategy, the stimulus signal, and the state information of the next cycle as transfer samples in the experience replay pool, and sampling a small batch of transfer samples from the experience replay pool to update the parameters of the reinforcement learning decision model.
[0035] In this embodiment, when the receiving end aggregates the data packets from each of the heterogeneous communication links, it maintains a reordering buffer for each data stream and sets a dynamic timeout timer. The timeout threshold of the dynamic timeout timer is dynamically set according to the maximum round-trip delay value among all currently active heterogeneous communication links.
[0036] Example 2 like Figure 2 and Figure 3 As shown, based on Embodiment 1, this embodiment proposes an emergency management communication multi-network aggregation transmission optimization system based on reinforcement learning, including: The multi-network interface module is configured to support access to at least two heterogeneous communication networks.
[0037] The network status awareness module is configured to collect real-time status information of each heterogeneous communication link.
[0038] The multi-path aggregation proxy module is deployed on the sending end and configured to receive data streams from upper-layer applications, identify the priority level of data packets, and perform data packet distribution operations according to the data distribution strategy output by the reinforcement learning decision engine.
[0039] The multipath aggregation module, deployed at the receiving end, is configured to aggregate and reassemble the data packets arriving from each link and generate transmission quality feedback information.
[0040] The reinforcement learning decision engine includes a reinforcement learning decision model, configured to output the data distribution strategy based on the real-time state information provided by the network state awareness module, and to calculate the incentive signal based on the transmission quality feedback information to update itself.
[0041] In this embodiment, the multi-path aggregation proxy module is implemented based on the user-space protocol stack and identifies the service priority level of the data packet by distinguishing service tags or application layer identifiers.
[0042] In one embodiment, the multi-path aggregation proxy module is implemented based on the user-space protocol stack and bypasses the kernel protocol stack through DPDK technology to reduce packet processing latency.
[0043] The multipath aggregation module maintains a reordering buffer and a dynamic timeout timer to reorder out-of-order data packets and determine packet loss for data packets that have not arrived within the timeout period.
[0044] Specifically, the multi-path aggregation module maintains a reordering buffer and a dynamic timeout timer to reorder out-of-order data packets and determine packet loss for data packets that have not arrived within the timeout period, ensuring that upper-layer applications obtain an orderly and reliable data stream.
[0045] In this embodiment, the heterogeneous communication network includes at least two of the following: satellite communication network, 4G / 5G cellular network, and dedicated core network.
[0046] The transmitting end and the receiving end establish multiple parallel transmission paths through multiple heterogeneous communication links to achieve multi-network aggregated transmission.
[0047] To facilitate understanding of the embodiments of this solution by those skilled in the art, the working principle of this solution will now be briefly explained in conjunction with specific application scenarios: Upon system power-on, the reinforcement learning model, pre-trained and converged in the offline simulation environment, is loaded into the decision engine. All network interfaces are initialized, and a reliable control signaling channel is established with the receiver. After system startup, the network state awareness module performs state acquisition actions at a preset fixed period (e.g., 100ms). First, it acquires the real-time state information of the heterogeneous communication links between the sender and receiver. In one specific implementation, for each active link i, the real-time state information includes: available bandwidth (Mbps), round-trip time (ms), packet loss rate (percentage), and current transmission queue length (bytes). Simultaneously, the awareness module also extracts the service attribute information of the data stream to be transmitted from the upper-layer application's transmission queue. This includes, but is not limited to: the service priority level P of the data packet (divided into high priority EF, medium priority AF, and low priority BE), the data packet size (bytes), and the data stream's lifetime (ms). All the parameters acquired are aggregated and encoded into a multi-dimensional state vector s, which serves as the input for subsequent decision-making.
[0048] The generated state vector s is used as the input to the reinforcement learning decision model, which employs the Deep Deterministic Policy Gradient (DDPG) algorithm. This model internally includes a policy network (Actor network) and a value network (Critic network). The policy network takes the state vector s as input and, after a series of nonlinear transformations through fully connected hidden layers, finally outputs an action decision for multiple heterogeneous communication links—that is, a data distribution policy. This data distribution policy is specifically represented by a transmission weight vector W = [w_1, w_2, ..., w_K], where K is the number of currently available links, and w_i are non-negative real numbers whose sum is 1.
[0049] The multi-path aggregation proxy module receives the output transmission weight vector W. First, it allocates a transmission quota for the current period to each link according to the weight ratio of each link (i.e., the maximum amount of data that the link is allowed to send in this period). After completing the quota allocation, the aggregation proxy module scans the data packets in the queue to be sent and executes the priority-aware scheduling logic: High-priority packets are processed first: The system checks all high-priority packets (such as voice / command packets) and schedules them to be sent in the remaining quota of the link with the lowest current round-trip time (RTT).
[0050] Gradual degrading filling: After the highest priority packet has been processed or the lowest latency link quota has been exhausted, the next highest priority packets are processed in turn and allocated to the remaining quota of the next lowest latency link, and so on.
[0051] Through the above scheduling mechanism, this invention ensures that, at the macro level, the weight allocation of each link is satisfied, while at the micro level, packet-level scheduling, the most critical services are provided with optimal latency transmission guarantee.
[0052] Data packets are transmitted to the receiving end via various heterogeneous links. After receiving the data packets, the multipath aggregation module at the receiving end maintains a reordering buffer and reorders out-of-order data packets according to the globally incrementing sequence number carried by the data packets. At the same time, the aggregation module sets a dynamic timeout timer for each data stream. The timeout threshold of this timer is dynamically set according to the maximum round-trip time (RTT) value among all currently active links (e.g., set to 1.5 times the maximum RTT). If a data packet is not received after the timer expires, it is determined that the packet is lost. The receiving end aggregates the data reception status of each link in this period (including successful reception confirmation, packet delay statistics, packet loss marking, etc.) to generate transmission quality feedback information and sends it back to the sending end through the control channel.
[0053] After receiving the transmission quality feedback information from the receiver, the sending end calculates a composite reward value RR as an incentive signal based on the feedback information. Specifically, the composite reward value R is calculated by weighting and combining the normalized throughput component Throughput_norm, delay component AvgDelay_norm, priority violation component VioCount_norm, and load balancing component Balance_norm. The throughput component is as follows: Throughput is the total throughput of successful transmissions, and T_max is the sum of the available bandwidth of each path.
[0054] The time delay component is: AvgDelay is the average delay of all currently transmitted data streams from the sender to the receiver, and D_max is the maximum tolerable delay for high-priority data packets.
[0055] Priority violation components are: VioCount represents the number of times a high-priority data packet exceeds the maximum latency. This represents the total number of high-priority data packets sent during this period.
[0056] The load balancing components are: , This represents the standard deviation of bandwidth utilization for each link within this period. To determine the theoretical maximum standard deviation of bandwidth utilization for each link within this period, we can take the standard deviation when each path's capacity is half full and half empty.
[0057] The compound reward value R is: Among them, α, β, γ, and δ are preset adjustable weight coefficients, which are used to flexibly configure according to different emergency task priorities (such as prioritizing throughput or prioritizing latency).
[0058] Subsequently, the reinforcement learning decision engine stores the real-time state information of the current period, the data distribution strategy, the incentive signal, and the real-time state of the next period as a transition sample in the experience replay pool. When the number of samples in the pool meets the preset threshold, the system randomly samples a small batch of samples from the pool and adopts the standard update process of the DDPG algorithm, that is, updates the value network by minimizing the temporal difference (TD) error and updates the policy network along the Q-value gradient rising direction indicated by the value network, thereby completing the policy optimization of the reinforcement learning decision model, enabling it to generate better data distribution strategies in subsequent periods.
[0059] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning, characterized in that, Includes the following steps: Acquire real-time status information of the heterogeneous communication link between the sending and receiving ends, as well as the service attribute information of the data stream to be transmitted; The real-time status information and the business attribute information are used as inputs to the reinforcement learning decision model, which then outputs a data distribution strategy for the heterogeneous communication link based on the current environmental state. Based on the data distribution strategy and in combination with the priority level of each data packet in the data stream to be transmitted, the high-priority data packets are preferentially scheduled to the link with the lowest round-trip latency for transmission. The transmitting end receives transmission quality feedback information from the receiving end, calculates an incentive signal based on the transmission quality feedback information, and feeds the incentive signal back to the reinforcement learning decision model to drive the reinforcement learning decision model to update the policy, thereby optimizing the subsequent data distribution strategy.
2. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 1, characterized in that, The real-time status information includes at least one of the following parameters: available bandwidth, round-trip time, packet loss rate, and current sending queue length; The service attribute information includes at least one of the following parameters: the service priority level of the data packet, the size of the data packet, and the data stream's lifetime.
3. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 1, characterized in that, The step of using the real-time state information and the service attribute information as input to the reinforcement learning decision model, and having the reinforcement learning decision model output a data distribution strategy for the heterogeneous communication link based on the current environment state, specifically involves: encoding the real-time state information and the service attribute information into a state vector as input to the reinforcement learning decision model; the reinforcement learning decision model outputting a transmission weight vector corresponding to the heterogeneous communication link; allocating transmission quotas to the corresponding heterogeneous communication link based on the transmission weight vector; and prioritizing the scheduling of high-priority data packets within the transmission quota of the heterogeneous communication link for transmission through the link with the lowest current transmission latency. The transmission weight vectors in the multiple heterogeneous communication links are all non-negative real numbers, and the sum of the transmission weight vectors corresponding to the multiple heterogeneous communication links is 1; The reinforcement learning model employs a deep deterministic policy gradient algorithm, comprising a policy network and a value network. The policy network takes a state vector as input and outputs deterministic path weights through a Softmax layer. The value network evaluates the Q-value corresponding to the state-action sequence to assist in the training of the reinforcement learning decision model.
4. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 1, characterized in that, The transmitting end receives transmission quality feedback information from the receiving end, and calculates the incentive signal based on the transmission quality feedback information as follows: the incentive signal is a composite reward value R, which is obtained by weighting and combining the normalized throughput component Throughput_norm, delay component AvgDelay_norm, priority violation component VioCount_norm, and load balancing component Balance_norm. The throughput component is: Throughput is the total throughput of successful transmissions, and T_max is the sum of the available bandwidth of each path; The time delay component is: AvgDelay is the average delay of all currently transmitted data streams from the sender to the receiver, and D_max is the maximum tolerable delay of the high-priority data packets. The priority violation component is: VioCount is the number of times the high-priority data packet exceeds the maximum delay. The total number of high-priority data packets sent during this period; The load balancing component is: , This represents the standard deviation of bandwidth utilization for each link within this period. To determine the theoretical maximum standard deviation of bandwidth utilization for each link within this period, we can take the standard deviation when each path's capacity is half fully loaded and half unloaded. The compound reward value R is: Wherein, α, β, γ, and δ are preset adjustable weight coefficients.
5. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 4, characterized in that, The reinforcement learning decision model adopts a deployment method that combines offline pre-training and online fine-tuning. Specifically, in the offline stage, the policy network and the value network are pre-trained in a simulation environment until convergence. During the online phase, the policy network outputs the data distribution policy in a deterministic manner without introducing random exploration noise. At the same time, the transition samples generated during the online operation are continuously stored in the experience replay pool, and the policy network and the value network are updated online with a learning rate lower than that in the offline phase. Gradient pruning is performed during the update process to constrain the magnitude of parameter changes.
6. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 5, characterized in that, The step of feeding back the incentive signal to the reinforcement learning decision model and driving the reinforcement learning decision model to update the policy in order to optimize the subsequent data distribution strategy specifically involves: storing the current real-time state information, the data distribution strategy, the incentive signal, and the state information of the next cycle as the transfer samples in the experience replay pool, and sampling a small batch of the transfer samples from the experience replay pool to update the parameters of the reinforcement learning decision model.
7. The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning according to claim 1, characterized in that, When the receiving end aggregates the data packets from each of the heterogeneous communication links, it maintains a reordering buffer for each data stream and sets a dynamic timeout timer. The timeout threshold of the dynamic timeout timer is dynamically set according to the maximum round-trip delay value among all currently active heterogeneous communication links.
8. A multi-network aggregation transmission optimization system for emergency management communication based on reinforcement learning, characterized in that, The method for optimizing multi-network aggregation transmission in emergency management communication based on reinforcement learning, as described in claims 1-7, includes: A multi-network interface module, configured to support access to at least two heterogeneous communication networks; The network status awareness module is configured to collect real-time status information of each heterogeneous communication link; The multi-path aggregation proxy module is deployed on the sending end and configured to receive data streams from upper-layer applications, identify the priority level of data packets, and perform data packet distribution operations according to the data distribution strategy output by the reinforcement learning decision engine. A multipath aggregation module, deployed at the receiving end, is configured to aggregate and reassemble the data packets arriving from each link and generate transmission quality feedback information. The reinforcement learning decision engine includes a reinforcement learning decision model, configured to output the data distribution strategy based on the real-time state information provided by the network state awareness module, and to calculate the incentive signal based on the transmission quality feedback information to update itself.
9. The emergency management communication multi-network aggregation transmission optimization system based on reinforcement learning according to claim 8, characterized in that, The multi-path aggregation proxy module is implemented based on the user-space protocol stack and identifies the service priority level of the data packet by distinguishing service tags or application layer identifiers. The multipath aggregation module maintains a reordering buffer and a dynamic timeout timer to reorder out-of-order data packets and determine packet loss for data packets that have not arrived within the timeout period.
10. The emergency management communication multi-network aggregation transmission optimization system based on reinforcement learning according to claim 8, characterized in that, The heterogeneous communication network includes at least two of the following: satellite communication network, 4G / 5G cellular network, and dedicated core network; The transmitting end and the receiving end establish multiple parallel transmission paths through multiple heterogeneous communication links to achieve multi-network aggregated transmission.