Low-latency packet transmission method and system based on reinforcement learning and stream encoding driving

Abstract

The present application relates to the technical field of wireless communication, and in particular to a low-latency packet transmission method and system based on reinforcement learning and stream coding driving.The present application discloses a low-latency packet transmission method based on reinforcement learning and stream coding driving and implemented in QUIC.The method uses SC to replace the traditional loss recovery mechanism of QUIC based on retransmission, providing low-delay loss recovery without packet retransmission, effectively improving the high end-to-end ordered delivery delay problem in NTNs.The present application proposes a congestion control algorithm LBE based on reinforcement learning RL.The algorithm identifies the system state by tracking and identifying ACK and uses the SARS algorithm for learning.By using the epsilon-greedy strategy for exploration, LBE balances between exploring new possibilities and using known optimal strategies, solving the low link utilization problem in NTNs and improving the effective throughput of transmission.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and more specifically to a low-latency packet transmission method and system based on reinforcement learning and stream coding. Background Technology

[0002] Non-terrestrial networks (NTNs) will play a crucial role in the upcoming 6G networks. In NTNs, long-distance lossy links will be ubiquitous. As a key factor affecting end-to-end application experience, the design of transport layer protocols in these scenarios is of paramount importance.

[0003] The widely used Transmission Control Protocol (TCP) faces two problems in NTNs: low link utilization and high-end-to-end ordered delivery delay. The former is caused by TCP's Congestion Control Algorithm (CCA). Classical CCA uses two phases, Slow Start (SS) and Congestion Avoidance (CA), to adjust the congestion window (CWND), a major factor determining link utilization. In NTNs, due to the high round-trip time (RTT), the growth of CWND in both SS and CA can be quite slow. Furthermore, CCA may misinterpret random packet loss on the radio link as congestion, leading to unnecessary CWND drops. The latter is mainly due to the head-of-line (HoL) blocking problem caused by TCP's retransmission-based loss recovery mechanism. This mechanism introduces at least one RTT delay before ordered packet delivery and causes subsequent packets to be blocked while waiting for HoL-lost packets to be recovered. The latency caused by HoL blocking can lead to fluctuations in the effective throughput delivered to the application layer, severely disrupting the user experience in scenarios such as streaming media and interactive applications.

[0004] In recent years, many CCA variants of TCP, such as Cubic, Hybla, and BBR, have emerged to address these issues. Among them, the machine learning-based PCC (Performance-oriented Congestion Control) algorithm performs exceptionally well in various scenarios, including dynamic high-bandwidth delay-product links. However, the rigidity of the TCP protocol limits the deployment and widespread adoption of these TCP variants. In contrast, designs based on UDP (User Datagram Protocol) do not suffer from these problems. A typical example is Quick UDP Internet Connections (QUIC), proposed by Google. Unlike TCP, which runs in kernel space, QUIC runs in user space, thus avoiding the deployment issues faced by TCP. However, because QUIC still employs congestion control and lost packet recovery mechanisms similar to TCP, the problems faced by TCP on long-distance lossy links also exist in QUIC.

[0005] To address these issues, forward erasure correction (FEC) can be used as an enhancement / alternative to retransmission-based loss recovery mechanisms to achieve lower end-to-end ordered delivery latency. Compared to retransmission, FEC sends repair packets in a redundant manner. When a packet is lost, FEC attempts to recover the lost packet using subsequently received repair packets without requiring additional time to notify the sender to retransmit. Garrido and Michel et al. added an FEC mechanism to the QUIC protocol. However, because it still relies on a request-based recovery mechanism to ensure protocol reliability, its performance in NTN networks is not ideal. Summary of the Invention

[0006] The purpose of this invention is to solve the above problems and proposes a low-latency packet transmission method and system based on reinforcement learning and stream coding.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The low-latency packet transmission method based on reinforcement learning and stream coding includes the following steps:

[0009] S1: QUIC connection initialization. During QUIC connection initialization, the sending and receiving ends will initialize the stream coding parameters. The sending end's stream coding parameters include the size of the file to be transmitted, the number of source packets to be sent, the Galois field size, the size of the transmitted packet, the frequency of sending repair packets, and the random coding coefficient seed. When the requested file size and the number of source packets to be sent are both 0, it indicates that the packets arrive sequentially and there is no fixed size. The receiving end's stream coding parameters include the Galois field size, the size of the transmitted packet, the frequency of sending repair packets, and the random coding coefficient seed.

[0010] S2: The application transmits the data to be sent to the QUIC sender in bytes;

[0011] S3: At the QUIC sending end, it checks whether there are any remaining congestion windows and flow control windows. If there are no remaining congestion windows or flow control windows, it will repeat S3.

[0012] S4: When both congestion control and flow control allow data transmission, the QUIC sender generates and encrypts packets; the encrypted QUIC packets are then fed into the SCEncoder module; considering a send buffer, packets sent from QUIC to UDP ordered by arrival time are called source packets and are denoted as s0, s1, ... Each s has the same size, and after appending the necessary header SCheader, it is adapted to the maximum transmission unit (MTU) of the underlying network;

[0013] S5: The QUIC transmitter selects whether to send a source packet from the buffer or a repair packet based on the target bit rate of the current stream coding.

[0014] S6: Packets arrive at the receiving end via an NTN network; the receiving end decodes the received packets and submits them to the QUIC; the SCDecoder removes the SC header from the received packets and delivers them to the QUIC; the decoder is initially in an ordered state, and if its ordered state is interrupted, it means that a packet loss has occurred, and the decoder immediately enters an out-of-order state and is activated; after the ordered transmission is interrupted, the decoder is activated and uses a flexible window called the decoding window (DW) to buffer the received source packets and repair packets; after receiving a sufficient number of repair packets, the decoder recovers all the lost source packets in the DW, and then forwards the recovered source packets to the QUIC and restores the ordered state; regardless of whether the decoder is activated, subsequent arriving packets are still delivered to the QUIC according to the default settings, so that the QUIC promptly sends ACK packets back to the sending end;

[0015] S7: The sending end processes the ACK packet, estimates the round-trip time (RTT), the number of acknowledged packets, and the link packet loss rate; the LBE congestion control algorithm updates the link congestion window based on the number of acknowledged packets and the RTT; the SC encoder adaptively adjusts the target bit rate of stream coding based on the link packet loss rate.

[0016] S8: Repeat S4 to S7 until the data transmission ends.

[0017] As a preferred technical solution of the present invention, a lossy packet recovery mechanism based on SC without retransmission is adopted; when random packet loss occurs, the receiver will not wait for retransmission but will activate the decoder; the decoder will attempt to recover the lost packet using subsequently received repair packets; the SC encoding and decoding process is as follows: the SCEncoder module provides the SC encoding function, the generated QUIC packets are input into the SCEncoder module and different operations are performed according to their type; the QUIC packet type includes packets that need to be retransmitted when the packet is lost and packets that do not need to be retransmitted; only QUIC packets that need to be retransmitted when the packet is lost are protected; for the packets to be protected, the SCEncoder treats them as an abstract byte stream and adds an SC header to the beginning, and then passes them to the UDP sending queue; the protected packets are called source packets after passing through the SCEncoder; for unprotected packets, the SCEncoder directly passes them to the UDP sending queue; the sender selects whether to send repair packets based on the target bitrate of the stream encoding; the repair packets are a linear combination of the source packets that have already been sent; let i seq This represents the ID of the most recently sent source packet, initialized with i. seq =-1, i = -1 after each source packet is sent seq Add 1; the repair group is represented as:

[0018]

[0019] Where c k For the repair group with ID k; g k,i To from a finite field The stream coding coefficients are randomly extracted from the data, where k = 0, 1, 2, 3... represents the number of the repair group; w s Corresponding to the earliest source packet number in the current sending queue; initializing w s =0, based on the feedback from the receiving end, the original packets that have been confirmed as received will be removed from the queue, at which point w s An update will be made; let w e =i seq , [w s w e This is called the encoding window of the current repair group;

[0020] The SCDecoder module provides decoding functionality for stream-encoded transmissions; the receiving end inputs all received packets into the SCDecoder; the SCDecoder removes the SC header from the received source packets and delivers them to the QUIC session. If packet loss occurs, the decoder is activated and data is recovered using the received repair packets; let i ord Represents the latest ordered transmission packet number, initialized i ord =-1, the decoder is initially in an ordered state; if the next source packet received by the decoder is not s iord+1 This means that the ordered transmission has been interrupted, and the decoder enters an out-of-order state. At this time, the decoder will buffer the received packets and attempt to decode them; the buffered packets are out-of-order source packets with numbers greater than i. ord +1 or repair group, where w e >i ord +1; let And also ordered The largest source packet received; This is called the decoder's current DW; as the number of buffered packets increases, the window may grow. The decoder uses Gaussian elimination for decoding, dynamically constructing a linear equation system AS=B and performing online forward elimination, where the rows of A and B are the coding coefficients and coding information symbols of the buffered packets, respectively. When decoding is successful, all decoded source packets in the decoding window are transmitted to the upper-layer application, the decoder returns to an ordered state, and ordered transmission continues. Restart; when the decoder is activated, subsequent packets will still be forwarded to the upper layer as before; when the QUIC session receives a packet, it sends an ACK packet back to the sender.

[0021] As a further preferred technical solution of the present invention, an adaptive FEC algorithm based on the link packet loss rate is used to dynamically adjust the FEC code rate; an adaptive algorithm based on the current loss rate is used to adjust the FEC code rate, which adds additional repair packets in addition to the current loss rate to ensure that the decoding time does not approach infinity; the FEC code rate is defined as the ratio of the number of repair packets to the total number of packets sent, and the formula is:

[0022]

[0023] Among them, S i and R i These represent the number of source packets and the number of repair packets sent at the current time, respectively; (Definition) The ID of the largest source data packet confirmed by the sender. Let QUIC estimate the total number of lost packets at the current moment; the smoothed packet loss rate is defined as the instantaneous packet loss rate after low-pass filtering, and the formula is:

[0024]

[0025] Here, α is usually set to 0.9;

[0026] The SC's FEC rate is adjusted according to the following steps: After each source packet is sent, check whether a repair packet needs to be sent; the current f and If the difference is less than the target value δ∈(0,1), a repair packet is sent; δ represents the extra repair packet insertion rate, which is inversely proportional to the decoding delay of SC in the case of random packet loss; a larger δ value corresponds to a smaller expected decoding delay.

[0027] As a further preferred embodiment of the present invention, a bandwidth estimation congestion control algorithm (LBE) based on reinforcement learning (RL) is employed. The LBE algorithm tracks ACKs to identify the system state, and the reinforcement learning RL agent learns using the SARSA algorithm based on the system state and explores using an ε-greedy strategy. The LBE algorithm includes the following steps:

[0028] First, the round-trip time (RTT) is taken as an observation period, and the following four-dimensional state is proposed to describe the system:

[0029]

[0030] in, SR is the ratio of the maximum RTT to the minimum RTT observed within the t-th observation period. t SRR is the linearly weighted average of the six most recent RTTs within the t-th observation period. t It is SR t With SR t-1 The ratio, BW, is the estimated link bandwidth obtained at this time using the Tcp-Jersey algorithm. The Tcp-Jersey algorithm uses a time-sliding window estimator to estimate the bandwidth, and its estimation is performed per ACK; assuming the arrival time of the nth ACK is t. n Then BW is represented as:

[0031]

[0032] Among them, L n t is the size of the nth ACK confirmation packet. n-1 It is the time when the previous ACK arrived, RTT. n It is time t n The instantaneous RTT; tile coding is used to discretize high-dimensional continuous values ​​and reduce the number of states; by using four layers of tile coding, the originally infinite state space is limited to 4*10*10*10*5=20,000 states;

[0033] Secondly, it provides the definition of action space, and based on the operation phase of LBE, it divides the action into two phases, namely the improved SS phase and the RL training phase.

[0034] In the improved SS phase, due to the influence of the uplink transmission buffer in the satellite backhaul link, the traditional window increase mechanism of SS causes link disturbances. A gain variable α is introduced to adjust the window growth rate to overcome this, which is inversely proportional to the delay change. The adjusted overall window growth rate is non-linear, lower than the exponential window growth rate of traditional SS, but higher than the linear growth rate of traditional CA.

[0035] In the RL phase, the action space includes linearly increasing LI, non-linearly increasing NLI, linearly decreasing LD, non-linearly decreasing NLD, and remaining unchanged. The action is non-linear when the product of the current TCP-Jersey estimated bandwidth and link delay is greater than CWND; otherwise, the action is linear. A window adjustment factor μ is introduced, representing the expected RTT required for CWND to grow from its current value to BDP. μ is variable and inversely proportional to the difference between the current CWND and the estimated BDP of the link. The update magnitude is limited to no more than 300 maximum segment lengths per update. The specific action design is shown in the following formula:

[0036]

[0037] Finally, the reward function for LBE is defined as follows:

[0038] Reward = Utility t+RTT -Utility t (2)

[0039] Utility t =α*log Throughput t -βlog RTT t (3)

[0040] Where α and β represent the weighting coefficients of throughput and RTT, and the reward function is Utility. t Reflects the expected design goal; Formula (3) embodies the expected design goal, and the reward value is defined by the difference between the results of two consecutive times; the two consecutive times are t and t+RTT, and the reward rule is expressed as:

[0041]

[0042] Here, ∈ represents the tolerance for continuous reward differences; only differences greater than ∈ can generate positive reinforcement. In the RL stage, the SARSA algorithm is used for learning, while an ε-greedy policy is used for exploration, striking a balance between exploring new possibilities and utilizing known optimal policies.

[0043] A transmission system based on a low-latency packet transmission method driven by reinforcement learning and stream coding includes: a transmitter, a receiver, a stream code transmission unit, and a congestion control unit;

[0044] Sender: The sender is used to send packets. An SC encoder (SCEncoder) is introduced between the QUIC and UDP interfaces of the sender, and the QUIC retransmission module is disabled. The types of packets include: Unprotected packets: packets not protected by stream coding, which cannot be recovered if lost in the link; Source packets: packets protected by stream coding; Repair packets: repair packets generated by the encoder to recover lost source packets.

[0045] Receiver: The receiver receives packets and recovers lost packets. An SC decoder (SCDecoder) is introduced between the QUIC and UDP interfaces of the receiver. When a loss occurs, the decoder recovers the lost packets using subsequently received repair packets. The retransmission module is disabled in the QUIC, but the ACK mechanism is complete; the receiver's QUIC sends an ACK to the sender according to its default mechanism. Stream Coding Transmission Unit: The stream coding transmission unit includes an SCEncoder module and an SCDecoder module, providing encoding and decoding functions for stream coding transmission, respectively. A send buffer and an adaptive FEC module are added to it. In the send buffer, packets sent from the QUIC to the UDP are sorted according to their arrival time; these data packets are called source packets. The stream coding transmission unit also uses a packet-level adaptive FEC coding based on packet loss rate, called stream coding. By estimating the link packet loss rate, the stream coding transmission unit adaptively sends repair packets. When a loss occurs, the decoder is activated and buffers the received source packets and repair packets through an elastic window called the decoding window. After receiving a sufficient number of repair packets, the decoder successfully recovers all lost source packets in the DW.

[0046] Congestion Control Unit: The congestion control unit provides congestion control functions suitable for NTNs networks. The congestion control unit is based on RL and is divided into two phases: the improved SS phase and the RL training phase. In the improved SS phase, link disturbances caused by UT buffers are avoided by reducing the growth rate of CWND. In the RL training phase, the SARSA algorithm is used for learning, and an ε-greedy strategy is used for exploration to achieve a balance between exploring new possibilities and utilizing known optimal strategies.

[0047] The low-latency packet transmission method and system based on reinforcement learning and stream coding proposed in this invention have the following technical advantages compared with existing technologies:

[0048] (1) This invention discloses a low-latency packet transmission method based on reinforcement learning and stream coding and implements it in QUIC. This method uses SC to replace the traditional retransmission-based loss recovery mechanism of QUIC, providing low-latency loss recovery without the need for packet retransmission, and effectively improving the high-end-to-end ordered delivery latency problem in NTNs.

[0049] (2) This invention proposes a congestion control algorithm, LBE, based on reinforcement learning (RL). This algorithm identifies the system state by tracking and recognizing ACKs and uses the SARSA algorithm for learning. By utilizing an ε-greedy strategy for exploration, LBE achieves a balance between exploring new possibilities and utilizing known optimal strategies, solving the low link utilization problem in NTNs and improving the effective throughput of transmission.

[0050] (3) This invention proposes an adaptive algorithm to adjust the FEC bitrate. The algorithm inserts additional repair groups on the basis of the current loss rate and dynamically adjusts the FEC bitrate, ensuring that the decoding time does not approach infinity without consuming too much bandwidth resources. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of a low-latency packet transmission method based on reinforcement learning and stream coding according to an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0053] like Figure 1 As shown, the low-latency packet transmission method based on reinforcement learning and stream coding includes the following steps:

[0054] S1: QUIC Connection Initialization. During QUIC connection initialization, both the sending and receiving ends initialize their stream coding parameters. The sending end's stream coding parameters include the requested file size, the number of source packets to be sent, the Galois field size, the transmission packet size, the frequency of sending repair packets, and the random coding coefficient seed. If both the requested file size and the number of source packets to be sent are 0, it indicates that packets arrive sequentially without a fixed size. The receiving end's stream coding parameters include the Galois field size, the transmission packet size, the frequency of sending repair packets, and the random coding coefficient seed.

[0055] S2: The application transmits the data to be sent to the QUIC sender in bytes.

[0056] S3: At the QUIC sending end, it checks whether there are any remaining congestion windows and flow control windows. If there are no remaining congestion windows or flow control windows, it will repeat S3.

[0057] S4: When both congestion control and flow control allow data transmission, the QUIC sender generates and encrypts packets. The encrypted QUIC packets are then fed into the SCEncoder module, where different operations are performed based on their type. To facilitate identification of packets to be protected, the SCEncoder appends a header to the QUIC packets before transmission and ensures that the generated packets do not exceed the MTU by limiting the maximum size of the generated QUIC packets. The packets are then passed to the UDP send queue. The packets sent from QUIC to UDP are referred to as source packets.

[0058] S5: The sending end selects whether to send a repair group based on the target bit rate of the current stream coding; the repair group is a linear combination of the source groups that have already been sent. Let i seq Indicates the number of the most recently sent source packet, initialized i. seq =-1, i = -1 after each source packet is sent seq Add 1. The repair group is represented as:

[0059]

[0060] Where c k For the repair group numbered k; g k,i The stream coding coefficients are randomly extracted from a finite field, k = 0, 1, 2, 3... where k is the ID of the repaired packet; w s This is the number of the oldest unacknowledged source packet in the current transmission queue. The target code rate is the FEC code rate dynamically adjusted based on the link packet loss rate using an adaptive FEC algorithm, and can be described as:

[0061]

[0062] Among them, S i and R i These represent the number of source packets and the number of repair packets sent at the current time, respectively. (Definition) The largest source packet sequence number acknowledged by the sender. This represents the estimated total number of lost packets by QUIC at the current moment. The smoothed packet loss rate, defined as the instantaneous packet loss rate after low-pass filtering, can be described as:

[0063]

[0064] Typically, α is set to 0.9. The FEC rate of SC is adjusted according to the following rules:

[0065] After each source packet is sent, a check is performed to determine if a repair packet needs to be sent. If the current f is... If the difference is less than the target value δ (δ∈(0,1)), a repair packet is sent. Here, δ represents the additional repair packet insertion rate, which is inversely proportional to the decoding delay of SC in the case of random packet loss. Specifically, a larger δ value corresponds to a smaller expected decoding delay.

[0066] S6: The receiving end decodes the received packets and submits them to QUIC. The SCDecoder removes the SC header from the received packets and delivers them to the session. The decoder initially operates in an ordered state; if its ordered state is disrupted, it indicates packet loss, and the decoder immediately enters an out-of-order state and becomes active. The active decoder uses subsequently received repair packets to recover the data. Let i ord Represents the latest ordered source packet ID, initialized i ord = -1, the decoder is initially in an ordered state. If the next source packet received by the decoder is not s iord+1 This means that ordered transmission has been interrupted, and the decoder enters an out-of-order state. At this time, the decoder will buffer the received packets and attempt to decode them. The buffered packets are out-of-order source packets (whose numbers are greater than i). ord +1) or repair group (where w e >i ord +1). Let And also ordered The largest source packet received. This is called the current DW of the decoder. As the number of buffered packets increases, the window also grows. After receiving enough repair packets, the decoder uses online Gaussian elimination to decode, i.e., constructing a dynamic linear equation system AS = B and performing forward elimination, where the rows of A and B are the coding coefficients of the buffered packets (unordered source packets are considered as special repair packets with only one non-zero element 1 in their coding coefficients) and the coded information symbols, respectively. When decoding is successful, all decoded source packets in the decoding window are transmitted to the upper-layer application, the decoder returns to an ordered state, and ordered transmission continues... Restart. Meanwhile, regardless of whether the decoder is active or not, arriving packets are still delivered to the upper layer according to QUIC's default settings, ensuring that QUIC promptly sends ACK packets back to the sender.

[0067] S7: The sending end processes the ACK packets, estimates the round-trip time (RTT), the number of acknowledged packets, and the link packet loss rate. The RL-based LBE congestion control algorithm is adopted. This algorithm can track ACKs to identify the system state. The RL agent learns using the SARSA algorithm based on the system state and explores using an ε-greedy strategy. The specific description of the LBE algorithm is as follows:

[0068] A complete reinforcement learning design requires determining the action space and state space, defining the reward signal, and selecting an update algorithm for the state-action value function.

[0069] First, a round-trip time (RTT) is used as an observation period, and the following four-dimensional state is proposed to describe the system:

[0070]

[0071] in, SR is the ratio of the maximum RTT to the minimum RTT observed within the t-th observation period. t SRR is the linearly weighted average of the six most recent RTTs within the t-th observation period. t It is SR t With SR t-1 The ratio, BW, is the estimated link bandwidth obtained at this time using the Tcp-Jersey algorithm. The Tcp-Jersey algorithm uses a time-sliding window estimator to estimate the bandwidth, and its estimation is performed per ACK. Assume the arrival time of the nth ACK is t. n Then BW can be represented as:

[0072]

[0073] Where: L n t is the size of the nth ACK confirmation packet. n-1 It is the time when the previous ACK arrived, RTT. n It is time t n The instantaneous RTT. Note that the four state variables mentioned above are all continuous variables, and directly using them as states may lead to the curse of dimensionality. Therefore, tile coding is used to discretize high-dimensional continuous values ​​and reduce the number of states. By using four layers of tile coding, the originally infinite state space is limited to 4*10*10*10*5 = 20,000 states.

[0074] Secondly, the action space is defined. Based on the LBE operation phase, the action is divided into two phases: the improved SS phase and the RL training phase.

[0075] In the improved SS phase, the traditional window growth mechanism of SS may cause link interference due to the influence of the uplink transmission buffer in the satellite backhaul link. A gain variable α is introduced to adjust the window growth rate to overcome this problem. The adjusted overall window growth rate is non-linear, lower than the exponential window growth rate of traditional SS, but higher than the linear growth rate of traditional CA.

[0076] In the RL phase, the action space includes linear increase (LI), non-linear increase (NLI), linear decrease (LD), non-linear decrease (NLD), and invariance. The action is non-linear when the product of the current TCP-Jersey estimated bandwidth and link delay is greater than CWND. Otherwise, the action is linear. A window adjustment factor μ is introduced, representing the expected RTT required for CWND to grow from its current value to BDP. μ is variable and inversely proportional to the difference between the current CWND and the estimated BDP of the link. To prevent large update values ​​from affecting link delay, the update magnitude is limited to no more than 300 MSS per update. The specific action design is as follows:

[0077]

[0078] Finally, the reward function for LBE is defined as follows:

[0079] Reward = Utility t+RTT -Utility t (2)

[0080] Utility t =α*logThroughput t -βlogRTT t (3)

[0081] Where α and β represent the weighting coefficients of throughput and RTT, and the reward function is Utility. t This reflects the expected design goals. Formula (3) embodies the expected design goals. Although its results can be used to evaluate performance, a larger value does not necessarily mean better performance, implying that network performance is not necessarily optimal. Therefore, the reward value in this paper is defined by the difference between the results of two consecutive times. The two consecutive times are t and t+RTT, and the reward rule is expressed as:

[0082]

[0083] Here, ∈ represents the tolerance for continuous reward differences; only differences greater than ∈ can generate positive reinforcement. In the RL phase, the SARSA (State-Action-Reward-State-Action) algorithm is used for learning, which typically produces better online performance than Q-learning. It also utilizes an ε-greedy policy for exploration, striking a balance between exploring new possibilities and leveraging known optimal policies. The complete LBE algorithm is given in Algorithm 1 below:

[0084]

[0085]

[0086] Here, γ is the discount factor, representing the short-term metric of the agent, and α is the learning rate, which determines the convergence speed and difficulty of the network. Q(S) C Q C ) represents the current state (S) C ) and actions (Q) C The Q value of ) can also be understood as the value of the current state S. C Execute action Q C The expected reward is obtained. The algorithm uses an ε-greedy strategy for exploration, where one action is randomly selected with probability ε, and the current best action is selected with probability 1-ε, in order to strike a balance between exploring new possibilities and utilizing known optimal strategies.

[0087] S8: Repeat S4 to S7 until the packet transmission ends.

[0088] This invention also discloses a low-latency packet transmission system driven by RL and FEC. The system includes a transmitter, a receiver, a stream code transmission unit, and a congestion control unit. The system model is specifically as follows: Figure 1 As shown.

[0089] The sending end is used to transmit packets. This invention introduces an SC encoder (SCEncoder) between the QUIC and UDP interfaces at the sending end and disables the QUIC retransmission module. The packet types include: unprotected packets: packets not protected by stream coding, which cannot be recovered if lost in the link; source packets: packets protected by stream coding; and repair packets: repair packets generated by the encoder to recover lost source packets.

[0090] The receiving end is used to receive packets and recover lost packets. This invention introduces an SC decoder (SCDecoder) between the QUIC and UDP interfaces at the receiving end. When loss occurs, the decoder can recover lost packets from subsequently received repair packets. Although the retransmission module is disabled in the QUIC, the ACK mechanism is complete; the receiving end QUIC sends an ACK to the sending end according to its default mechanism.

[0091] The stream-coded transmission unit improves the reliability of packet transmission systems. The stream-coded transmission unit includes an SCEncoder module and an SCDecoder module, providing encoding and decoding functions for stream-coded transmission, respectively. This invention incorporates a send buffer and an adaptive FEC module. In the send buffer, packets sent from QUIC to UDP are ordered by arrival time. These data packets are called source packets. The stream-coded transmission unit also employs a packet-level adaptive FEC coding based on packet loss rate, called stream coding. By estimating the link packet loss rate, the stream-coded transmission unit adaptively sends repair packets. When a loss occurs, the decoder is activated and buffers received source packets and repair packets through an elastic window called the decoding window. After receiving a sufficient number of repair packets, the decoder can successfully recover all lost source packets in the DW.

[0092] The congestion control unit provides congestion control functions suitable for NTNs networks. Based on Restrictive Logic (RL), the congestion control unit consists of two phases: an improved Service Strategies (SS) phase and an RL training phase. In the improved SS phase, link disturbances caused by the Undertaker buffer are avoided by reducing the growth rate of the Control WND (CWND). In the RL phase, the SARSA (State-Action-Reward-State-Action) algorithm is used for learning, and an ε-greedy strategy is employed for exploration, striking a balance between exploring new possibilities and utilizing known optimal strategies.

[0093] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A low-latency packet transmission method based on reinforcement learning and stream coding, characterized in that, Includes the following steps: S1: QUIC connection initialization. During QUIC connection initialization, the sending and receiving ends will initialize the stream coding parameters. The sending end's stream coding parameters include the size of the file to be transmitted, the number of source packets to be sent, the Galois field size, the size of the transmitted packet, the frequency of sending repair packets, and the random coding coefficient seed. When the requested file size and the number of source packets to be sent are both 0, it indicates that the packets arrive sequentially and there is no fixed size. The receiving end's stream coding parameters include the Galois field size, the size of the transmitted packet, the frequency of sending repair packets, and the random coding coefficient seed. S2: The application transmits the data to be sent to the QUIC sender in bytes; S3: At the QUIC sending end, it checks whether there are any remaining congestion windows and flow control windows. If there are no remaining congestion windows or flow control windows, it will repeat S3. S4: When both congestion control and flow control allow data transmission, the QUIC sender generates and encrypts packets; the encrypted QUIC packets are then fed into the SCEncoder module; considering a send buffer, the packets sent from QUIC to UDP, ordered by arrival time, are called source packets and denoted as... Each They have the same size and, after adding the necessary SC header, are adapted to the maximum transmission unit (MTU) of the underlying network. S5: The QUIC transmitter selects whether to send a source packet from the buffer or a repair packet based on the target bit rate of the current stream coding. S6: The packet reaches the receiving end through an NTN network; the receiving end decodes the received packet and submits it to QUIC; the SCDecoder removes the SC header from the received packet and delivers it to QUIC; The decoder is initially in an ordered state. If its ordered state is interrupted, it means that a packet loss has occurred, and the decoder immediately enters an out-of-order state and is activated. After the ordered transmission is interrupted, the decoder is activated and uses a flexible window called the decoding window (DW) to buffer the received source packets and repair packets. After receiving a sufficient number of repair packets, the decoder recovers all the lost source packets in the DW, and then forwards the recovered source packets to the QUIC and restores the ordered state. Regardless of whether the decoder is activated, subsequent arriving packets are still delivered to the QUIC according to the default settings, so that the QUIC can promptly send ACK packets back to the sender. S7: The sending end processes the ACK packet, estimates the round-trip time (RTT), the number of acknowledged packets, and the link packet loss rate; The LBE congestion control algorithm updates the link congestion window based on the number of confirmed packets and the round-trip time (RTT); the SC encoder adaptively adjusts the target bitrate of stream coding based on the link packet loss rate. S8: Repeat S4 to S7 until the data transmission is complete; A lossless packet recovery mechanism based on SC (Synchronous Array Encoding) is adopted. When random packets are lost, the receiver does not wait for retransmission but activates the decoder. The decoder attempts to recover the lost packets using subsequently received repair packets. The SC encoding and decoding process is as follows: The SCEncoder module provides SC encoding functionality. The generated QUIC packets are input into the SCEncoder module and different operations are performed according to their type. The QUIC packet types include packets that need to be retransmitted when lost and packets that do not need to be retransmitted. Only QUIC packets that need to be retransmitted when lost are protected. For packets to be protected, the SCEncoder treats them as an abstract byte stream and adds an SC header to the beginning before passing them to the UDP send queue. The protected packets are called source packets after passing through the SCEncoder. For unprotected packets, the SCEncoder directly passes them to the UDP send queue. The sending end selects whether to send a repair group based on the target bit rate of the current stream coding; the repair group is a linear combination of the source groups that have already been sent; let Indicates the ID of the most recently sent source packet, initialized. =-1, after each source packet is sent, Add 1; the repair group is represented as: ； in Group the repairs for ID k; To from a finite field The stream coding coefficients are randomly extracted from the data, and k = 0, 1, 2, 3... are the numbers of the repair groups; Corresponding to the earliest source packet number in the current sending queue; initialization =0, based on the feedback from the receiving end, the original packets that have been acknowledged as received will be removed from the queue. An update will be made; = , [ , This is called the encoding window of the current repair group; The SCDecoder module provides decoding functionality for stream-encoded transmissions; the receiving end inputs all received packets into the SCDecoder; the SCDecoder removes the SC header from the received source packets and delivers them to the QUIC session. If packet loss occurs, the decoder is activated and data is recovered using the received repair packets; Indicates the latest ordered transmission packet number, initialized. =-1, the decoder is initially in an ordered state; if the next source packet received by the decoder is not... This means that ordered transmission has been interrupted, and the decoder enters an out-of-order state. At this time, the decoder will buffer the received packets and attempt to decode them; the buffered packets are out-of-order source packets, and their numbers are greater than [number missing]. +1 or repair group, where > +1; let And also ordered The largest source packet received; , This is called the decoder's current DW; as the number of buffered packets increases, the window may grow. Growth; the decoder uses Gaussian elimination for decoding, that is, dynamically constructing a linear equation system AS = B and performing online forward elimination, where the rows of A and B are the coding coefficients and coding information symbols of the buffered packets, respectively; when decoding is successful, all decoded source packets in the decoding window are transmitted to the upper layer application, the decoder is restored to an ordered state, and ordered transmission is carried out. = Restart; when the decoder is activated, subsequent packets will still be forwarded to the upper layer as before; when the QUIC session receives a packet, it sends an ACK packet back to the sender; An adaptive FEC algorithm based on link packet loss rate is used to dynamically adjust the FEC code rate; An adaptive algorithm based on the current loss rate is used to adjust the FEC rate. This algorithm adds extra repair packets in addition to the current loss rate to ensure that the decoding time does not approach infinity. The FEC rate is defined as the ratio of the number of repair packets to the total number of packets sent, as shown in the formula: ； in, and These represent the number of source packets and the number of repair packets sent at the current time, respectively; (Definition) The ID of the largest source data packet confirmed by the sender. Let QUIC estimate the total number of lost packets at the current moment; the smoothed packet loss rate is defined as the instantaneous packet loss rate after low-pass filtering, and the formula is: ； in, It is usually set to 0.9; The SC's FEC rate is adjusted according to the following steps: After each source packet is sent, check whether a repair packet needs to be sent; currently... and The difference is less than the target value If ∈(0,1), then a repair packet is sent; This represents the additional repair packet insertion rate, which, in the case of random packet loss, is inversely proportional to the decoding latency of SC; a larger rate indicates a higher rate. The value corresponds to a smaller expected decoding latency.

2. The low-latency packet transmission method based on reinforcement learning and stream coding as described in claim 1, characterized in that, A bandwidth estimation congestion control algorithm (LBE) based on reinforcement learning (RL) is adopted. The LBE algorithm tracks back-to-back (ACK) responses to identify the system state. The RL agent learns from the system state using the SARSA algorithm and explores using an ε-greedy strategy. The LBE algorithm includes the following steps: First, the round-trip time (RTT) is taken as an observation period, and the following four-dimensional state is proposed to describe the system: ； in, It is the ratio of the maximum RTT to the minimum RTT observed within the t-th observation period. It is the linearly weighted average of the six most recent RTTs within the t-th observation period. yes and The ratio, BW, is the link estimated bandwidth obtained at this time using the Tcp-Jersey algorithm. The Tcp-Jersey algorithm uses a time-sliding window estimator to estimate the bandwidth, and its estimation is performed on a per-ACK basis; assuming the arrival time of the nth ACK is... Then BW is represented as: ； in, It is the size of the nth ACK acknowledgment packet. It is the time when the previous ACK arrived, RTT. n It is a moment The instantaneous RTT; tile coding is used to discretize high-dimensional continuous values ​​and reduce the number of states; by using four layers of tile coding, the originally infinite state space is limited to 4*10*10*10*5=20,000 states; Secondly, it provides the definition of action space, and based on the operation phase of LBE, it divides the action into two phases, namely the improved SS phase and the RL training phase. In the improved SS phase, due to the impact of the uplink transmission buffer in the satellite backhaul link, the traditional window increase mechanism of SS causes link disturbances, introducing a gain variable. The window growth rate is adjusted to overcome this limitation, as it is inversely proportional to the delay change. The adjusted overall window growth rate is non-linear, lower than the exponential window growth rate of traditional SS, but higher than the linear growth rate of traditional CA. During the RL phase, the action space includes linearly increasing LI, nonlinearly increasing NLI, linearly decreasing LD, nonlinearly decreasing NLD, and remaining unchanged. The action is nonlinear when the product of the current TCP-Jersey estimated bandwidth and link delay is greater than CWND; otherwise, the action is linear. A window adjustment factor is introduced. It represents the expected RTT value required for CWND to grow from its current value to BDP. It is variable and inversely proportional to the difference between the current CWND and the estimated BDP of the link; the update magnitude is limited to no more than 300 maximum segment lengths each time; the specific action design is shown in the following formula: ； Finally, the reward function for LBE is defined as follows: ； ； in, and The weighting coefficients representing throughput and RTT, and the reward function. Reflects the expected design objective; Formula (3) embodies the expected design objective, and the reward value is defined by the difference between the results of two consecutive times; the two consecutive times are and The reward rules are expressed as follows: ； in, This represents tolerance for differences in consecutive rewards; only when it is greater than [a certain value]... Only differences can generate positive reinforcement; in the RL stage, the SARSA algorithm is used for learning, while an ε-greedy policy is used for exploration, striking a balance between exploring new possibilities and utilizing known optimal policies.

3. A transmission system based on the low-latency packet transmission method driven by reinforcement learning and stream coding as described in any one of claims 1-2, characterized in that, include: Transmitter, receiver, stream code transmission unit, and congestion control unit; Sender: The sender is used to send packets. An SC encoder (SCEncoder) is introduced between the QUIC and UDP interfaces of the sender, and the QUIC retransmission module is disabled. The types of packets include: Unprotected packets: packets not protected by stream coding, which cannot be recovered if lost in the link; Source packets: packets protected by stream coding; Repair packets: repair packets generated by the encoder to recover lost source packets. Receiver: The receiver receives packets and recovers lost packets. An SC decoder (SCDecoder) is introduced between the QUIC and UDP interfaces at the receiver. When a loss occurs, the decoder recovers the lost packets using subsequently received repair packets. The retransmission module is disabled in the QUIC, but the ACK mechanism is complete; the receiver's QUIC sends an ACK to the sender according to its default mechanism. Stream Coding Transmission Unit: The stream coding transmission unit includes an SCEncoder module and an SCDecoder module, providing encoding and decoding functions for stream coding transmission, respectively. A send buffer and an adaptive FEC module are added. In the send buffer, packets sent from the QUIC to UDP are sorted by arrival time; these data packets are called source packets. The stream coding transmission unit also employs a packet-level adaptive FEC coding based on packet loss rate, called stream coding. By estimating the link packet loss rate, the stream coding transmission unit adaptively sends repair packets. When a loss occurs, the decoder is activated and buffers the received source packets and repair packets through an elastic window called the decoding window. After receiving a sufficient number of repair packets, the decoder successfully recovers all lost source packets in the DW. Congestion Control Unit: The congestion control unit provides congestion control functions suitable for NTNs networks. The congestion control unit is based on RL and is divided into two phases: the improved SS phase and the RL training phase. In the improved SS phase, link disturbances caused by UT buffers are avoided by reducing the growth rate of CWND. In the RL training phase, the SARSA algorithm is used for learning, and an ε-greedy strategy is used for exploration to achieve a balance between exploring new possibilities and utilizing known optimal strategies.

Images