A method for bufferbloat control in a wireless communication environment

Abstract

The present application relates to the technical field of computer network and deep reinforcement learning network management, in particular to a wireless communication environment-oriented invisible buffer control method, the core of the method is to convert the invisible buffer governance problem into an edge gateway side single-point control problem, and a reinforcement learning intelligent control layer is constructed to consider the delay, throughput and fairness, so as to realize the inhibition of the expansion of the wireless link invisible buffer. The flow state perception factor and time sequence are extracted: the congestion evolution trend is inferred; the continuous back propagation mechanism is introduced. The reinforcement learning strategy neural network is adopted, the active queue management mechanism is linked to trigger the congestion signal, the congestion window is reduced, and the upstream sending end congestion window is indirectly regulated. The present application is suitable for multiple shared wireless backhaul scenarios, without modifying the terminal equipment or base station network infrastructure, the network delay can be significantly reduced, the high throughput efficiency can be maintained, the multi-flow concurrent fairness can be improved, and the rationality of the invisible buffer control is improved.

Description

Technical Field

[0001] This invention relates to the field of computer network and deep reinforcement learning network management technology, specifically to an invisible buffer control method for wireless communication environments. Background Technology

[0002] In network environments where 5G and Wi-Fi are the primary access technologies, wireless channels often exhibit communication fluctuations. To accommodate sudden jitter in the physical link, base stations and hardware drivers typically pre-configure packet buffer queues for data retransmission, and the relevant buffer paths can be categorized into lower-layer wireless modules. However, most of these encapsulated buffers cannot be directly observed by the kernel software probes of edge gateways, appearing as imperceptible and uncontrollable invisible buffer expansion from the kernel software perspective.

[0003] Existing control strategies face the following problems: First, traditional threshold-based single-point queue management, such as the Random Early Detection (RED) algorithm and the Controlled Delay (CoDel) algorithm, is limited by the feedback lag of fixed parameters and is difficult to adapt to the unfair allocation caused by high-concurrency traffic and link contention. Second, due to the limited computing power of devices, traffic scheduling schemes based on large networks with high parameter volume are difficult to perform closed-loop calculations on gateways with weak computing power, and are prone to model forgetting and policy failure due to lack of resources in online non-stationary scenarios.

[0004] Further analysis reveals that in wireless links containing upstream fair flow queue scheduling mechanisms, once the downstream invisible buffer overflows and causes packet loss, different service flows will experience unbalanced packet loss due to differences in burst patterns. This leads to actual throughput deviating from the upstream fair allocation result, thereby undermining multi-flow fairness. Therefore, relying solely on the software kernel's proactive queue management mechanism without suppressing invisible buffer backlog makes it difficult to stably achieve low latency and high fairness in wireless scenarios, resulting in poor rationality of invisible buffer control.

[0005] In wireless access networks, edge gateway devices face the problem of invisible buffer bloat. Compared to the easily observable kernel-mode queues in Ethernet or fiber broadband, data packets accumulate in the underlying hardware and proprietary driver queues in wireless architectures. Existing congestion control schemes face limitations in this scenario: First, the end-to-end control loop of transport layer congestion control algorithms is relatively long, affecting response speed and easily leading to continuous queuing; second, link-layer active queue management usually only involves visible queues, and packet loss due to downstream invisible buffer overflows can reduce the fairness that upstream mechanisms aim to ensure; third, some schemes that coordinate intervention in the base station or device kernel protocol stack are difficult to modify and have protocol compatibility limitations, resulting in poor rationality of invisible buffer control. Summary of the Invention

[0006] To address the technical problem of poor rationality in invisible buffer control, this invention proposes an invisible buffer control method for wireless communication environments. Its core lies in transforming the uncontrollable buffer expansion problem into a rate modulation problem based on single-point observation by the edge gateway module and execution by the single-point intelligent control module: by actively implementing bidirectional traffic shaping at the gateway, the visible queue (controlled state) is transformed into a controlled queue, thereby triggering congestion signals, such as packet loss or feedback random early detection signals, inducing the transmitter to actively reduce the congestion window, thereby clearing the deep invisible backlog in advance.

[0007] This invention provides an invisible buffer control method for wireless communication environments. The core of this method is to transform the invisible buffer governance problem into a single-point control problem at the edge gateway. A reinforcement learning intelligent control layer is constructed, taking into account latency, throughput, and fairness, to suppress the expansion of the invisible buffer in the wireless link. To achieve accurate perception of the unobservable buffer state, this invention designs a Markov decision process state perception method to extract flow state perception factors and time series sequences. For the invisibility of the buffer state, the congestion evolution trend is inferred by fusing historical observation sequences and flow dynamic leading indicators. For the non-stationarity of the wireless environment, a continuous backpropagation mechanism is introduced to reset inefficient nodes under maturity protection and dynamically reshape inefficient neurons, maintaining the long-term adaptability of the policy. To achieve online real-time control, a reinforcement learning policy neural network is used as the reinforcement learning policy network. It performs bidirectional rate dynamic adjustment in a non-uniform discrete action space and outputs the transmission rate adjustment amount. This, combined with an active queue management mechanism, triggers congestion signals, reduces the congestion window, and indirectly regulates the congestion window of the upstream transmitter. This method is applicable to various shared wireless backhaul scenarios. It requires no modification to terminal equipment or base station network infrastructure, and can significantly reduce network latency while maintaining high throughput efficiency and improving multi-stream concurrency fairness. Specifically, the method includes: A state-aware method based on a partially observable Markov decision process is constructed. The flow state-aware factors and time series are extracted. The state observation variables, action space and composite reward feedback of the gateway environment are defined. Network traffic observation data is collected and time series preprocessed. The flow state-aware factors are extracted and time series feature sequences are generated based on a hybrid memory construction mechanism. Design a non-uniform discrete action space and perform bidirectional rate dynamic adjustment under quadratic function constraints and non-uniform mapping. Input the temporal feature sequence into the reinforcement learning policy neural network and output the uplink and downlink bidirectional transmission rate adjustment ratio. A policy update method based on near-end policy optimization is adopted, combined with a continuous backpropagation mechanism to perform inefficient node reset under maturity protection; The edge gateway updates the rate control boundaries for uplink and downlink in real time according to the uplink and downlink bidirectional transmission rate adjustment ratio, forms visible queues and kernel visible queues on the gateway side, and actively triggers congestion signals to link the terminal-side transport layer congestion control; implements product reward function calculation, and performs elimination and reset of inefficient neurons.

[0008] Optionally, the method for extracting the flow state perception factor includes: Obtain the total number of active session connections at the current moment, calculate the deviation between it and the average number of connections within a preset historical sliding window, and map this deviation to a preset closed interval using a nonlinear bounded function, serving as a leading indicator for warning of concurrent business flow impacts.

[0009] Optionally, the hybrid memory construction mechanism includes: trigger determination, retention phase, suspension and reactivation phase, and reconstruction phase; trigger determination: when the number of active flows is detected to be lower than a preset proportion threshold of the historical window average within a first preset number of control cycles, the network link is determined to enter a low-load sleep period; retention phase: the second preset number of valid feature vectors before the low-load sleep trigger is captured and frozen to form a retained historical feature window; suspension and reactivation phase: during the sleep period, the reinforcement learning model parameter update is paused, and after the number of active flows is detected to rise above the preset proportion threshold, sampling is reactivated to capture the feature vectors in the initial recovery phase; reconstruction phase: the retained historical feature window and the feature vectors in the initial recovery phase are concatenated and recombined to restore the context dependency sequence across the sleep period.

[0010] Optionally, the uplink and downlink bidirectional transmission rate adjustment ratio is in a quadratic function constrained space with non-uniform density. The selection is as follows: The specific mapping relationship is constructed as follows: Among them, belonging to the set of integers Basic integer arithmetic sequence Its generation algorithm is as follows: and These are parameters calibrated based on the physical performance of the hardware; To ensure the bidirectional sign function of the adjustment direction; the control fine-tuning gears near the origin are denser using squared term mapping to provide fine-grained fine-tuning capabilities under steady-state conditions; when the reinforcement learning model selects an action... Then, the current rate limit. Perform a percentage multiplicative update: This enables nonlinear progressive modulation of the maximum transmission rate threshold for either the uplink or downlink.

[0011] Optionally, the composite reward feedback adopts a four-dimensional product mechanism, and its reward function is constructed by multiplying four components together, specifically including: The components include throughput utilization, which is positively correlated with current link bandwidth utilization; monotonically decreasing delay penalty, which is negatively correlated with the measured delay exceeding the preset target delay level; stability penalty, which is negatively correlated with the delay measurement timing standard deviation; and a balance coordination coefficient, which is negatively correlated with the absolute difference between uplink and downlink rate limits to constrain unidirectional resource occupancy.

[0012] Optionally, the continuous backpropagation mechanism performs the elimination and reset steps, including: The output contribution utility of a neuron is evaluated based on the product of the current activation intensity of each hidden layer neuron node in the reinforcement learning model and the sum of the absolute values ​​of the weights connected to its downstream network nodes. The historical utility is then smoothed over time using an exponential moving average. An online learning maturity count is maintained for each neuron. When the model is updated, neurons whose maturity exceeds a preset threshold and whose output contribution utility is at the bottom of the global hierarchy are selected according to a preset elimination ratio, their weights are unconnected, and their parameters are initialized.

[0013] The present invention has the following beneficial effects: This invention provides an invisible buffer control method for wireless communication environments. By employing state perception of a partially observable Markov decision process, bidirectional rate dynamic adjustment of a non-uniform discrete action space, a product reward function, and a continuous backpropagation mechanism, it achieves delay suppression, throughput maintenance, and fairness improvement in invisible buffer governance, thereby enhancing the rationality of invisible buffer control. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of an invisible buffer control method for wireless communication environments according to the present invention. Figure 2 This is a visual flowchart of an invisible buffer control method for wireless communication environments according to the present invention. Figure 3 This is a comparison diagram of the existing technical architecture and the architecture of this invention; Figure 4 This is a three-layer control flowchart of the present invention; Figure 5 This is a comparison chart of fairness and throughput in this invention; Figure 6 This is a box plot of the delay in a multi-load network according to the present invention. Detailed Implementation

[0015] In end-to-end links where the terminal device acts as the sender to the receiver, the common path using existing technical architectures is from the terminal device to the edge gateway to the receiver. In this path, an inconsistent state may occur on the upstream sender side, where the kernel-visible queue is visible but the downstream invisible buffer expands, accompanied by overflowing data packets. Simultaneously, for downlink data flows via the edge gateway, time-varying rate constraints can be applied at the gateway side to actively form a controlled queue between the visible queue and the kernel-visible queue. An active queue management mechanism can then proactively trigger congestion signals, causing the sender to converge its congestion window, thereby emptying the invisible buffer and maintaining the total queuing delay within the target range. This mechanism provides a feasible technical foundation for online rate regulation based on reinforcement learning.

[0016] This invention provides a method for controlling invisible buffers in wireless communication environments. Its core lies in transforming the uncontrollable buffer expansion problem into a rate modulation problem based on single-point observation by an edge gateway module and execution by a single-point intelligent control module. By actively implementing bidirectional traffic shaping at the gateway, the visible queue (controlled state) is transformed into a controlled queue, thereby triggering congestion signals, such as packet loss or explicit congestion notifications, inducing the transmitter to actively reduce the congestion window, thus clearing deep, invisible backlogs in advance. The technical solution adopted by this invention to solve its technical problem is as follows: First, to overcome the partial observability challenge posed by the invisible buffer, a partially observable Markov decision process state-aware method is proposed to extract the flow state-aware factor and the time series. Since the delay or throughput at a single moment cannot distinguish whether the network is in a steady state or experiencing rapid congestion, this model extracts past data. The state space is constructed using historical observation sequences of each control cycle, and a flow state sensing factor is introduced. It serves as a leading early warning indicator before bandwidth exhaustion. Its calculation algorithm is as follows: in This represents the total number of active Transmission Control Protocol (TCP) connections measured at the current moment. This is the local mean of the number of connections within a preset historical sliding window. Then, by applying... The algorithm mapping constrains the eigenvalues ​​to Within the closed interval. In addition, to address the issue of state sequence distortion caused by low network load intervals, a four-stage hybrid memory construction mechanism including retention, suspension, reactivation, and reconstruction is designed. When traffic sleep is detected, the effective feature vector is frozen and concatenated with the new state at a fixed length after waking up, thereby restoring the contextual causal dependencies before and after the traffic surge, forming a temporal feature sequence covering multiple control cycles.

[0017] Secondly, to address the issue of action divergence that easily occurs in reinforcement learning during dynamic control, a quadratic function constraint and non-uniform mapping mechanism based on a non-uniform discrete action space is designed and applied to bidirectional rate dynamic adjustment. The reinforcement learning policy neural network predicts the bidirectional transmission rate adjustment ratio, using a squared term mapping to densify the control fine-tuning gears near the origin, forming a fine-tuning interval; simultaneously, the outer large-span response space is sparsified to form a backoff interval for rapid backoff under extreme congestion conditions. After each decision, the gateway rate limit is updated by a percentage to obtain the transmission rate adjustment amount.

[0018] Next, to balance throughput, latency, and fairness in half-duplex media, a composite reward feedback (throughput, latency, fairness) mechanism is defined, and a product reward function r(t) is used to prevent the strategy from getting trapped in local optima in multi-dimensional objectives. Its form is: Among them, throughput component Incentives to maximize network bandwidth utilization; latency penalty component Based on a set target threshold, such as 80ms, a monotonically decreasing decay penalty is applied; stability multiplier. Periodic oscillations caused by containment and control strategies; balance coordination coefficient Penalize extreme one-way resource hoarding behavior between uplink and downlink.

[0019] Finally, the model is iteratively trained using the Proximal Policy Optimization (PPO) algorithm combined with a continuous backpropagation mechanism, and inefficient node resetting is performed under maturity protection. To address network non-stationarity and model parameter failure issues arising during online operation, the mechanism calculates the output contribution utility of neuron nodes online. : in, It is the attenuation factor; Indicates activation effect; Used to measure the weight of a signal propagating downstream. Specifies the reset rate. This control mechanism deweights and initializes neurons whose utility is measured to be extremely low within a certain step threshold during each update, thereby eliminating and resetting inefficient neurons. This operation is based on... The computational scale is increased to update the neural network representation state matrix, enabling the model to have the self-learning ability to adapt to fluctuations.

[0020] refer to Figure 1 The present invention illustrates the steps of an invisible buffer control method for wireless communication environments. Figure 2 This is an intuitive flow of an invisible buffer control method for wireless communication environments according to the present invention. Figure 3 This is a comparison diagram of the existing technical architecture and the architecture of this invention. (a) The existing technical architecture corresponds to the terminal device to the edge gateway, then to the base station / hardware driver, and finally to the receiving end. It shows that the expansion of the kernel visible queue and the invisible buffer coexist, and overflow data packets may occur. (b) The architecture of this invention deploys a reinforcement learning control module in the edge gateway, so that the kernel visible queue and the invisible buffer work together to realize the active triggering of congestion signals. Figure 4 This invention presents a three-layer control flowchart, comprising an environment observation layer, a reinforcement learning control layer, and an execution and feedback layer. The reinforcement learning control layer is also known as the reinforcement learning intelligent control layer. The environment observation layer covers multiple control cycles through a sliding window and forms a temporal feature sequence. The reinforcement learning intelligent control layer includes a reinforcement learning policy neural network, quadratic function constraints, non-uniform mapping, fine-tuning intervals, and extreme backoff intervals, and outputs the transmission rate adjustment amount. The execution and feedback layer includes actuator / transmission rate control, network / link environment, result observation, continuous backpropagation, elimination and reset of inefficient neurons, and a product reward function. Figure 5 This is a joint comparison chart of fairness and throughput. The horizontal axis represents the workload in terms of concurrent TCP sessions, the left axis represents Jain's Fairness Index (JFI), and the right axis represents the throughput (Mbps). The legend includes JFI of the prior art, JFI of the present invention, throughput of the prior art, and throughput of the present invention. The results illustrate the beneficial effect of the present invention in maintaining a high degree of fairness. Figure 6 This is a box plot of network latency under various loads, with network latency on the vertical axis. The comparison group includes existing technologies and the present invention. The load levels include 64 streams, 96 streams, and 128 streams, and the human-perceived latency threshold is marked. The results show that the present invention has the beneficial effect of reducing latency compared to existing technologies under various loads.

[0021] like Figure 1 As shown, an invisible buffer control method for wireless communication environments includes the following steps: Step S1: Construct a state-aware method based on a partially observable Markov decision process, extract flow state-aware factors and time series sequences, define state observation variables, action space and composite reward feedback of the gateway environment, collect network traffic observation data and perform time series preprocessing, extract flow state-aware factors and generate time series feature sequences based on a hybrid memory construction mechanism.

[0022] In some embodiments, a state-aware method based on a partially observable Markov decision process can be constructed, which extracts flow state-aware factors and time series sequences, defines state observation variables, action space, and composite reward feedback of the gateway environment, collects network traffic observation data and performs time series preprocessing, extracts flow state-aware factors, and generates time series feature sequences based on a hybrid memory construction mechanism.

[0023] The method for extracting the flow state perception factor includes: obtaining the total number of active session connections at the current moment, calculating the deviation between the current number of connections and the average number of connections within a preset historical sliding window, and mapping the deviation to a preset closed interval through a nonlinear bounded function as a leading indicator for warning of concurrent business flow impact.

[0024] The hybrid memory construction mechanism specifically includes trigger determination, retention phase, suspension and reactivation phase, and reconstruction phase. Trigger determination: When the number of active flows is detected to be lower than a preset percentage threshold of the historical window average within a first preset number of control cycles, the network link is determined to enter a low-load sleep period. Retention phase: A second preset number of valid feature vectors before the low-load sleep trigger are extracted and frozen to form a retained historical feature window. Suspension and reactivation phase: During the sleep period, reinforcement learning model parameter updates are paused. After the number of active flows is detected to rise above the preset percentage threshold, sampling is reactivated to capture the feature vectors from the initial recovery phase. Reconstruction phase: The retained historical feature window and the feature vectors from the initial recovery phase are concatenated and recombined to restore the context dependency sequence across the sleep period.

[0025] The composite reward feedback adopts a four-dimensional product mechanism. Its reward function is constructed by multiplying four components together, which include: a throughput utilization component that is positively correlated with the current link bandwidth utilization; a monotonically decreasing delay penalty component that is negatively correlated with the measured delay exceeding the preset target delay; a stability penalty component that is negatively correlated with the delay measurement timing standard deviation; and a balance coordination coefficient component that is negatively correlated with the absolute difference between the uplink and downlink rate limits to constrain unidirectional resource occupancy.

[0026] Step S2: Design a non-uniform discrete action space and perform bidirectional rate dynamic adjustment under quadratic function constraints and non-uniform mapping. Input the temporal feature sequence into the reinforcement learning policy neural network and output the uplink and downlink bidirectional transmission rate adjustment ratio.

[0027] In some embodiments, a non-uniform discrete action space can be designed, and bidirectional rate dynamic adjustment can be performed under quadratic function constraints and non-uniform mapping. The temporal feature sequence is input into the reinforcement learning policy neural network, and the uplink and downlink bidirectional transmission rate adjustment ratio and transmission rate adjustment amount are output.

[0028] Uplink and downlink bidirectional transmission rate adjustment ratio in a quadratic function constrained space with non-uniform density The selection is as follows, and the specific mapping relationship is constructed as follows: Among them, belonging to the set of integers Basic integer arithmetic sequence Its generation algorithm is as follows: and These are parameters calibrated based on the physical performance of the hardware; To ensure a bidirectional sign function for adjustment direction; the control fine-tuning gears near the origin are denser using a squared term mapping to provide fine-grained fine-tuning capabilities under steady-state conditions, while the outer large-span action space is sparsified for a larger operational space with extreme latency; when the reinforcement learning model selects an action... Then, the current rate limit. Perform a percentage multiplicative update: This enables nonlinear progressive modulation of the maximum transmission rate threshold for either the uplink or downlink.

[0029] Step S3: A policy update method based on near-end policy optimization is adopted, combined with a continuous backpropagation mechanism to perform inefficient node reset under maturity protection.

[0030] In some embodiments, a policy update method based on near-end policy optimization can be adopted, combined with a continuous backpropagation (CBP) mechanism to perform inefficient node reset under maturity protection, in order to maintain the activity of model parameters.

[0031] The CBP continuous backpropagation mechanism performs the following steps for elimination and reset: Based on the current activation intensity of each hidden layer neuron node in the reinforcement learning model and the product of the absolute values ​​of the weights connected to its downstream network nodes, the output contribution utility of the neuron is evaluated, and historical utility is smoothed over time using an exponential moving average; an online learning maturity count is maintained for each neuron, and during model updates, neurons with maturity exceeding a preset threshold and output contribution utility at the bottom of the global hierarchy are selected according to a preset elimination ratio, their weights are unconnected, and parameter initialization is performed.

[0032] Step S4: Update the rate control boundaries of the edge gateway for the uplink and downlink in real time according to the uplink and downlink bidirectional transmission rate adjustment ratio, form a visible queue and a kernel visible queue on the gateway side, and actively trigger a congestion signal to link the terminal side transmission layer congestion control; implement the product reward function calculation, and perform the elimination and reset of inefficient neurons.

[0033] In some embodiments, the rate control boundaries of the edge gateway for uplink and downlink can be updated in real time according to the rate adjustment ratio, forming visible queues and kernel visible queues on the gateway side, and actively triggering congestion signals to coordinate with the terminal-side transport layer congestion control; implementing product reward function calculation, and eliminating and resetting inefficient neurons.

[0034] This invention provides an invisible buffer control system for wireless communication environments, used to implement an invisible buffer control method for wireless communication environments, specifically including the following execution components: Environmental observation layer: used to extract flow state sensing factors and generate time-series feature sequences by covering multiple control cycles through a sliding window; Reinforcement learning intelligent control layer: includes reinforcement learning policy neural network, quadratic function constraint and non-uniform mapping unit, used to output transmission rate adjustment amount in fine-tuning interval and extreme back-off interval; Execution and Feedback Layer: Includes actuator / transmission rate control, network / link environment and result observation unit, used to form visible queues and kernel visible queues at the edge gateway, and actively trigger congestion signals; Composite reward feedback module: used to calculate the product reward function r(t) based on the controlled actual network latency and throughput; Continuous backpropagation module: used to perform elimination and reset of inefficient neurons under maturity protection conditions.

[0035] This embodiment is deployed on an edge gateway router, using ARM microarchitecture devices, and the control cycle is set to... To adapt to the feedback rhythm of the transport layer congestion control algorithm, the system link is: terminal device (transmitter) → edge gateway (CPE) → base station / hardware driver (wireless medium) → receiver. A reinforcement learning control module is set up in the edge gateway module to perform online rate control.

[0036] First, system deployment and initialization.

[0037] During system initialization, the parameters of the reinforcement learning policy neural network are loaded and used as the implementation of the reinforcement learning policy network. Uplink and downlink monitoring channels are established, and the environment observation layer, reinforcement learning intelligent control layer, and execution and feedback layer are initialized. During the initialization phase, the target latency threshold, action space boundary, sliding window length, maturity parameters, and replacement rate parameters in the continuous backpropagation mechanism are set for resetting inefficient nodes under maturity protection.

[0038] Second, environmental observation layer and state structure.

[0039] The environmental observation layer collects network observations during each control cycle, including , , , , .in, and These represent the uplink and downlink throughput, respectively. Indicates the time delay measurement value. and This represents the uplink and downlink flow state perception factors. By covering multiple control cycles through a sliding window, a temporal feature sequence is formed, and a partially observable Markov decision process is executed for state perception to extract the flow state perception factors and the temporal sequence.

[0040] The downlink flow state perception factor is calculated as follows: During the low-load sleep phase, a hybrid memory mechanism consisting of four stages—retention, suspension, reactivation, and reconstruction—is employed to ensure the continuity and effectiveness of the timing characteristic sequence.

[0041] Third, strengthen the intelligent control layer and rate regulation through learning.

[0042] The reinforcement learning intelligent control layer performs policy reasoning on the temporal feature sequence, executes bidirectional rate dynamic adjustment within a non-uniform discrete action space, and outputs the transmission rate adjustment amount. The action mapping employs quadratic function constraints and non-uniform mapping, with the area near the origin representing the fine-tuning interval and the outer large-action region representing the extreme backoff interval. The action space expression is: The system performs a multiplicative update on the current rate limit based on the output action: and the updated rate boundary Applied to edge gateway traffic shapers.

[0043] Fourth, the execution and feedback layer.

[0044] The execution and feedback layer includes executor / transmission rate control, network / link environment, and result observation. After the executor issues the rate boundary, the gateway side forms a visible queue and a kernel visible queue, and actively triggers a congestion signal through the active queue management mechanism. That is, it triggers a congestion signal, reduces the congestion window, and prompts the upstream sender to converge the congestion window, thereby suppressing the expansion of the invisible buffer and turning the invisible buffer from a backlogged state to a low-occupancy state.

[0045] The effectiveness of the strategy is evaluated using a composite reward feedback mechanism (throughput, latency, fairness), where the reward is defined as the product reward function r(t): During the continuous learning phase, the utility of neurons is calculated through continuous backpropagation, and inefficient neurons are eliminated and reset.

[0046] Fifth, performance verification of the implementation examples.

[0047] To verify the effectiveness of this invention, a comparative experiment was conducted under the 2.4GHz 802.11n protocol. The experimental platform was an ARM edge gateway device, which executed the RL intelligent rate modulation module online with a control cycle of 50ms. The workload was set to three levels of concurrent full-load TCP sessions: 64 streams, 96 streams, and 128 streams. The evaluation metrics included Jain's Fairness Index (JFI), throughput (Mbps), and network latency (Latency / ms).

[0048] The algorithm parameters used in this embodiment are as follows: the target delay threshold is set to... The maximum time delay measurement range is set to The control decision cycle is set to... The sliding window length for partially observable Markov decision processes is set to... The corresponding historical time-series context length is The action space boundaries are set to... , The reinforcement learning policy neural network employs an actor-critic algorithm with a shared encoding and branching output structure. Both the actor and critic networks use two hidden layers, each with 64 neurons. The continuous backpropagation mechanism uses the following parameters: the replacement rate is set to... The maturity threshold is set to 10,000 steps. The standard batch size for the online training phase is set to 2048. This combination of parameters is used to balance latency control accuracy, policy stability, and online adaptive capability on low-performance edge gateway devices.

[0049] The experiment compared the present invention with two existing control strategies, referred to as Existing Technology 1: FQ-CoDel algorithm and Existing Technology 2: ABC algorithm.

[0050] Fairness and throughput performance: such as Figure 5 As shown, across the full load range of 64, 96, and 128 streams, the JFI of this invention consistently remains above 0.99 while maintaining stable throughput. In contrast, the comparative method exhibits significant fairness degradation when congestion intensifies, with the JFI dropping to 0.92 in the worst case. This demonstrates that this invention can achieve synergistic optimization of throughput, latency, and fairness under high concurrency conditions.

[0051] Network latency performance: such as Figure 6 As shown, the latency distribution of this invention is significantly better than that of prior art 1 and prior art 2 under all three load levels. Under the highest load, the median network latency is reduced by more than 80%.

[0052] In summary, by comprehensively applying the above technical solutions, we achieve synergistic optimization of invisible buffer latency suppression, throughput maintenance, and multi-stream fairness. Experimental results show that in environments with various wireless network technologies, median latency can be reduced by more than 80%, the fairness index remains above 0.99, and it can be directly deployed under existing transport layer protocols. The lightweight model can run on low-computing-power gateway devices without requiring modifications to terminal equipment or operator network infrastructure.

Claims

1. A method for controlling an invisible buffer in a wireless communication environment, characterized in that, Includes the following steps: A state-aware method based on a partially observable Markov decision process is constructed. The flow state-aware factors and time series are extracted. The state observation variables, action space and composite reward feedback of the gateway environment are defined. Network traffic observation data is collected and time series preprocessed. The flow state-aware factors are extracted and time series feature sequences are generated based on a hybrid memory construction mechanism. Design a non-uniform discrete action space and perform bidirectional rate dynamic adjustment under quadratic function constraints and non-uniform mapping. Input the temporal feature sequence into the reinforcement learning policy neural network and output the uplink and downlink bidirectional transmission rate adjustment ratio. A policy update method based on near-end policy optimization is adopted, combined with a continuous backpropagation mechanism to perform inefficient node reset under maturity protection; The edge gateway updates the rate control boundaries for uplink and downlink in real time according to the uplink and downlink bidirectional transmission rate adjustment ratio, forms visible queues and kernel visible queues on the gateway side, and actively triggers congestion signals to link the terminal-side transport layer congestion control; implements product reward function calculation, and performs elimination and reset of inefficient neurons.

2. The invisible buffer control method for wireless communication environments according to claim 1, characterized in that, Methods for extracting flow state sensing factors include: Obtain the total number of active session connections at the current moment, calculate the deviation between it and the average number of connections within a preset historical sliding window, and map this deviation to a preset closed interval using a nonlinear bounded function, serving as a leading indicator for warning of concurrent business flow impacts.

3. The invisible buffer control method for wireless communication environments according to claim 1, characterized in that, The hybrid memory construction mechanism includes: trigger determination, retention phase, suspension and reactivation phase, and reconstruction phase. Trigger determination: When the number of active flows is detected to be lower than a preset proportion threshold of the historical window average within a first preset number of control cycles, the network link is determined to enter a low-load sleep period. Retention phase: The second preset number of valid feature vectors before the low-load sleep trigger is captured and frozen to form a retained historical feature window. Suspension and reactivation phase: During the sleep period, the reinforcement learning model parameter update is paused. After the number of active flows is detected to rise above the preset proportion threshold, sampling is reactivated to capture the feature vectors in the initial recovery phase. Reconstruction phase: The retained historical feature window and the feature vectors in the initial recovery phase are concatenated and recombined to restore the context dependency sequence across the sleep period.

4. The invisible buffer control method for wireless communication environments according to claim 1, characterized in that, Uplink and downlink bidirectional transmission rate adjustment ratio in a quadratic function constrained space with non-uniform density The selection is as follows: The specific mapping relationship is constructed as follows: Among them, belonging to the set of integers Basic integer arithmetic sequence Its generation algorithm is as follows: and These are parameters calibrated based on the physical performance of the hardware; To ensure the bidirectional sign function of the adjustment direction; the control fine-tuning gears near the origin are denser using squared term mapping to provide fine-grained fine-tuning capabilities under steady-state conditions; when the reinforcement learning model selects an action... Then, the current rate limit. Perform a percentage multiplicative update: This enables nonlinear progressive modulation of the maximum transmission rate threshold for either the uplink or downlink.

5. The invisible buffer control method for wireless communication environments according to claim 1, characterized in that, The composite reward feedback adopts a four-dimensional product mechanism, and its reward function is constructed by multiplying four components together, specifically including: The components include throughput utilization, which is positively correlated with current link bandwidth utilization; monotonically decreasing delay penalty, which is negatively correlated with the measured delay exceeding the preset target delay level; stability penalty, which is negatively correlated with the delay measurement timing standard deviation; and a balance coordination coefficient, which is negatively correlated with the absolute difference between uplink and downlink rate limits to constrain unidirectional resource occupancy.

6. The invisible buffer control method for wireless communication environments according to claim 1, characterized in that, The continuous backpropagation mechanism executes the elimination and reset steps, including: The output contribution utility of a neuron is evaluated based on the product of the current activation intensity of each hidden layer neuron node in the reinforcement learning model and the sum of the absolute values ​​of the weights connected to its downstream network nodes. The historical utility is then smoothed over time using an exponential moving average. An online learning maturity count is maintained for each neuron. When the model is updated, neurons whose maturity exceeds a preset threshold and whose output contribution utility is at the bottom of the global hierarchy are selected according to a preset elimination ratio, their weights are unconnected, and their parameters are initialized.

Images