A real-time communication system and method suitable for non-stationary network environments.
By combining the NS_Transformer framework and the Actor-Critic algorithm, the dynamic switching strategy solves the problems of bandwidth prediction and bitrate control in non-stationary network environments, achieving efficient and low-energy bandwidth prediction and bitrate control, alleviating the cold start problem, and improving the real-time response capability of Web RTC.
Patent Information
- Application Number
- CN202410080888.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-01-19
AI Technical Summary
Traditional bandwidth prediction methods are poor at handling the non-stationarity of the highly heterogeneous modern Internet. Transformer-based methods are computationally intensive and energy-intensive, while reinforcement learning methods suffer from cold start and overestimation of value during training and decision-making, affecting real-time response in Web RTC scenarios.
Combining a bandwidth prediction model, training module, compression module, knowledge transfer module, and bitrate control module based on the NS_Transformer framework, the bandwidth prediction model is trained using the NS_Transformer framework. Knowledge distillation is performed using a recurrent neural network. The Actor-Critic algorithm and the heuristic Web RTC built-in congestion control GCC algorithm are combined to dynamically switch strategies to cope with non-stationary network environments.
It achieves efficient bandwidth prediction and bit rate control in non-stationary network environments, reduces computational load and energy consumption, alleviates cold start problems, and improves the accuracy of bandwidth prediction and real-time response capabilities.
Smart Images

Figure CN117749775B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of streaming video transmission, and in particular to a real-time communication system and method for non-stationary network environments. Background Technology
[0002] Users of real-time communication (Web RTC) expect continuously improving Quality of Experience (QoE) in a faster network environment. In the field of video transmission, Adaptive Rate Modulation (ABR) algorithms are a hot research topic and a key technology for improving video transmission efficiency. Popular ABR algorithms are based on network bandwidth prediction, but the modern internet is highly heterogeneous, with dynamically changing network bandwidth, making bandwidth prediction far from ideal in terms of accuracy. It is worth noting that changes in network conditions amplify the non-stationarity of the bandwidth sequence itself, which is even more pronounced in the highly heterogeneous modern internet with its coexistence of 3G / 4G / 5G and cellular / WiFi wireless links, significantly reducing the accuracy of bandwidth prediction algorithms in real-world networks. Furthermore, compared to video-on-demand / live streaming, Web RTC has a lower tolerance for bandwidth fluctuations and demands higher accuracy in bandwidth prediction.
[0003] To date, researchers have proposed many effective real-time video transmission algorithms, among which reinforcement learning (RL) methods have demonstrated great potential, but they also have their own significant shortcomings. Reinforcement learning starts with trial and error and gradually makes more reasonable decisions based on experience, but the initial trial and error may cause sudden performance fluctuations in the early stages of the algorithm's operation, which can seriously affect transmission quality—the so-called "cold start" problem. At the same time, the Temporal Difference (TD) algorithm commonly used in reinforcement learning often overestimates the true value of actions, causing bias in the results and affecting the subsequent learning process.
[0004] In summary, the present invention urgently needs to solve the following technical problems:
[0005] Traditional bandwidth prediction methods (such as LSTM-based machine learning methods) are poor at handling the non-stationarity of the highly heterogeneous modern Internet. Transformer-based methods, which perform well in predicting time series data, are computationally intensive and energy-intensive, making them unsuitable for Web RTC scenarios requiring real-time responses. Existing popular reinforcement learning methods also suffer from cold start problems and overestimation of value during training and decision-making. Summary of the Invention
[0006] To address the technical shortcomings of traditional bandwidth prediction methods in handling the non-stationarity of the highly heterogeneous modern Internet, this invention aims to propose a real-time communication system and method suitable for non-stationary network environments. For Web RTC in non-stationary network environments, an algorithm combining a reinforcement learning-based RL model with the heuristic-based traditional congestion control GCC algorithm built into Web RTC is used to obtain a rate control strategy.
[0007] This invention is achieved using the following technical solution:
[0008] A real-time communication system suitable for non-stationary network environments is characterized by comprising a bandwidth prediction model based on the NS_Transformer framework, a training module, a compression module, a knowledge transfer module, and a bit rate control module.
[0009] The training module trains a bandwidth prediction model based on the NS_Transformer framework. It uses massive amounts of network bandwidth data as input to the NS_Transformer prediction model and supervises learning by using subsequent bandwidth data outside the current time window as labels to obtain the trained NS_Transformer prediction model. This module further includes a preprocessing unit for preprocessing and batch partitioning the collected massive network bandwidth tracking dataset, collecting network bandwidth data from the preprocessed dataset, and generating a network bandwidth configuration file. Finally, it includes a prediction model unit for constructing a bandwidth prediction model based on the NS_Transformer framework. The model, a bandwidth prediction model based on the NS_Transformer framework, takes time series data of bandwidth, packet loss rate, and latency as input. The prediction model unit further includes a stabilization subunit, which performs normalization on the input of the bandwidth prediction model based on the NS_Transformer framework in the first stage, normalizing the time series data of network state variables, including network bandwidth, packet loss rate, and latency, respectively, and then using them as input again. In the second stage, it performs inverse normalization on the output of the bandwidth prediction model based on the NS_Transformer framework to recover the distribution information lost during normalization. The destabilization subunit utilizes an attention mechanism to achieve destabilization for non-stationary time series data.
[0010] The compression module is used to take the prediction model based on the NS_Transformer framework trained by the training module as the teacher network, set a recurrent neural network model (RNN) as the student network, and use the RNN to learn knowledge distillation extraction from the trained prediction model based on the NS_Transformer framework to obtain a lightweight prediction model based on the NS_Transformer framework.
[0011] The knowledge transfer module is used to share the neural network parameters of the lightweight prediction model based on the NS_Transformer framework obtained by the compression module with the Actor neural network, complete the knowledge transfer learning based on neural network fine-tuning, and obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge, in which the Actor neural network performs code rate decision and the Critic neural network performs state value calculation.
[0012] The bitrate control module is used to train the RL model containing bandwidth prediction knowledge obtained after knowledge transfer from the knowledge transfer module, and to design a bitrate control strategy by combining the reinforcement learning-based RL model with the traditional congestion control GCC algorithm built into the heuristic Web RTC to achieve bandwidth prediction.
[0013] The batch partitioning process includes randomly sampling the network bandwidth tracking dataset, calculating the average and variance of the data bandwidth, and dividing it into two categories according to the median of the average: within each dataset, the dataset with higher bandwidth is set as the first dataset, and the dataset with lower bandwidth is split into the second dataset.
[0014] The configuration file is used to perform the following processes:
[0015] Within a delay range of 0-200ms, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.8 and 0.2 respectively. Within a packet loss rate of 0-20%, random samples are taken according to a normal distribution with a mean of 4% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset A.
[0016] Within a delay range of 0-200ms, random sampling is performed directly. Within a packet loss rate of 0-20%, random sampling is performed using a normal distribution with a mean of 10% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset B and dataset C.
[0017] Within a delay range of 0-200ms, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.2 and 0.8 respectively. Within a packet loss rate of 0-20%, random samples are taken according to a normal distribution with a mean of 16% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset D.
[0018] The configuration file is used to perform the following processes:
[0019] Dataset A has relatively high bandwidth and weak non-stationarity; Dataset B has relatively high bandwidth and strong non-stationarity; Dataset C has relatively low bandwidth and weak non-stationarity; and Dataset D has relatively low bandwidth and strong non-stationarity.
[0020] The bit rate control module further includes:
[0021] The Actor network in the applied Actor-Critic algorithm has shared the knowledge of the bandwidth prediction model based on the NS_Transformer framework through the knowledge transfer module of the bandwidth prediction model based on the NS_Transformer framework. It is trained in a simulated Web RTC system environment, and the reward function is set as follows:
[0022]
[0023] Where N represents a time window W t The number of RTP packets within, q n This represents the throughput measured at the receiving end, l n d n The last term represents the packet loss rate and latency at the transport layer. Used to enhance video smoothness by penalizing drastic bitrate fluctuations, each term is preceded by hyperparameters α, β, η. Represents non-negative numbers;
[0024] Meanwhile, the trajectory data (s,a,r,s') obtained by the interaction between the Actor and the Web RTC system environment is stored in the playback buffer for gradient calculation and updating the Critic network parameters;
[0025] This paper combines a reinforcement learning-based RL model with the traditional congestion control (GCC) algorithm built into Web RTC, based on heuristics, and applies it to a real-world network environment to achieve bit rate control prediction. The specific strategy for dynamically switching between the two algorithms is as follows:
[0026] A network detector based on the Transport CC algorithm of Web RTC is constructed to detect network status, calculate the coefficient of variation (CV), and dynamically set γ. tThis represents the network congestion threshold at time t. When the network condition detector detects increased network congestion, it enters an algorithm switching state. This state is triggered when the CV value is greater than or equal to the threshold γ. t Switch to the GCC algorithm when the CV value is less than the threshold γ. t A reinforcement learning RL neural network is used when network congestion is not aggravated.
[0027] A real-time communication method suitable for non-stationary network environments includes:
[0028] Step 1: Train the bandwidth prediction model based on the NS_Transformer framework. Use massive amounts of network bandwidth data as input to the prediction model, and use subsequent bandwidth data outside the current time window as labels for supervised learning to obtain the trained prediction model based on the NS_Transformer framework. This module further includes the preprocessing unit, used to preprocess and batch-divide the collected massive network bandwidth tracking dataset, collect network bandwidth data from the preprocessed network bandwidth tracking dataset, and generate a network bandwidth configuration file; and the prediction model unit, used to construct the bandwidth prediction model based on the NS_Transformer framework, based on NS_Transformer... The bandwidth prediction model of the NS_Transformer framework takes as input time series data of bandwidth, packet loss rate, and latency. The prediction model unit further includes a stabilization subunit, which performs normalization of the input to the bandwidth prediction model based on the NS_Transformer framework in the first stage, normalizing the time series data of network state variables, including network bandwidth, packet loss rate, and latency, respectively, and then using these normalized data as input again. In the second stage, it performs inverse normalization of the output of the bandwidth prediction model based on the NS_Transformer framework to recover the distribution information lost during normalization. Finally, the destableization subunit utilizes an attention mechanism to destableize non-stationary time series data.
[0029] Step 2: Use the bandwidth prediction model based on the NS_Transformer framework trained by the training module as the teacher network, set a recurrent neural network (RNN) as the student network, and use the RNN to learn knowledge distillation extraction from the trained bandwidth prediction model based on the NS_Transformer framework to obtain a lightweight bandwidth prediction model based on the NS_Transformer framework.
[0030] Step 3: Share the compressed neural network parameters based on the NS_Transformer framework with the Actor neural network to complete the knowledge transfer learning based on neural network fine-tuning, and obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge. In this model, the Actor neural network makes code rate decisions, while the Critic neural network calculates state value.
[0031] Step 4: Train the machine learning RL model containing bandwidth prediction knowledge obtained in Step 3. Combine the reinforcement learning-based RL model with the traditional congestion control GCC algorithm built into Web RTC based on heuristics to design a bitrate control strategy to achieve bandwidth prediction.
[0032] Compared with existing technologies, the bandwidth prediction results output by this invention take into account the non-stationarity and time-series dependence of the input data, and achieve a good prediction level on massive network bandwidth data with non-stationarity characteristics. Attached Figure Description
[0033] Figure 1 This is a block diagram of a real-time communication system applicable to non-stationary network environments according to the present invention;
[0034] Figure 2 This is a flowchart illustrating the implementation process of a real-time communication method applicable to non-stationary network environments according to the present invention.
[0035] Figure 3 Here is a diagram of the NS_Transformer framework structure;
[0036] Figure 4 This is a structural diagram of the knowledge distillation module of the prediction model compression module of the present invention;
[0037] Figure 5 This is a structural diagram of the knowledge transfer module of the prediction model of the present invention;
[0038] Figure 6 This is a structural diagram of the reinforcement learning algorithm (actor-critic algorithm) of the present invention;
[0039] Figure 7 This is a structural diagram of the bitrate control module of the present invention;
[0040] Figure label:
[0041] 100. Training module; 200. Compression module; 300. Knowledge transfer module; 400. Bitrate control module. Detailed Implementation
[0042] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.
[0043] To better illustrate the present invention, the technical solution will now be further described in conjunction with specific embodiments and the accompanying drawings.
[0044] like Figure 1 and Figure 2 The present invention illustrates a real-time communication system suitable for non-stationary network environments, comprising a bandwidth prediction model based on the NS_Transformer (Non-stationary Transformer) framework, a training module 100, a compression module 200, a knowledge transfer module 300, and a bitrate control module 400.
[0045] The training module 100 uses a prediction model based on the NS_Transformer (Non-stationary Transformer) framework for training. It takes massive network bandwidth data as input to the prediction model and uses subsequent bandwidth data outside the current time window as labels for supervised learning to obtain a trained prediction model.
[0046] The problem of predicting non-stationary network bandwidth data is solved by combining the sequence stabilization module and the destabilization attention module in the NS_Transformer framework.
[0047] The training module 100 further includes a preprocessing unit 110 and a prediction model unit 120.
[0048] The preprocessing unit 110 preprocesses and batches the collected massive network bandwidth tracking dataset. The preprocessing includes at least data cleaning (removing duplicate data, handling missing values and outliers, etc.), data integration (including data column concatenation and row concatenation, etc.) and data transformation (including feature encoding, standardization and normalization, etc.). Network bandwidth data is collected from the preprocessed network bandwidth tracking dataset, and the network bandwidth data is unified into Mbits / s to generate a network bandwidth configuration file.
[0049] Furthermore, loss and latency were set for each configuration file, with latency between 0-200ms and loss between 0-20%; the preprocessed dataset was randomly divided into two groups: 80% for training and 20% for testing.
[0050] The batch partitioning involves randomly sampling the network bandwidth tracking dataset, calculating the average and variance of the data bandwidth, and dividing it into two categories according to the median of the average: within each dataset, the dataset with higher bandwidth is set as the first dataset, and the dataset with lower bandwidth is split into the second dataset.
[0051] Since the variance of a dataset represents a certain degree of stationarity, the first and second datasets are further divided into datasets with weak and strong non-stationarity, respectively, based on the median variance. This results in four categories of datasets: ① Dataset A, with relatively high bandwidth and weak non-stationarity; ② Dataset B, with relatively high bandwidth and strong non-stationarity; ③ Dataset C, with relatively low bandwidth and weak non-stationarity; and ④ Dataset D, with relatively low bandwidth and strong non-stationarity.
[0052] The average bandwidth and stability of the network are directly related to the latency and packet loss rate during data transmission. Higher and more stable bandwidth results in lower latency and packet loss rates. Therefore, in simulating a real-world environment, the configuration file is used to perform the following processing:
[0053] 1) Within the 0-200ms delay range, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.8 and 0.2 respectively. Within the 0-20% packet loss rate, random samples are taken according to a normal distribution with a mean of 4% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset A.
[0054] 2) Within a delay range of 0-200ms, random sampling is performed directly. Within a packet loss rate of 0-20%, random sampling is performed using a normal distribution with a mean of 10% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset B and dataset C.
[0055] 3) Within the 0-200ms delay range, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.2 and 0.8 respectively. Within the 0-20% packet loss rate, random samples are taken according to a normal distribution with a mean of 16% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset D.
[0056] A more informative network state dataset was generated by executing the new configuration file. This dataset will be used as the training and testing object in the subsequent model training process.
[0057] The prediction model unit 120 is used to construct a bandwidth prediction model based on the Non-stationary Transformer framework. The input of the prediction model is the time series of bandwidth, packet loss rate, and latency (rate, loss, delay). The input Tansformer window size and the prediction Tansformer window size are both set to 32. The number of attention heads is set to 8, the number of encoder layers is set to 2, the number of decoder layers is set to 1, the fully connected dimension is set to 2048, the initial learning rate is set to 0.01, an exponential decay strategy is adopted, and the model is trained using the MSE loss function and the Adam optimizer. The test set is used to test the fit of the bandwidth prediction model until the bandwidth prediction model is a model with high accuracy and prediction ability.
[0058] The prediction model unit 120 further includes a stabilization processing subunit 1201 and a destabilization processing subunit 1202:
[0059] The stabilization processing subunit 1201 is divided into two stages. The first stage performs normalization, which normalizes the time series of various network state variables—network bandwidth, packet loss rate, and latency—into their respective time dimensions. This normalized input eliminates scale differences between variables in multivariate time series data. The second stage performs de-normalization, storing the original mean and variance of the sequences within each window. These statistics are then used to re-scale the model's output to recover the distribution information lost during normalization. During the de-scale transformation, the output y' is obtained using the following formula:
[0060] y'=y·σ x +μ x (1)
[0061] Where y is the original output of the prediction model, σ x μ is the standard deviation of the original time series. x This is the mean of the original time series.
[0062] The de-stabilization subunit 1202 utilizes an attention mechanism to de-stabilize non-stationary time series data. This subunit designs a de-stabilization attention mechanism for non-stationary time series data; it uses the normalized input and the statistics (mean and variance) stored during normalization to approximate the attention map that would be obtained with the original input before normalization. The final attention result uses the Transformer's calculation formula, as shown below:
[0063]
[0064] Where V represents the vector value of the attention map, and d k The hyperparameters represent the hidden layer dimensions. The update result is obtained from the inputs Q and K using the following formula:
[0065]
[0066] Where, σ x μ represents the standard deviation of the input series. Q and μ K Let Q and K represent the mean values of Q and K respectively, and 1 represents a vector of all 1s.
[0067] The final expression is shown below:
[0068]
[0069] like Figure 3 The diagram shows the structure of the NS_Transformer framework. The NS_Transformer framework includes a normalization layer and an inverse normalization layer for sequence stabilization: In the first stage, normalization is used to normalize each variable of the network state along the time dimension, eliminating scale differences between different variables in multivariate time-series data. In the second stage, the model's output is rescaled using training-related statistics to recover the distribution information lost during normalization. Finally, the NS_Transformer output prediction result is obtained after considering the non-stationarity and temporal dependencies of the input data. This invention chooses to build a bandwidth prediction model based on the NS_Transformer framework. By training this model, bandwidth prediction is achieved, thus enabling better prediction of changes in network state.
[0070] The compression module 200 is used to take the prediction model trained by the training module 100 as the teacher network and set a recurrent neural network (RNN) as the student network. The RNN learns knowledge distillation from the trained prediction model to obtain a compressed prediction model. Specifically, the RNN model and the prediction model jointly handle the same problem, and the parameters of the RNN model are adjusted to mimic the decision-making of the prediction model. This achieves both allowing the RNN model to acquire knowledge from the parameters of the prediction model and realizing a lightweight prediction model. This mainstream method of model lightweighting maintains good predictive ability while making the prediction model lightweight, reducing its resource consumption, improving running speed, and better adapting to Web RTC scenarios. Figure 4The diagram shown illustrates the structure of the knowledge distillation module in the prediction model compression module of this invention. This structure includes a teacher network and a student network, gradually allowing the RNN to learn the knowledge of the NS_Transformer framework prediction model, thereby achieving good predictive ability and high accuracy. As the student network gradually acquires the teacher's knowledge, the performance of both networks gradually becomes similar.
[0071] The knowledge transfer module 300 is used to share the neural network parameters of the lightweight prediction model obtained by the compression module 200 with the Actor neural network. That is, to share the network parameters of the recurrent neural network (RNN) with the Actor neural network parameters of the Actor-Critic algorithm in the field of reinforcement learning (RL), and complete the knowledge transfer learning based on neural network fine-tuning to obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge. In this model, the Actor neural network makes code rate decisions, while the Critic neural network calculates state values.
[0072] like Figure 5 The diagram shows the structure of the knowledge transfer module of the prediction model of this invention. The task of the prediction model is network bandwidth prediction, and the ultimate goal is real-time transmission bitrate decision-making; therefore, their actual tasks are different. s ≠T t However, it is noted that both have similar source domains D. s =D t Therefore, although the two tasks address different problems, it's easy to see that their model inputs and outputs are similar, assuming that the source and target tasks share common knowledge at the model level. The algorithm used in Deep Reinforcement Learning (DRL) is the Actor-Critic algorithm, where the Actor network is responsible for fitting the bitrate selection action policy. Therefore, the parameters of the trained prediction model neural network are shared with the Actor neural network, enabling transfer learning based on neural network fine-tuning. With the knowledge of the already trained model, the Actor neural network can perform better in decision-making, while the Actor-Critic algorithm can converge quickly in the training environment and better predict non-stationary bandwidth data.
[0073] The bitrate control module 400 is used to train an RL model containing bandwidth prediction knowledge, obtained after knowledge transfer from the prediction model knowledge transfer module 300. It then simulates a Web RTC scenario using existing data and outputs a bitrate control strategy. This strategy aims to maximize user experience quality and improve bitrate decision-making capabilities. To completely solve the cold start problem, a more robust bitrate decision-making algorithm is constructed by combining an RL model with heuristic methods to handle more extreme situations. Specifically, the bitrate control process using a hybrid learning-based and heuristic approach involves: in cases of rapid network fluctuations, a more conservative heuristic-based GCC algorithm is used to maintain the lower limit of bitrate control; in other cases, a learning-based method is preferred to aim for the upper limit of bitrate control. The two methods are dynamically switched according to the actual scenario.
[0074] like Figure 7 As shown, the bitrate control module of this invention is based on a combination of learning and heuristic methods. By dynamically switching between the two methods, the robustness and prediction accuracy of the bitrate decision algorithm are enhanced.
[0075] First, a network detector is built to detect the network status based on the Transport CC algorithm in Web RTC. This algorithm determines the current network congestion by detecting the increase or decrease of the slope of the received delay sequence within the detection window, and obtains the network status at the current moment.
[0076] Specifically, the interval sequence Δd(t) between data packets is used. i () represents the packet queue on the network path, defined as follows:
[0077] Δd(t i )=(a i -a i-1 )-(s i -s i-1 (5)
[0078] Among them, a i and s i These represent the i-th state and the selected action of a data packet within a time window of t, respectively. The historical sequence within a time window is represented as Δd(t0), Δd(t1), ..., Δd(t2). T T represents the total number of time windows. The sequence is then passed to a trendline filter, and the window period W is calculated using the least squares method. t The slope K of the delay difference d sequence within the interval t If K t Greater than the threshold γ tIf this indicates that network congestion is imminent, the algorithm will switch to the congestion control algorithm (GCC) built into Web RTC; otherwise, it will continue to control transmission.
[0079] In addition, the threshold γ is dynamically adjusted. t To make the algorithm more adaptable to dynamically changing network environments and avoid extreme cases caused by fixed thresholds, the adaptive threshold adjustment algorithm used is as follows:
[0080] γ(t i )=γ(t i-1 )+ΔT·k γ (t i )(|m(t i )|-γ(t i-1 (6)
[0081] Where, k γ Describe a constant to determine γ(t) i The rate at which the threshold increases or decreases. When the delay gradient decreases, the threshold decreases at a slower rate; when the delay gradient increases, the threshold also increases at a slower rate, making it more adaptable to dynamically changing network environments.
[0082] Given the current network changes, an adaptive selector is built to choose between learning-based and heuristic methods. This module is entered when the network condition detector detects increased network congestion.
[0083] First, calculate the coefficient of variation (CV) of the GCC and RL predictions to observe the degree of deviation between the two algorithms. If the CV value is greater than the threshold β, the RL prediction differs too much from the GCC prediction, while GCC tends to make conservative decisions and is more suitable for use in unstable network conditions to ensure that the user experience does not deteriorate significantly. If the CV is within the threshold β, it can be considered that the predictions of GCC and RL are not significantly different. Under the premise of ensuring that the user experience does not deteriorate significantly, it is preferable to use the RL algorithm to learn from a more severe environment. The knowledge transfer module 300 of the prediction model retains the output of RL and transfers the knowledge of the lightweight prediction model after knowledge distillation in the compression module 200 of the prediction model to the reinforcement learning model. The two models have the same input data and can share knowledge. At the same time, the reinforcement learning model with prediction knowledge alleviates the cold start problem of the initial training of the model. That is, the final reinforcement learning model initially contains the knowledge of bandwidth prediction to make bitrate decisions. The specific steps of this module are as follows:
[0084] The present invention provides a real-time communication method applicable to non-stationary network environments, comprising the following steps:
[0085] Step 1: Train the prediction model using the NS_Transformer (Non-stationary Transformer) framework. Input massive amounts of network bandwidth data into the prediction model and use subsequent bandwidth data from the time series as labels for supervised learning to obtain the trained prediction model. The specific process is as follows:
[0086] Step 1-1: Preprocess and batch-divide the collected massive network bandwidth tracking dataset. The preprocessing includes at least data cleaning (removing duplicate data, handling missing and outliers, etc.), data integration (including column and row concatenation, etc.), and data transformation (including feature encoding, standardization, and normalization, etc.). Network bandwidth data is collected from the preprocessed dataset, and the unit of network bandwidth data is standardized to Mbits / s to generate network bandwidth configuration files. Furthermore, loss and latency are set for each configuration file, with latency between 0-200ms and loss between 0-20%. The preprocessed dataset is randomly divided into two groups: 80% for training and 20% for testing.
[0087] The batch partitioning involves randomly sampling the network bandwidth tracking dataset, calculating the average and variance of the data bandwidth, and dividing it into two categories according to the median of the average: within each dataset, the dataset with higher bandwidth is set as the first dataset, and the dataset with lower bandwidth is split into the second dataset.
[0088] Since the variance of a dataset represents a certain degree of stationarity, the first and second datasets are further divided into datasets with weak and strong non-stationarity, respectively, based on the median variance. This results in four categories of datasets: ① Dataset A, with relatively high bandwidth and weak non-stationarity; ② Dataset B, with relatively high bandwidth and strong non-stationarity; ③ Dataset C, with relatively low bandwidth and weak non-stationarity; and ④ Dataset D, with relatively low bandwidth and strong non-stationarity.
[0089] The average bandwidth and stability of the network are directly related to the latency and packet loss rate during data transmission. Higher and more stable bandwidth results in lower latency and packet loss rates. Therefore, in simulating a real-world environment, the configuration file is used to perform the following processing:
[0090] 1) Within the 0-200ms delay range, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.8 and 0.2 respectively. Within the 0-20% packet loss rate, random samples are taken according to a normal distribution with a mean of 4% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset A.
[0091] 2) Within a delay range of 0-200ms, random sampling is performed directly. Within a packet loss rate of 0-20%, random sampling is performed using a normal distribution with a mean of 10% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset B and dataset C.
[0092] 3) Within the 0-200ms delay range, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.2 and 0.8 respectively. Within the 0-20% packet loss rate, random samples are taken according to a normal distribution with a mean of 16% and a standard deviation of 0.02. The sampling results are summarized and saved as dataset D.
[0093] A more informative network state dataset was generated by executing the new configuration file. This dataset will be used as the training and testing object in the subsequent model training process.
[0094] Steps 1-2: Construct a prediction model based on the Non-stationary Transformer framework. The input to the prediction model is the time series of bandwidth, packet loss rate, and latency (rate, loss, delay). The input Tansformer window size and the prediction Tansformer window size are both set to 32. The number of attention heads is set to 8, the number of encoder layers is set to 2, the number of decoder layers is set to 1, the fully connected dimension is set to 2048, the initial learning rate is set to 0.01, and an exponential decay strategy is adopted. The prediction model is trained using the MSE loss function and the Adam optimizer. The test set is used to test the goodness of fit of the prediction model. The model has high accuracy and prediction ability.
[0095] Furthermore, steps 1-2 also include stabilization processing and destabilization attention processing:
[0096] The stabilization process consists of two stages. First, normalization is performed, which normalizes the time series of network state variables—bandwidth, packet loss rate, and latency—into their respective time dimensions. This normalized input eliminates scale differences between variables in multivariate time series data. The second stage is de-normalization, which stores the original mean and variance of the sequences within each window. These statistics are then used to perform a reverse scaling transformation on the model's output to recover the distribution information lost during normalization. The output y' of the reverse scaling transformation is obtained using the following formula:
[0097] y'=y·σ x +μ x (7)
[0098] Where y is the original output of the prediction model, σ x μ is the standard deviation of the original time series. x This is the mean of the original time series.
[0099] The de-stationarization process employs a de-stationarization attention mechanism designed for non-stationary time series. It uses the normalized input and the statistics (mean and variance) stored during normalization to approximate the attention map that would be obtained with the original input before normalization. The final attention result is calculated using the Transformer formula, as shown below:
[0100]
[0101] Where V represents the vector value of the attention map, and d k The hyperparameters represent the hidden layer dimensions. The update result is obtained from the inputs Q and K using the following formula:
[0102]
[0103] Where, σ x μ represents the standard deviation of the input time series. Q and μ K Let Q and K represent the means of Q and K respectively, and 1 represent a vector of all 1s. The final expression is shown below:
[0104] ;
[0105] Step 2: Use the trained prediction model as the teacher network and set up a recurrent neural network (RNN) as the student network. Gradually use the RNN to learn knowledge distillation and extraction from the pre-trained prediction model based on the NS_Transformer framework, and perform knowledge transfer with the prediction model in the input domain. Continuously adjust the network parameters of the RNN model so that the RNN model can imitate the decision-making of the teacher network, so that it can make the same decision as the teacher network in the same scenario, in order to obtain the knowledge in the teacher network parameters. This results in a trained RNN model and a compressed lightweight prediction model, which is a lightweight model that meets the needs of real-time audio and video (Web RTC) scenarios.
[0106] This step employs the Sequence-Level Knowledge Distillation (SeqKD) algorithm, described in detail below:
[0107] The output y of the teacher network * As shown in the following formula:
[0108]
[0109] Where x represents each source feature in the original network state training data, y' represents the possible output value of the network state training data, and π * This represents the probability distribution mapping corresponding to the teacher network;
[0110] Each new correspondence x→y * A new teacher-generated dataset D* was created to train the student network, and Used as a loss function The calculation finally yields the overall expected average loss value. As shown in the following formula:
[0111]
[0112] in, This represents the output of the teacher network over t time windows. This represents the output of the teacher network within a time window of less than t, where T represents the total number of time windows. y represents the expectation operation of summing over the whole. * This represents the situation where, given input x, the data distribution D... * The target output is as follows.
[0113] The student model trained by this loss function recovers the significant sample efficiency gain of the prediction model, while maintaining the fast inference of the prediction model in Web RTC and reducing online training time, thus achieving the desired lightweight goal.
[0114] Step 3: The trained RNN neural network model parameters obtained in Step 2 are shared with the Actor neural network parameters of the Actor-Critic algorithm in the field of reinforcement learning (RL) to complete the knowledge transfer learning based on neural network fine-tuning, and obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge, in which the Actor neural network performs code rate decision and the Critic neural network performs state value calculation.
[0115] This step alleviates the cold start problem in RL algorithm training and further considers transferring the knowledge in the bandwidth prediction model to the machine learning (RL) model;
[0116] Step 4: Train the RL model containing bandwidth prediction knowledge obtained after knowledge transfer in Step 3. Simulate a Web RTC scenario using existing data, output a bitrate control strategy, and further train the deep machine learning (DRL) model to improve the model's bitrate decision-making ability. Combine the reinforcement learning-based RL model with the heuristic WebRTC-based built-in traditional congestion control GCC algorithm to construct a network detector based on the Web RTC Transport CC algorithm to detect network status, calculate the coefficient of variation (CV), and dynamically set γ. t This represents the network congestion threshold at time t. When the network condition detector detects increased network congestion, it enters an algorithm switching state: when the CV value is greater than or equal to the threshold γ. t Switching to the heuristic-based Web RTC built-in traditional congestion control GCC algorithm, when the CV value is less than the threshold γ t When network congestion is not aggravated, a reinforcement learning-based RL model is used. The specific process is as follows:
[0117] Step 4-1, Modeling the bitrate decision problem based on deep reinforcement learning (DRL): Using W = 200ms as the basic unit time window, periodically collect data packets and statistics from RTP packets within the fixed time window of W = 200ms to aggregate them into an environment state vector s. t ∈S, the agent output action value is a under the current environment state. t ∈A; After the action interaction, the environment transitions to the next state s. t+1 ∈S, and simultaneously update the reward value r. t+1 For any given value ∈ R, maximizing the reward corresponds to the highest-level decision made by the current agent. The ultimate goal of the bitrate decision problem model is to obtain the optimal strategy.
[0118] Step 4-2 yields the aggregated state quantity S, which consists of statistics from the current window and multiple historical windows. t , as input to the bitrate control strategy;
[0119] Configure the parameters for the state space and action space: For the state space, within each time window W = 200ms, obtain the statistics of the data packets received within the window based on the information carried in the RTP packet header. As shown in the following formula:
[0120]
[0121] Among them, rate t a represents the received bitrate of the previous time window. t-1 This represents the action taken based on the statistics of the (t-1)th time window, d t Indicates average packet delay, lt This represents the packet loss rate, and t represents the time window.
[0122] By aggregating the statistics of the current time window and multiple historical time windows, the state variable S is obtained. t As shown in the following formula:
[0123]
[0124] Where k represents the past k time windows;
[0125] Considering the environmental state data of the past k time windows, normalize each state input value to the interval [0,1] to speed up the convergence of the algorithm and improve the predictive ability of the model;
[0126] Regarding the action space, in each time window W = 200ms, the bitrate control strategy π φ State S t Mapped to a compact motion space A, with values ranging from 0.01 to 8 Mbps, representing the range of values for the target output video codec bitrate.
[0127] π φ :S t →A (15)
[0128] In different real-time video transmission scenarios, the value range of motion space A can be adjusted accordingly, and it will be fed back to the sender through RTCP packets;
[0129] Step 4-3: Define the reward function for the RL model that incorporates bandwidth prediction knowledge.
[0130] The strategy aims to continuously increase reward values to train agents and improve their skills, with the ultimate goal of maximizing discount returns. Where γ is the discount factor, and R t+k That is the actual reward value at time t+k.
[0131] The reward function r for each time window W = 200ms t As shown in the following formula:
[0132]
[0133] Where N represents a time window W t The number of RTP packets within, q n This represents the throughput measured at the receiving end, which is directly related to video quality. n and d n These are the packet loss rate and delay at the transport layer, α, β, η, The hyperparameters are non-negative and are determined using convex optimization methods such as Bayesian optimization. This indicates a method used to enhance video smoothness by penalizing drastic bitrate fluctuations. The reward function r... t Standardized to the interval [-1, 1].
[0134] The reward function incentivizes agents to improve bandwidth utilization and minimizes packet loss, latency, and frequent switching between significantly different bitrate values.
[0135] Step 4-4: Setting up and applying reinforcement learning algorithm architecture.
[0136] The deep reinforcement learning algorithm adopts an actor-critic architecture, which is a neural network consisting of two modules: actor and critic. The actor is responsible for learning the optimal policy π* and determining W at each time window. t The actions taken are determined by the Critic network, which is responsible for calculating the action value function Q(s,a).
[0137] To improve sample utilization efficiency, an off-policy approach is used, where trajectory data (s, a, r, s') obtained from the interaction between the Actor and the Web RTC system is stored in a playback buffer. Trajectory data is sampled from the playback buffer for gradient calculation and parameter updates.
[0138] The policy network parameters φ are updated using the traditional gradient descent method, as shown in the following equation.
[0139]
[0140] in, This represents the gradient of the network loss function J(φ) with respect to the parameter φ, π. φ (s) represents the network's decision in state s. This represents the gradient of the Q-network's predicted value for action a in state s with respect to action a, where N represents the total number of samples used for averaging. The Q-value output by the Critic network will guide the update of φ through chain derivation.
[0141] Simultaneously, the TD3 algorithm is used to update the Critic network parameters, which utilizes Double-Q learning to suppress the overestimation problem of the evaluation network. For example... Figure 6 As shown, both the policy network and the Q-value network have corresponding target networks. The target network is only used during network updates. The DRL rate controller interacts online with the Web RTC system and collects (s,a,r,s') trajectory information into the replay buffer.
[0142] During the update process, the TD target and TD error are calculated by sampling the trajectory information of a batch in the replay buffer. In this experiment, the batch size is set to 32. The gradient calculation formula used to update the Q network in Double-Q learning is as follows:
[0143]
[0144] Among them, J Q (θ i ) represents the loss function of the Q-network, which is the objective function to be minimized, θ i Let represent the network parameters of the i-th network, r(s,a) represent the immediate reward obtained by taking action a in state s, γ is a discount factor that weighs the proportion of future rewards, and ∈ represents a small positive number set to ensure numerical stability. Let s' represent the predicted value of the Q-network when it takes action a in state s, and let s' and a' represent the new state and action, respectively.
[0145] Steps 4-5: Construct a hybrid algorithm combining a reinforcement learning-based RL model and a heuristic-based traditional congestion control GCC algorithm built into Web RTC. The specific strategy for dynamically switching between the two algorithms is as follows:
[0146] First, a network detector based on the Transport CC algorithm of Web RTC is constructed to detect network status, calculate the coefficient of variation (CV), and dynamically set γ. t This represents the network congestion threshold at time t. When the network condition detector detects increased network congestion, it enters an algorithm switching state. This state is triggered when the CV value is greater than or equal to the threshold γ. t Switch to the GCC algorithm when the CV value is less than the threshold γ. t A reinforcement learning RL neural network is used when network congestion is not aggravated.
[0147] Those skilled in the art can make improvements and variations based on the above-described invention. Any modifications, improvements, and variations made on the basis of this invention should fall within the protection scope of this invention.
Claims
1. A real-time communication system suitable for non-stationary network environments, characterized in that, It includes a training module, a compression module, a knowledge transfer module, and a bitrate control module; The training module trains a bandwidth prediction model based on the NS_Transformer framework. It takes massive network bandwidth data as input to the prediction model and uses subsequent bandwidth data outside the current time window as labels for supervised learning to obtain a trained prediction model based on the NS_Transformer framework. The module further includes a preprocessing unit, which is used to preprocess and batch-divide the collected massive network bandwidth tracking dataset, collect network bandwidth data from the preprocessed network bandwidth tracking dataset, and generate a network bandwidth configuration file. The system also includes a prediction model unit for constructing a bandwidth prediction model based on the NS_Transformer framework. The input to the bandwidth prediction model based on the NS_Transformer framework is the time series of bandwidth, packet loss rate, and latency. The prediction model unit further includes a stabilization processing subunit, which normalizes the input of the bandwidth prediction model based on the NS_Transformer framework in the first stage, and normalizes the time series of network state variables, including network bandwidth, packet loss rate, and latency, in the time dimension, and then uses them as input again. In the second stage, the output of the bandwidth prediction model based on the NS_Transformer framework is denormalized to recover the distribution information lost during normalization; and a destationary subunit is performed, which uses an attention mechanism to achieve destationary processing for non-stationary time series data. The compression module is used to take the prediction model based on the NS_Transformer framework trained by the training module as the teacher network, set a recurrent neural network model (RNN) as the student network, and use the RNN to learn knowledge distillation extraction from the trained prediction model based on the NS_Transformer framework to obtain a lightweight prediction model based on the NS_Transformer framework. The knowledge transfer module is used to share the neural network parameters of the lightweight prediction model based on the NS_Transformer framework obtained by the compression module with the Actor neural network, complete the knowledge transfer learning based on neural network fine-tuning, and obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge, in which the Actor neural network performs code rate decision and the Critic neural network performs state value calculation. The bitrate control module is used to train the RL model containing bandwidth prediction knowledge obtained after knowledge transfer from the knowledge transfer module, and to design a bitrate control strategy by combining the reinforcement learning-based RL model with the traditional congestion control GCC algorithm built into the heuristic Web RTC to achieve bandwidth prediction.
2. The real-time communication system suitable for non-stationary network environments according to claim 1, characterized in that, in, The batch partitioning involves randomly sampling the network bandwidth tracking dataset, calculating the average and variance of the data bandwidth, and dividing it into two categories according to the median of the average: within each dataset, the dataset with higher bandwidth is set as the first dataset, and the dataset with lower bandwidth is split into the second dataset.
3. A real-time communication system suitable for non-stationary network environments according to claim 1, characterized in that, in, The configuration file is used to perform the following processes: Within a delay range of 0-200ms, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.8 and 0.2 respectively. Within a packet loss rate of 0-20%, random samples are taken according to a normal distribution with a mean of 4% and a standard deviation of 0.
02. The sampling results are summarized and saved as dataset A. Within a delay range of 0-200ms, random sampling is performed directly. Within a packet loss rate of 0-20%, random sampling is performed with a normal distribution having a mean of 10% and a standard deviation of 0.
02. The sampling results are summarized and saved as dataset B and dataset C. Within a delay range of 0-200ms, random samples are taken between 0-40ms and 40ms-200ms with probabilities of 0.2 and 0.8 respectively. Within a packet loss rate of 0-20%, random samples are taken according to a normal distribution with a mean of 16% and a standard deviation of 0.
02. The sampling results are summarized and saved as dataset D.
4. A real-time communication system suitable for non-stationary network environments according to claim 3, characterized in that, in, Dataset A has relatively high bandwidth and weak non-stationarity; Dataset B has relatively high bandwidth and strong non-stationarity; Dataset C has relatively low bandwidth and weak non-stationarity; and Dataset D has relatively low bandwidth and strong non-stationarity.
5. A real-time communication system suitable for non-stationary network environments according to claim 1, characterized in that, in, The bitrate control module further includes: The Actor network in the applied Actor-Critic algorithm has shared the knowledge of the bandwidth prediction model based on the NS_Transformer framework through the knowledge transfer module of the bandwidth prediction model based on the NS_Transformer framework. It is trained in a simulated Web RTC system environment, and the reward function is set as follows: in, Represents a time window The number of RTP packets within, This represents the throughput measured at the receiving end. , The last term represents the packet loss rate and latency at the transport layer. Used to enhance video smoothness by penalizing drastic bitrate fluctuations, each hyperparameter is preceded by a specific term. Represents non-negative numbers; Simultaneously, the trajectory data obtained by the interaction between the Actor and the Web RTC system environment It is stored in the replay buffer for gradient calculation and updating the Critic network parameters; This paper combines a reinforcement learning-based RL model with the traditional congestion control (GCC) algorithm built into Web RTC, based on heuristics, and applies it to a real-world network environment to achieve bit rate control prediction. The specific strategy for dynamically switching between the two algorithms is as follows: Construct a network detector based on the Transport CC algorithm of Web RTC to detect network status, calculate the coefficient of variation (CV), and dynamically set... Indicates the first A network congestion threshold is defined per unit time. When the network condition detector detects increased network congestion, it enters an algorithm switching state. This state is triggered when the CV value is greater than or equal to the threshold. Switch to the GCC algorithm when the CV value is less than the threshold. A reinforcement learning RL neural network is used when network congestion is not aggravated.
6. A real-time communication method suitable for non-stationary network environments, characterized in that, include: Step 1: The training module trains a bandwidth prediction model based on the NS_Transformer framework. It takes massive network bandwidth data as input to the prediction model and uses subsequent bandwidth data outside the current time window as labels for supervised learning to obtain a trained prediction model based on the NS_Transformer framework. This module further includes a preprocessing unit, which is used to preprocess and batch divide the collected massive network bandwidth tracking dataset, collect network bandwidth data from the preprocessed network bandwidth tracking dataset, and generate a network bandwidth configuration file. The system also includes a prediction model unit for constructing a bandwidth prediction model based on the NS_Transformer framework. The input to the bandwidth prediction model based on the NS_Transformer framework is the time series of bandwidth, packet loss rate, and latency. The prediction model unit further includes a stabilization processing subunit, which normalizes the input of the bandwidth prediction model based on the NS_Transformer framework in the first stage. This normalizes the time series of network state variables, including network bandwidth, packet loss rate, and latency, in the time dimension, and then uses them as input again. In the second stage, the output of the bandwidth prediction model based on the NS_Transformer framework is denormalized to recover the distribution information lost during normalization; and a destationary subunit is performed, which uses an attention mechanism to achieve destationary processing for non-stationary time series data. Step 2: Use the bandwidth prediction model based on the NS_Transformer framework trained by the training module as the teacher network, set a recurrent neural network (RNN) as the student network, and use the RNN to learn knowledge distillation extraction from the trained bandwidth prediction model based on the NS_Transformer framework to obtain a lightweight bandwidth prediction model based on the NS_Transformer framework. Step 3: Share the compressed neural network parameters based on the NS_Transformer framework with the Actor neural network to complete the knowledge transfer learning based on neural network fine-tuning, and obtain the reinforcement learning model Actor-Critic algorithm model containing bandwidth prediction knowledge. In this model, the Actor neural network makes code rate decisions, while the Critic neural network calculates state value. Step 4: Train the machine learning RL model containing bandwidth prediction knowledge obtained in Step 3. Combine the reinforcement learning-based RL model with the traditional congestion control GCC algorithm built into Web RTC based on heuristics to design a bitrate control strategy to achieve bandwidth prediction.
