Mobile scenario-based deep reinforcement learning bandwidth estimation method based on signal enhancement

Abstract

The application provides a bandwidth estimation method based on signal enhancement deep reinforcement learning in a mobile scenario, relates to the technical field of mobile communication, and comprises the following steps: collecting network transmission state data, normalizing and filtering RSSI values, constructing a state space comprising network transmission state features, timing features and RSSI values, inputting a deep reinforcement learning model of an actor-critic structure to perform bandwidth prediction, and triggering a bandwidth protection mechanism based on RSSI value differences. The application can effectively improve the bandwidth prediction accuracy in the mobile scenario, reduce the influence of network fluctuations, and improve the signal transmission quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile communication technology, in particular to a deep reinforcement learning bandwidth estimation method based on signal enhancement in a mobile scenario, which is suitable for scenarios where network dynamics change due to user movement in real-time communication. BACKGROUND

[0002] With the rapid development of mobile Internet, real-time applications such as video streaming, online gaming, and remote conferencing have increasing demands for network bandwidth. In a mobile scenario, due to factors such as user location changes, signal strength fluctuations, and network congestion, network bandwidth often changes dynamically, which poses a serious challenge to the quality of service of real-time applications. Accurate estimation of available bandwidth is crucial for optimizing network resource allocation and improving user experience.

[0003] Existing bandwidth estimation methods rely heavily on network transmission data (such as reception rate, delay, and packet loss rate), and do not fully consider the direct impact of RSSI changes in a mobile scenario on bandwidth, resulting in bandwidth estimation lagging behind when signal drops suddenly, causing code rate to change abruptly and communication quality to decline. In a dynamic mobile environment, the model lacks generalization ability and cannot adapt to signal changes at different speeds. Traditional heuristic algorithms struggle to balance the dynamic correlation between signal features and network state. Therefore, how to combine signal features with deep reinforcement learning to achieve accurate bandwidth estimation in a mobile scenario is a technical problem that needs to be solved. SUMMARY

[0004] The embodiments of the present application provide a deep reinforcement learning bandwidth estimation method based on signal enhancement in a mobile scenario, which can solve the problems in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a deep reinforcement learning bandwidth estimation method based on signal enhancement in a mobile scenario, comprising:

[0006] Collecting signal bandwidth data and an initial RSSI value at a preset sampling interval; converting the initial RSSI value to a normalized RSSI value through linear mapping, and performing sliding window mean filtering on the normalized RSSI value to obtain an RSSI value;

[0007] Constructing the signal bandwidth data as network transmission state features, remodeling a plurality of historical bandwidth prediction samples as time sequence features, and constructing a state space based on the RSSI value, the network transmission state features, and the time sequence features;

[0008] Inputting the state space into a deep reinforcement learning model, wherein the deep reinforcement learning model adopts an actor network-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates the long-term return of the current state;

[0009] The difference between the current RSSI value and the previous RSSI value is calculated, a bandwidth protection mechanism is triggered when the difference is less than a dynamic threshold, and an adjusted predicted bandwidth value is obtained; the adjusted predicted bandwidth value is fed back to the sending end through the real-time transmission control protocol, and the sending end adaptively adjusts the sending rate according to the current packet loss rate;

[0010] The adjusted predicted bandwidth value and the minimum sending rate are output as the final target code rate for guiding network resource allocation.

[0011] The initial RSSI value is converted into a normalized RSSI value through linear mapping, and the normalized RSSI value is subjected to sliding window mean filtering to obtain an RSSI value, including:

[0012] The signal fluctuation rate of the normalized RSSI value in the sliding time window is calculated, and the signal fluctuation rate is calculated based on the variance of the normalized RSSI value and the window mean;

[0013] The window size of the sliding time window is dynamically adjusted according to the signal fluctuation rate, the window size is set to a minimum window value when the signal fluctuation rate is greater than a fluctuation threshold, and the window size is set to a maximum window value when the signal fluctuation rate is less than a stationary threshold;

[0014] The normalized RSSI value in the sliding time window is subjected to weighted average filtering to obtain an RSSI value.

[0015] The state space input is input into a deep reinforcement learning model, the deep reinforcement learning model adopts an actor network-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates the long-term return of the current state, including:

[0016] The RSSI value is input into multiple fully connected layers of the deep reinforcement learning model to obtain a deep feature vector;

[0017] The deep feature vector is input into an actor network output layer, the actor network output layer contains multiple neurons and uses a softmax activation function to obtain a multi-dimensional action probability distribution; the bandwidth prediction range is divided into multiple prediction intervals, and each element in the multi-dimensional action probability distribution corresponds to a selection probability of a prediction interval; the bandwidth prediction interval with the highest selection probability is selected based on the action probability distribution, and the bandwidth prediction interval is mapped into a specific bandwidth prediction value through a logarithmic linear transformation;

[0018] The deep feature vector is input into a critic network output layer to obtain a state value, a time difference error is calculated based on the state value; and the network parameters of the actor network are updated according to the time difference error.

[0019] The method further comprises:

[0020] Based on the receiving rate, the data packet delay, the packet loss rate and the RSSI value, a reward function is constructed, and the deep reinforcement learning model is trained using a proximal policy optimization algorithm, comprising:

[0021] The receiving rate is logarithmically converted to obtain a bandwidth reward component, the data packet delay is limited within a preset time range to obtain a delay penalty component, the packet loss rate is taken as a transmission quality penalty component, the deviation of the RSSI value at the current time from the RSSI value at the previous time is calculated to obtain a signal fluctuation penalty component, and the bandwidth reward component, the delay penalty component, the transmission quality penalty component and the signal fluctuation penalty component are weighted and combined to obtain a reward function value at the current time;

[0022] The ratio of the new policy probability output by the actor network to the policy probability of the previous iteration is calculated to obtain a probability ratio; the weighted sum of the difference between the reward function values at the current time and subsequent multiple times and the state value is obtained to obtain an advantage function value; the probability ratio is multiplied by the advantage function value, and a policy entropy penalty term is added to obtain a policy loss value;

[0023] Randomly disconnect in each layer of neurons of the actor network to form a sparse network structure, and calculate the network parameter gradient based on the policy loss value; use the Adam optimizer to update the parameters in the sparse network structure to complete one policy iteration, and repeat the policy iteration until the discounted cumulative reward converges.

[0024] The difference between the current RSSI value and the RSSI value at the previous time is calculated, and when the difference is less than a dynamic threshold, a bandwidth protection mechanism is triggered to obtain an adjusted predicted bandwidth value, comprising:

[0025] The deviation of the RSSI value at the current time from the RSSI value at the previous time is calculated, and the absolute value thereof is taken as an RSSI difference value; the standard deviation and the mean of the historical RSSI values are obtained, and the weighted sum of the standard deviation and the mean is obtained to obtain a dynamic threshold;

[0026] When the RSSI difference value is less than the dynamic threshold, the bandwidth protection mechanism is triggered; the ratio of the RSSI difference value to the dynamic threshold is calculated, and the negative exponent of the ratio is taken to obtain a bandwidth factor; the current predicted bandwidth value and the historical bandwidth mean are multiplied by the bandwidth factor respectively and summed to obtain a protected bandwidth value;

[0027] Calculate an absolute value of a difference between the protection bandwidth value and the historical bandwidth mean value, when the absolute value of the difference is less than a preset adjustment amplitude, take the protection bandwidth value as an adjusted prediction bandwidth value, when the absolute value of the difference is greater than or equal to the preset adjustment amplitude, add or subtract the preset adjustment amplitude to the historical bandwidth mean value as an adjusted prediction bandwidth value.

[0028] The adjusted prediction bandwidth value is fed back to the sending end through the real-time transmission control protocol, and the sending end adaptively adjusts the sending rate according to the current packet loss rate, comprising:

[0029] The sending end counts the number of lost data packets and the total number of sent data packets in a preset time window, and obtains the current packet loss rate by dividing the number of lost data packets by the total number of sent data packets.

[0030] According to the current packet loss rate, determine the packet loss level, when the current packet loss rate is greater than the high packet loss threshold, obtain the down coefficient, when the current packet loss rate is less than the low packet loss threshold, obtain the up coefficient, and take the complement of the down coefficient or the up coefficient plus one as the rate adjustment coefficient.

[0031] The current transmission rate is multiplied by the rate adjustment coefficient to obtain a new sending rate, and the new sending rate and the sending rate at the previous time are multiplied by the smoothing factor and the complement of the smoothing factor, respectively, and added to obtain the smoothed sending rate at the current time.

[0032] The adjusted prediction bandwidth value is multiplied by the complement of the network jitter compensation coefficient to obtain a feedback bandwidth value, the current time smoothed sending rate, the feedback bandwidth value and the maximum rate limit are taken as the minimum value, and the minimum value and the minimum rate limit are taken as the maximum value to obtain the final sending rate.

[0033] The second aspect of the embodiment of the application provides a deep reinforcement learning bandwidth estimation system based on signal enhancement in a mobile scenario, comprising:

[0034] The first unit is configured to collect signal bandwidth data and an initial RSSI value at a preset sampling interval, convert the initial RSSI value into a normalized RSSI value through linear mapping, and perform sliding window mean filtering on the normalized RSSI value to obtain an RSSI value.

[0035] The second unit is configured to construct the signal bandwidth data into network transmission state features, reshape a plurality of historical bandwidth prediction samples into time sequence features, and construct a state space based on the RSSI value, the network transmission state features and the time sequence features.

[0036] The third unit is configured to input the state space into the deep reinforcement learning model, the deep reinforcement learning model adopts an actor network-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates long-term returns of a current state.

[0037] The fourth unit is configured to calculate a difference between the current RSSI value and a previous RSSI value, trigger a bandwidth protection mechanism when the difference is less than a dynamic threshold, obtain an adjusted predicted bandwidth value, and feed back the adjusted predicted bandwidth value to a sending end through a real-time transport control protocol, and the sending end adaptively adjusts a sending rate according to a current packet loss rate.

[0038] The fifth unit is configured to output the adjusted predicted bandwidth value and a minimum sending rate as a final target code rate, and guide network resource allocation.

[0039] The third aspect of the embodiment of the application,

[0040] An electronic device is provided, comprising:

[0041] A processor;

[0042] A memory for storing processor-executable instructions;

[0043] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0044] The fourth aspect of the embodiment of the application,

[0045] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0046] The beneficial effects of the present application are as follows:

[0047] The bandwidth estimation method combining signal enhancement and deep reinforcement learning effectively solves the problems of large network fluctuation and inaccurate bandwidth prediction in a mobile scenario, solves the problems of bandwidth estimation lag and poor robustness in a mobile environment through multi-feature dynamic fusion and adaptive protection mechanism, and improves real-time communication quality.

[0048] The application introduces an RSSI signal enhancement mechanism, combines network transmission state features and timing features to construct a complete state space, realizes more accurate bandwidth prediction through an actor-critic network structure, and enables the system to better adapt to complex and changeable mobile network environments.

[0049] The bandwidth protection mechanism and the adaptive sending rate adjustment strategy designed by the application effectively avoid network congestion and resource waste, improve video stream media transmission quality and user experience, and optimize network resource allocation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of a bandwidth estimation method based on signal enhancement in a mobile scenario according to an embodiment of the application is shown in

[0051] Figure 2 An actor-critic network structure of a deep reinforcement learning model is shown in

[0052] Figure 3 A deep reinforcement learning network optimization architecture is shown in DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below in detail with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0054] The technical solutions of the application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and some embodiments may not be described again for the same or similar concepts or processes.

[0055] Figure 1 A flowchart of a bandwidth estimation method based on signal enhancement in a mobile scenario according to an embodiment of the application is shown in Figure 1 The method comprises the following steps.

[0056] Signal bandwidth data and an initial RSSI value are collected at a preset sampling interval; the initial RSSI value is converted into a normalized RSSI value through linear mapping, and the normalized RSSI value is subjected to sliding window mean filtering to obtain an RSSI value;

[0057] The signal bandwidth data is constructed into network transmission state features, and a plurality of historical bandwidth prediction samples are reshaped into time sequence features; a state space is constructed based on the RSSI value, the network transmission state features, and the time sequence features;

[0058] The state space is input into a deep reinforcement learning model, the deep reinforcement learning model adopts an actor network-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates long-term returns of a current state.

[0059] The difference between the current RSSI value and the previous RSSI value is calculated, and when the difference is less than a dynamic threshold, a bandwidth protection mechanism is triggered to obtain an adjusted predicted bandwidth value; the adjusted predicted bandwidth value is fed back to the sending end through a real-time transmission control protocol, and the sending end adaptively adjusts the sending rate according to the current packet loss rate;

[0060] The adjusted predicted bandwidth value and the minimum sending rate are output as a final target code rate for guiding network resource allocation.

[0061] The signal bandwidth data includes a receiving rate, a packet delay, and a packet loss rate.

[0062] In an optional implementation, the initial RSSI value is converted into a normalized RSSI value through linear mapping, and a sliding window mean filtering is performed on the normalized RSSI value to obtain an RSSI value, including:

[0063] The signal fluctuation rate of the normalized RSSI value in the sliding time window is calculated, and the signal fluctuation rate is calculated based on the variance of the normalized RSSI value and the window mean value;

[0064] The window size of the sliding time window is dynamically adjusted according to the signal fluctuation rate, the window size is set to a minimum window value when the signal fluctuation rate is greater than a fluctuation threshold, and the window size is set to a maximum window value when the signal fluctuation rate is less than a stationary threshold;

[0065] A weighted average filtering is performed on the normalized RSSI value in the sliding time window to obtain an RSSI value.

[0066] In this embodiment, a RSSI signal processing method will be described in detail, which improves the stability and accuracy of the RSSI signal through linear mapping conversion and sliding window mean filtering technology.

[0067] The initial RSSI value received by the device is between-100 dBm and-20 dBm, and these original values need to be normalized for subsequent analysis. This method uses linear mapping to transform the initial RSSI value to the interval [0, 1]. In specific implementation, the following processing method can be used: assuming that the received initial RSSI value is-75 dBm, the preset minimum RSSI value is-100 dBm, and the maximum value is-20 dBm, then the normalized RSSI value is calculated as ((-75)-(-100)) / ((-20)-(-100))=0.3125. In this way, all RSSI values are mapped to between 0 and 1, which is convenient for subsequent processing.

[0068] After normalization, the fluctuation of the signal in the sliding time window needs to be calculated. Assume that the current sliding window size is 5 samples, and the normalized RSSI values in the window are [0.25, 0.28, 0.22, 0.26, 0.24]. First, the mean of these values is calculated, which is 0.25. Then the square of the difference between each value and the mean is calculated, which are (0.25-0.25)²=0, (0.28-0.25)²=0.0009, (0.22-0.25)²=0.0009, (0.26-0.25)²=0.0001, (0.24-0.25)²=0.0001. The sum of these squared differences is divided by the window size to obtain the variance, which is (0+0.0009+0.0009+0.0001+0.0001) / 5=0.0004. The signal fluctuation rate can be defined as the square root of the variance, which is 0.02.

[0069] According to the calculated signal fluctuation rate, the system dynamically adjusts the size of the sliding time window. Set the fluctuation threshold to 0.05, the stationary threshold to 0.01, the minimum window value to 3, and the maximum window value to 15. When the signal fluctuation rate is greater than the fluctuation threshold, it means that the signal changes dramatically and needs to respond quickly, so the window size is set to the minimum window value of 3; when the signal fluctuation rate is less than the stationary threshold, it means that the signal is relatively stable and a larger window can be used for smoothing, so the window size is set to the maximum window value of 15; when the signal fluctuation rate is between the two thresholds, the window size can be calculated proportionally. In this example, the signal fluctuation rate is 0.02, which is between 0.01 and 0.05, so the window size can be calculated to be 9.

[0070] For the normalized RSSI values in the dynamically adjusted sliding time window, weighted average filtering is performed. The weight coefficient of weighted average filtering is calculated by an exponential decay function, with newer data having higher weight and older data having lower weight. Assume that the adjusted window size is 9, and the normalized RSSI values in the window are [0.21, 0.23, 0.25, 0.28, 0.22, 0.26, 0.24, 0.27, 0.25].

[0071] The decay coefficient of the exponential decay function is determined according to the signal fluctuation rate. When the signal fluctuation rate is large, more attention should be paid to the latest data, and the decay coefficient should be larger; when the signal fluctuation rate is small, all data in the window can be considered more evenly, and the decay coefficient should be smaller. Assume that the decay coefficient calculated according to the signal fluctuation rate of 0.02 is 0.3.

[0072] Using this decay coefficient, the weight of each sample point can be calculated. Starting from the latest data, the weights are 1, e (-0.3) ≈0.74, e (-0.6)≈0.55, e (-0.9) ≈0.41, e (-1.2) ≈0.30, e (-1.5) ≈0.22, e (-1.8) ≈0.17, e (-2.1) ≈0.12, e (-2.4) ≈0.09. Normalize these weights so that their sum is 1, resulting in normalized weights of 0.28, 0.21, 0.15, 0.11, 0.08, 0.06, 0.05, 0.03, 0.03, respectively.

[0073] Using these weights to multiply and sum the corresponding RSSI values, the weighted average RSSI value is obtained: 0.21x0.03 + 0.23x0.03 + 0.25x0.05 + 0.28x0.06 + 0.22x0.08 + 0.26x0.11 + 0.24x0.15 + 0.27x0.21 + 0.25x0.28 = 0.252. This value is the RSSI value after sliding window mean filtering processing.

[0074] In practical applications, the fluctuation threshold, the stationary threshold, the minimum window value and the maximum window value can be adjusted according to the specific scene. For example, in a scene with fast moving speed, the fluctuation threshold can be reduced, so that the system is more likely to use a small window for processing, in order to quickly respond to environmental changes; in a relatively static scene, the stationary threshold can be increased, so that the system is more inclined to use a large window for smoothing, to obtain a more stable signal.

[0075] Through the above processing method, the fluctuation of the RSSI signal can be effectively reduced, the stability and reliability of the signal can be improved, and more accurate signal input can be provided for RSSI-based positioning, distance estimation and other applications. Experiments show that this method can better adapt to the dynamic changes of the signal than the traditional fixed window size mean filtering method, and can maintain the ability to respond quickly when the signal fluctuates sharply, and can provide smoother output when the signal is relatively stable.

[0076] Figure 2 An actor-critic network structure diagram of a deep reinforcement learning model. In an optional implementation, the state space is input into the deep reinforcement learning model, the deep reinforcement learning model adopts an actor-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates the long-term return of the current state, including:

[0077] The RSSI value is input into a plurality of fully connected layers of the deep reinforcement learning model to obtain a deep feature vector;

[0078] The deep feature vector is input into an actor network output layer, the actor network output layer comprising a plurality of neurons and using a softmax activation function, to obtain a multi-dimensional action probability distribution; a bandwidth prediction range is divided into a plurality of prediction intervals, and each element in the multi-dimensional action probability distribution corresponds to a selection probability of a prediction interval; a bandwidth prediction interval with the highest selection probability is selected based on the action probability distribution, and the bandwidth prediction interval is mapped to a specific bandwidth prediction value through a logarithmic linear transformation;

[0079] The deep feature vector is input into a critic network output layer to obtain a state value, and a time difference error is calculated based on the state value; network parameters of the actor network are updated according to the time difference error.

[0080] In this embodiment, a bandwidth prediction method based on deep reinforcement learning will be described in detail. This method adopts an actor-critic network structure, and through processing of RSSI values, network transmission state features and time sequence features, accurate prediction of bandwidth is realized.

[0081] Firstly, this method constructs a state space, which includes RSSI values, network transmission state features and time sequence features. RSSI values reflect signal strength indicators, network transmission state features include current throughput, packet loss rate, round-trip delay and other parameters, and time sequence features include historical bandwidth prediction values and actual bandwidth values. These features together constitute the input of the deep reinforcement learning model.

[0082] The constructed state space is input into the deep reinforcement learning model for processing. This model adopts an actor-critic network structure, in which the actor network is responsible for outputting the probability distribution of the bandwidth prediction interval, and the critic network evaluates the long-term return of the current state. The specific implementation process is as follows:

[0083] For the processing of RSSI values, the system inputs them into a first fully connected layer. This fully connected layer contains sixty-four neurons and uses a ReLU activation function. Taking a set of RSSI values in actual application as an example: -75dBm, -72dBm, -78dBm, after processing by the first fully connected layer, a RSSI feature vector with a dimension of 64 is obtained, which captures the deep features of signal strength.

[0084] For network transmission state features, the system inputs them into a second fully connected layer. This fully connected layer contains one hundred and twenty-eight neurons, also using a ReLU activation function. For example, the current throughput is 5Mbps, the packet loss rate is 2%, and the round-trip delay is 120ms. After processing by the second fully connected layer, these values are converted into a network state feature vector with a dimension of 128, which encodes the key information of the current network state.

[0085] For the time-series features, the system first reshapes them into a time-series tensor and then inputs them into a one-dimensional convolutional layer. The one-dimensional convolutional layer uses a ReLU activation function and can effectively capture the time dependence in the historical data. For example, given a sequence of predicted bandwidth values [4.2, 4.5, 4.8, 5.0, 4.9] Mbps and a sequence of actual bandwidth values [4.0, 4.3, 4.7, 5.1, 5.0] Mbps over the past 10 seconds, the one-dimensional convolutional layer generates a historical prediction feature vector that contains information about the trend of bandwidth changes.

[0086] After feature extraction is complete, the system concatenates the RSSI feature vector, the network state feature vector, and the historical prediction feature vector in the feature dimension to form a fusion feature vector. Assuming that the RSSI feature vector has a dimension of 64, the network state feature vector has a dimension of 128, and the historical prediction feature vector has a dimension of 32, the concatenated fusion feature vector has a dimension of 224.

[0087] The fusion feature vector is then input into a third fully connected layer to obtain a deep feature vector. The fully connected layer further extracts high-level features and compresses the information dimension, assuming that the output deep feature vector has a dimension of 96.

[0088] The deep feature vector is simultaneously input into the actor network output layer and the critic network output layer. The actor network output layer contains four neurons and uses a softmax activation function to output a four-dimensional action probability distribution. In this embodiment, the bandwidth prediction range is divided into four prediction intervals: 0.01-2 Mbps, 2-4 Mbps, 4-6 Mbps, and 6-8 Mbps, and the actor network output layer contains four neurons corresponding to the selection probabilities of these four prediction intervals.

[0089] In a certain prediction, the probability distribution output by the actor network is [0.12, 0.28, 0.45, 0.15], representing the selection probabilities of each bandwidth interval. The system selects the bandwidth prediction interval with the highest probability based on this probability distribution, which in this example is the third interval (4-6 Mbps) with a probability of 0.45.

[0090] After selecting the bandwidth prediction interval, the system maps the interval to a specific bandwidth prediction value through a logarithmic linear transformation. The logarithmic linear transformation takes into account the nonlinear characteristics of the bandwidth distribution, making the prediction more accurate. In this example, assume that the specific bandwidth prediction value obtained after the logarithmic linear transformation is 5.2 Mbps.

[0091] At the same time, the deep feature vector is input into the critic network output layer to obtain the state value. The critic network output layer usually has only one neuron that directly outputs the value estimate of the current state. For example, the critic network may output a state value of 0.78, indicating the expected long-term cumulative reward in the current state.

[0092] Based on the state value, the system calculates the time difference error. The time difference error reflects the gap between the predicted state value and the actual reward obtained plus the next state value. Assuming that the actual measured bandwidth is 5.3 Mbps, which is very close to the predicted value of 5.2 Mbps, the system will give a higher immediate reward, such as 0.95. Combined with the value estimate of the next state, the calculated time difference error is 0.17.

[0093] The system updates the parameters of the actor network according to the time difference error. Smaller time difference error indicates more accurate prediction, and smaller update amplitude; larger error means larger parameter adjustment. In this way, the actor network can continuously optimize its bandwidth prediction ability and improve prediction accuracy.

[0094] Through the above method, the system can accurately predict the future bandwidth value based on the RSSI value, network state and historical data of the current network environment, providing strong support for network resource allocation and quality of service guarantee.

[0095] Figure 3 The schematic diagram of the deep reinforcement learning network optimization architecture. In an optional implementation, the reward function is constructed based on the reception rate, the packet delay, the packet loss rate and the RSSI value, and the deep reinforcement learning model is trained using the proximal policy optimization algorithm, including:

[0096] The reception rate is logarithmically converted to obtain a bandwidth benefit component, the packet delay is limited within a preset time range to obtain a delay penalty component, the packet loss rate is taken as a transmission quality penalty component, the deviation between the RSSI value at the current time and the RSSI value at the last time is calculated to obtain a signal fluctuation penalty component, and the bandwidth benefit component, the delay penalty component, the transmission quality penalty component and the signal fluctuation penalty component are weighted and combined to obtain the reward function value at the current time;

[0097] The ratio of the new policy probability output by the actor network to the policy probability of the last iteration is calculated to obtain a probability ratio; the weighted sum of the difference between the reward function value and the state value from the current time to the subsequent multiple times is obtained to obtain an advantage function value; the probability ratio is multiplied by the advantage function value, and a policy entropy penalty term is added to obtain a policy loss value;

[0098] Randomly disconnect in each layer of neurons of the actor network to form a sparse network structure, and calculate the network parameter gradient based on the policy loss value; update the parameters in the sparse network structure using the Adam optimizer to complete one policy iteration, and repeat the policy iteration until the discounted cumulative reward converges.

[0099] In this embodiment, a network optimization method based on deep reinforcement learning is described in detail. This method constructs a reward function and uses a proximal policy optimization algorithm for training to achieve optimization of network performance.

[0100] The method first collects network performance indicators, including reception rate, packet delay, packet loss rate, and RSSI value. The reception rate reflects the throughput capacity of the network, the packet delay represents the time consumption of data transmission, the packet loss rate measures the reliability of network transmission, and the RSSI value represents the signal strength. These indicators together form the basis for evaluating network performance.

[0101] When constructing the reward function, the reception rate is logarithmically converted to obtain the bandwidth benefit component. For example, when the reception rate is 10 Mbps, its logarithmic value is about 1, and when the reception rate is 100 Mbps, its logarithmic value is about 2. This logarithmic conversion can smooth the rate difference of different orders of magnitude, making the sensitivity of the reward function to rate changes more reasonable. For packet delay, a threshold of 1 second is set, and when the delay exceeds 1 second, the delay penalty component increases with the increase of delay. In specific implementation, a nonlinear function can be used, when the delay is 0.5 seconds, the penalty value is 0.5; when the delay is 1 second, the penalty value is 1; when the delay is 2 seconds, the penalty value is 4, showing the characteristic of rapid increase of penalty after exceeding the threshold.

[0102] The packet loss rate directly affects the reward function as the transmission quality penalty component. For example, when the packet loss rate is 5%, the transmission quality penalty component is 0.05; when the packet loss rate is 20%, the transmission quality penalty component is 0.2. In addition, in order to capture the stability of the network signal, the deviation of the current RSSI value from the last RSSI value is calculated to obtain the signal fluctuation penalty component. If the current RSSI is -70 dBm and the last RSSI is -65 dBm, the deviation is 5 dBm, and the signal fluctuation penalty component can be set to 0.05.

[0103] The reward function is obtained by weighted combination of the above components. Assuming that the bandwidth benefit weight is 0.6, the delay penalty weight is 0.2, the transmission quality penalty weight is 0.1, and the signal fluctuation penalty weight is 0.1. When the bandwidth benefit component is 2, the delay penalty component is 0.5, the transmission quality penalty component is 0.05, and the signal fluctuation penalty component is 0.05, the reward function value is 2x0.6-0.5x0.2-0.05x0.1-0.05x0.1=1.09.

[0104] To consider long-term benefits, multiply the reward function value in each time window by the time power of the discount factor. Assuming the discount factor is 0.95, the time window is 5, the current time reward is 1.09, the next time reward is 1.15, and the next time reward is 1.2, and so on. The discounted cumulative reward is 1.09 + 1.15 * 0.95 + 1.2 * 0.95^2 +..., in this way, the system pays more attention to recent rewards, but also does not completely ignore long-term benefits.

[0105] During training, the ratio of the new policy probability output by the actor network to the last iteration policy probability is calculated to obtain the probability ratio. For example, the probability of a new policy for an action is 0.7, and the old policy is 0.6, then the probability ratio is 1.167. To prevent unstable policy updates, set the probability threshold to 1.2 to ensure that the probability ratio does not exceed this value.

[0106] The calculation of the advantage function value considers the difference between the reward function value and the state value from the current time to the subsequent multiple times. Assuming the current time reward is 1.09, the state value is 1, and the difference is 0.09; the next time reward is 1.15, the state value is 1.1, and the difference is 0.05, and so on. The weighted sum of these differences can be obtained as 0.09 * 0.5 + 0.05 * 0.3 +... = 0.06.

[0107] The policy loss value is obtained by multiplying the probability ratio and the advantage function value, and adding the policy entropy penalty term. Policy entropy is used to encourage exploration and prevent premature convergence of the policy. If the policy entropy is 0.3 and the entropy weight is 0.01, the entropy penalty term is 0.003. When the probability ratio is 1.167 and the advantage function value is 0.06, the policy loss value is 1.167 * 0.06 - 0.003 = 0.067.

[0108] To improve the generalization ability of the model, randomly disconnect 50% of the connections in each layer of the actor network to form a sparse network structure. This technique is called Dropout, which can effectively prevent overfitting. After calculating the network parameter gradient based on the policy loss value, the Adam optimizer is used to update the parameters in the sparse network structure. The Adam optimizer combines the advantages of the momentum method and the adaptive learning rate method, and can more effectively optimize deep neural networks.

[0109] During training, the above steps are repeated until the discounted cumulative reward converges. The convergence criteria can be set to a change rate of less than 1% in the discounted cumulative reward within 10 consecutive training cycles. In this way, the deep reinforcement learning model can learn the policy of selecting the optimal action under different network states, thereby optimizing the network performance.

[0110] In an alternative embodiment, the system calculates the difference between the current RSSI value and the previous RSSI value, triggers the bandwidth protection mechanism when the difference is less than a dynamic threshold, and obtains an adjusted predicted bandwidth value, including:

[0111] The system calculates the deviation between the current RSSI value and the previous RSSI value, and takes the absolute value of the deviation as the RSSI difference. The system obtains the standard deviation and the mean of the historical RSSI values, and performs a weighted sum of the standard deviation and the mean to obtain a dynamic threshold.

[0112] When the RSSI difference is less than the dynamic threshold, the system triggers the bandwidth protection mechanism. The system calculates the ratio of the RSSI difference to the dynamic threshold, takes the negative exponent of the ratio to obtain a bandwidth factor. The system multiplies the current predicted bandwidth value and the historical bandwidth mean by the bandwidth factor respectively and performs a sum to obtain a protected bandwidth value.

[0113] The system calculates the absolute value of the difference between the protected bandwidth value and the historical bandwidth mean. When the absolute value of the difference is less than a preset adjustment amplitude, the system takes the protected bandwidth value as the adjusted predicted bandwidth value. When the absolute value of the difference is greater than or equal to the preset adjustment amplitude, the system takes the historical bandwidth mean plus or minus the preset adjustment amplitude as the adjusted predicted bandwidth value.

[0114] In a bandwidth prediction method, the system triggers a bandwidth protection mechanism by comparing the difference between the current RSSI value and the previous RSSI value with a dynamic threshold, and obtains an adjusted predicted bandwidth value. The specific implementation process of the method is as follows:

[0115] The system first obtains the RSSI value at the current time and the RSSI value at the previous time. For example, the RSSI value at the current time is -65 dBm, and the RSSI value at the previous time is -62 dBm. The system calculates the difference between the two RSSI values, and takes the absolute value of the difference as the RSSI difference. In this example, the RSSI difference is |-65-(-62)|=|3|=3 dBm.

[0116] The system obtains the RSSI values within a certain time window from historical data, for example, the RSSI values within the last 10 seconds. Assume that these historical RSSI values are {-60 dBm, -62 dBm, -63 dBm, -61 dBm, -62 dBm, -64 dBm, -63 dBm, -62 dBm, -61 dBm, -62 dBm}. The system calculates the standard deviation and the mean of these historical RSSI values. In this example, the standard deviation of the historical RSSI values is 1.2 dBm, and the mean is -62 dBm.

[0117] The system calculates a dynamic threshold value based on the standard deviation and mean of historical RSSI values. The dynamic threshold value is calculated by weighted sum of the standard deviation and mean. Assuming the weight of the standard deviation is 0.8 and the weight of the mean is 0.05, the dynamic threshold value is 0.8x1.2+0.05x|-62|=0.96+3.1=4.06dBm.

[0118] The system compares the RSSI difference value with the dynamic threshold value. In this example, the RSSI difference value is 3dBm, which is less than the dynamic threshold value 4.06dBm, thus triggering the bandwidth protection mechanism.

[0119] The system calculates the ratio of the RSSI difference value and the dynamic threshold value. In this example, the ratio is 3 / 4.06=0.739. The system takes the negative exponential of the ratio to obtain the bandwidth factor. Assuming the bandwidth factor is e^(-0.739)=0.478.

[0120] The system obtains the current predicted bandwidth value and the historical bandwidth mean value. Assuming the current predicted bandwidth value is 8Mbps and the historical bandwidth mean value is 10Mbps. The system multiplies the current predicted bandwidth value and the historical bandwidth mean value by the bandwidth factor respectively and sums them up to obtain the protected bandwidth value. In this example, the protected bandwidth value is 0.478x8+(1-0.478)x10=3.824+5.22=9.044Mbps.

[0121] The system calculates the absolute value of the difference between the protected bandwidth value and the historical bandwidth mean value. In this example, the absolute value of the difference is |9.044-10|=0.956Mbps. The system compares the absolute value of the difference with a preset adjustment amplitude. Assuming the preset adjustment amplitude is 1Mbps.

[0122] Since the absolute value of the difference 0.956Mbps is less than the preset adjustment amplitude 1Mbps, the system takes the protected bandwidth value 9.044Mbps as the adjusted predicted bandwidth value. If the absolute value of the difference is greater than or equal to the preset adjustment amplitude, the system takes the historical bandwidth mean value plus or minus the preset adjustment amplitude as the adjusted predicted bandwidth value according to the size relationship between the protected bandwidth value and the historical bandwidth mean value.

[0123] The implementation of the bandwidth protection mechanism can be achieved through the following specific steps: the system maintains a historical queue of RSSI values, adds a new RSSI value to the queue and removes the earliest RSSI value after each sampling, and keeps the length of the queue fixed. The system regularly calculates the standard deviation and mean of the RSSI values in the queue for the calculation of the dynamic threshold value.

[0124] In practical applications, the system can adjust parameters according to changes in the network environment. For example, in an environment with large signal fluctuations, the weight of the standard deviation can be increased to make the dynamic threshold more sensitive to signal changes; in a relatively stable environment, the weight of the standard deviation can be reduced to make the dynamic threshold more stable.

[0125] The calculation of the bandwidth factor uses a negative exponential function, which can make the bandwidth factor smaller when the RSSI difference is closer to the dynamic threshold, and the protection level of the current predicted bandwidth value higher; the smaller the RSSI difference, the larger the bandwidth factor, and the lower the protection level of the current predicted bandwidth value.

[0126] The setting of the preset adjustment amplitude can prevent the bandwidth prediction value from fluctuating sharply. When the difference between the protected bandwidth value and the historical bandwidth average is large, the system limits the change amplitude of the adjusted predicted bandwidth value to ensure smooth transition of the network bandwidth.

[0127] In a specific application scenario, the signal strength received by a mobile device at different locations will change. When a user moves from one location to another, the RSSI value may change, but this change may not immediately affect the actual available bandwidth. Through this method, the system can identify the small changes in the RSSI value and trigger the bandwidth protection mechanism when necessary to avoid unnecessary fluctuations in the bandwidth prediction value and improve user experience.

[0128] This method is particularly suitable for mobile network environments such as 4G / 5G networks, WiFi networks, etc., and can effectively address the problem of inaccurate bandwidth prediction caused by signal strength fluctuations, improving user experience for applications such as video streaming, online gaming, etc. that require high network bandwidth.

[0129] In an optional implementation, the adjusted predicted bandwidth value is fed back to the sending end through the Real-time Transport Control Protocol, and the sending end adaptively adjusts the sending rate according to the current packet loss rate, including:

[0130] The sending end counts the number of lost data packets and the total number of sent data packets within a preset time window, and divides the number of lost data packets by the total number of sent data packets to obtain the current packet loss rate;

[0131] According to the current packet loss rate, determine the packet loss level, when the current packet loss rate is greater than the high packet loss threshold, obtain the down adjustment coefficient, when the current packet loss rate is less than the low packet loss threshold, obtain the up adjustment coefficient, the complement of the down adjustment coefficient or the up adjustment coefficient plus one is used as the rate adjustment coefficient;

[0132] The new transmission rate is obtained by multiplying the current transmission rate by the rate adjustment coefficient. The new transmission rate and the transmission rate at the previous moment are then multiplied by the smoothing factor and the complement of the smoothing factor, respectively, and added together to obtain the smoothed transmission rate at the current moment.

[0133] Multiply the adjusted predicted bandwidth value by the complement of the network jitter compensation coefficient to obtain the feedback bandwidth value; take the minimum value of the current smoothed transmission rate, the feedback bandwidth value, and the maximum rate limit, and take the maximum value of the minimum value and the minimum rate limit to obtain the final transmission rate.

[0134] This invention provides a method for network bandwidth prediction and adaptive rate adjustment based on a real-time transmission control protocol. In practice, this method feeds back the adjusted predicted bandwidth value to the sending end via the real-time transmission control protocol, and the sending end adaptively adjusts the transmission rate according to the current packet loss rate.

[0135] The sending end counts the number of lost data packets and the total number of data packets sent within a preset time window to calculate the packet loss rate at the current moment. Specifically, the sending end sets a time window, for example, 500 milliseconds, and records the total number of data packets sent within this time window. packets The number of data packets that the receiving end has not acknowledged as received (lost) packets The packet loss rate at the current moment. rate via lost packets Divide by total packets For example, if 1000 data packets are sent within a 500-millisecond time window, and 20 of them do not receive an acknowledgment, then the current packet loss rate is 20 / 1000 = 0.02, or 2%.

[0136] The packet loss level is determined based on the current packet loss rate, and a corresponding adjustment coefficient is obtained accordingly. In practical applications, a high packet loss threshold can be set to 0.05 (i.e., 5%), and a low packet loss threshold to 0.01 (i.e., 1%). When the current packet loss rate exceeds the high packet loss threshold, the system determines it to be in a high packet loss state and needs to reduce the sending rate. At this time, the down adjustment coefficient is obtained. factor The adjustment factor can be set according to the severity of the packet loss rate. For example, when the packet loss rate is 6%, the adjustment factor can be set to "down". factor It can be set to 0.2. The rate adjustment factor is calculated as the complement of the down adjustment factor, i.e., 1 - down. factor In this example, it is 0.8. When the current packet loss rate is less than the low packet loss threshold, the system determines it to be in a low packet loss state, indicating good network conditions, and the sending rate can be increased. At this time, the up-adjustment coefficient is obtained. factorThe rate adjustment factor can be set to a small positive value, such as 0.05, to avoid increasing the rate too quickly and causing network congestion. The rate adjustment factor is calculated by adding one to the rate adjustment factor, which is 1.05 in this example. When the packet loss rate is between the low and high thresholds, the current sending rate remains unchanged, and the rate adjustment factor is 1.

[0137] The new transmission rate is obtained by multiplying the current transmission rate by a rate adjustment factor. For example, if the current transmission rate is 5 Mbps and the rate adjustment factor is 0.8, the new transmission rate is 5 * 0.8 = 4 Mbps. To avoid overly drastic rate adjustments, a smoothing factor is introduced. factor (e.g., 0.7) Perform smoothing. Multiply the new transmission rate and the transmission rate at the previous moment by the smoothing factor and its complement, respectively, and add them together to obtain the smoothed transmission rate at the current moment. Specifically, the smoothed transmission rate at the current moment = new transmission rate * smoothness factor. factor + Previous transmission rate * (1-smooth) factor For example, if the new transmission rate is 4 Mbps, the previous transmission rate was 5 Mbps, and the smoothing factor is 0.7, then the smoothed transmission rate at the current moment is 4 * 0.7 + 5 * 0.3 = 4.3 Mbps.

[0138] The adjusted predicted bandwidth value is multiplied by the complement of the network jitter compensation coefficient to obtain the feedback bandwidth value. Network jitter compensation coefficient. compensation (e.g., 0.1) is used to handle network jitter and ensure sufficient bandwidth margin when the network is unstable. The feedback bandwidth value is calculated as: Feedback bandwidth value = Adjusted predicted bandwidth value * (1 - jitter) compensation For example, if the adjusted predicted bandwidth is 10Mbps and the network jitter compensation coefficient is 0.1, then the feedback bandwidth is 10*(1-0.1)=9Mbps.

[0139] Determining the final transmission rate requires considering several constraints. The minimum value among the current smoothed transmission rate, feedback bandwidth, and maximum rate limit is taken to ensure it does not exceed the network's capacity and the system's set limits. Then, the maximum value between this minimum value and the minimum rate limit is taken to ensure basic transmission quality. For example, if the current smoothed transmission rate is 4.3 Mbps, the feedback bandwidth is 9 Mbps, the maximum rate limit is 15 Mbps, and the minimum rate limit is 1 Mbps, then the final transmission rate is the maximum of the minimum of these three values ​​and the minimum rate limit, i.e., the maximum of min(4.3, 9, 15) and 1, resulting in 4.3 Mbps.

[0140] In practical application scenarios, the method can dynamically adjust the sending rate according to the network condition. When the network is congested (high packet loss rate), the system will reduce the sending rate to reduce the network burden; when the network condition is good (low packet loss rate), the system will appropriately increase the sending rate to improve the transmission efficiency. By introducing a smoothing factor and a network jitter compensation coefficient, the method can fully utilize the available bandwidth resources while ensuring transmission stability. In addition, the setting of the maximum and minimum rate limits ensures that the transmission rate fluctuates within a reasonable range, avoiding transmission problems in extreme cases.

[0141] The embodiment of the application provides a bandwidth estimation system based on signal enhancement in a mobile scenario, comprising:

[0142] The first unit is configured to collect signal bandwidth data and an initial RSSI value at a preset sampling interval, convert the initial RSSI value into a normalized RSSI value through linear mapping, and perform sliding window mean filtering on the normalized RSSI value to obtain an RSSI value.

[0143] The second unit is configured to construct the signal bandwidth data into network transmission state features, reshape a plurality of historical bandwidth prediction samples into time sequence features, and construct a state space based on the RSSI value, the network transmission state features and the time sequence features.

[0144] The third unit is configured to input the state space into a deep reinforcement learning model, wherein the deep reinforcement learning model adopts an actor network-critic network structure, the actor network outputs a probability distribution of a bandwidth prediction interval, and the critic network evaluates long-term returns of a current state.

[0145] The fourth unit is configured to calculate a difference value between a current RSSI value and a previous RSSI value, trigger a bandwidth protection mechanism when the difference value is less than a dynamic threshold, and obtain an adjusted predicted bandwidth value; the fourth unit is further configured to feed back the adjusted predicted bandwidth value to a sending end through a real-time transmission control protocol, and the sending end adaptively adjusts a sending rate according to a current packet loss rate.

[0146] The fifth unit is configured to output the adjusted predicted bandwidth value and a minimum sending rate as a final target code rate, and use the final target code rate to guide network resource allocation.

[0147] The third aspect of the embodiment of the application provides an electronic device, comprising:

[0148] a processor;

[0149] a memory for storing processor-executable instructions;

[0150] The processor is configured to call the instructions stored in the memory to execute the method described above.

[0151] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon computer program instructions, which when executed by a processor, implement the method described above.

[0152] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0153] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions recorded in the above-mentioned embodiments can be modified, or some or all of the technical features thereof can be replaced equivalently; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep reinforcement learning bandwidth estimation method based on signal enhancement in mobile scenarios, characterized in that, include: Signal bandwidth data and initial RSSI values ​​are collected at preset sampling intervals; The initial RSSI value is converted into a normalized RSSI value through a linear mapping, and the normalized RSSI value is then subjected to a sliding window mean filter to obtain the RSSI value. The signal bandwidth data is used to construct network transmission state features, and the most recent historical bandwidth prediction samples are reshaped into time-series features. Based on the RSSI value, the network transmission state features, and the time-series features, a state space is constructed. The state space is input into a deep reinforcement learning model, which employs an actor network-critic network structure. The actor network outputs the probability distribution of the bandwidth prediction interval, and the critic network evaluates the long-term benefits of the current state. Calculate the difference between the current RSSI value and the RSSI value at the previous moment. When the difference is less than a dynamic threshold, trigger the bandwidth protection mechanism to obtain the adjusted predicted bandwidth value. The adjusted predicted bandwidth value is fed back to the sending end through a real-time transmission control protocol, and the sending end adaptively adjusts the transmission rate according to the current packet loss rate. The adjusted predicted bandwidth value and the minimum transmission rate are used as the final target bit rate output to guide network resource allocation.

2. The method according to claim 1, characterized in that, The initial RSSI value is converted to a normalized RSSI value through a linear mapping, and a sliding window mean filter is performed on the normalized RSSI value to obtain the RSSI value, including: Calculate the signal volatility of the normalized RSSI value within a sliding time window, the signal volatility being calculated based on the variance of the normalized RSSI value and the window mean; The window size of the sliding time window is dynamically adjusted according to the signal volatility. When the signal volatility is greater than the volatility threshold, the window size is set to the minimum window value, and when the signal volatility is less than the stability threshold, the window size is set to the maximum window value. The normalized RSSI values ​​within the sliding time window are subjected to weighted average filtering to obtain the RSSI values.

3. The method according to claim 1, characterized in that, The state space is input into a deep reinforcement learning model, which employs an actor-critic network structure. The actor network outputs a probability distribution of the bandwidth prediction interval, and the critic network evaluates the long-term benefits of the current state, including: The RSSI value is input into multiple fully connected layers of the deep reinforcement learning model to obtain a deep feature vector; The deep feature vector is input into the output layer of the actor network, which contains multiple neurons and uses the softmax activation function to obtain a multidimensional action probability distribution. The bandwidth prediction range is divided into multiple prediction intervals, and each element in the multidimensional action probability distribution corresponds to the selection probability of a prediction interval. Based on the action probability distribution, the bandwidth prediction interval with the highest probability is selected, and the bandwidth prediction interval is mapped to a specific bandwidth prediction value through a log-linear transformation. The deep feature vector is input into the output layer of the critic network to obtain the state value, and the temporal difference error is calculated based on the state value; the network parameters of the actor network are updated according to the temporal difference error.

4. The method according to claim 1, characterized in that, The method further includes: A reward function is constructed based on the receiving rate, packet delay, packet loss rate, and RSSI value. The deep reinforcement learning model is trained using a near-end policy optimization algorithm, including: The bandwidth gain component is obtained by logarithmically transforming the receiving rate, the delay is limited to a preset time range to obtain the delay penalty component, the packet loss rate is used as the transmission quality penalty component, the deviation between the RSSI value at the current moment and the RSSI value at the previous moment is calculated to obtain the signal fluctuation penalty component, and the bandwidth gain component, the delay penalty component, the transmission quality penalty component and the signal fluctuation penalty component are weighted and combined to obtain the reward function value at the current moment. The probability ratio is obtained by calculating the ratio of the new policy probability output by the actor network to the policy probability of the previous iteration; the advantage function value is obtained by weighted summing the differences between the reward function value and the state value from the current time to multiple subsequent time steps; the probability ratio is multiplied by the advantage function value, and a policy entropy penalty term is added to obtain the policy loss value. In each layer of the actor network, connections are randomly disconnected to form a sparse network structure. The gradient of the network parameters is calculated based on the policy loss value. The parameters in the sparse network structure are updated using the Adam optimizer to complete one policy iteration. The policy iteration is repeated until the discount cumulative reward converges.

5. The method according to claim 1, characterized in that, Calculate the difference between the current RSSI value and the RSSI value at the previous time. When the difference is less than a dynamic threshold, trigger a bandwidth protection mechanism to obtain an adjusted predicted bandwidth value, including: Calculate the deviation between the current RSSI value and the previous RSSI value, and take its absolute value as the RSSI difference; obtain the standard deviation and mean of the historical RSSI values, and perform a weighted summation of the standard deviation and the mean to obtain the dynamic threshold. When the RSSI difference is less than the dynamic threshold, a bandwidth protection mechanism is triggered; the ratio of the RSSI difference to the dynamic threshold is calculated, and the negative exponent of the ratio is used to obtain the bandwidth factor; the current predicted bandwidth value and the historical average bandwidth value are multiplied by the bandwidth factor and summed to obtain the protected bandwidth value; Calculate the absolute value of the difference between the protection bandwidth value and the historical average bandwidth value. When the absolute value of the difference is less than a preset adjustment range, the protection bandwidth value is used as the adjusted predicted bandwidth value. When the absolute value of the difference is greater than or equal to the preset adjustment range, the historical average bandwidth value plus or minus the preset adjustment range is used as the adjusted predicted bandwidth value.

6. The method according to claim 1, characterized in that, The adjusted predicted bandwidth value is fed back to the sending end via a real-time transmission control protocol. The sending end adaptively adjusts the transmission rate based on the current packet loss rate, including: The sending end counts the number of lost data packets and the total number of sent data packets within a preset time window, and divides the number of lost data packets by the total number of sent data packets to obtain the packet loss rate at the current moment. The packet loss level is determined based on the current packet loss rate. When the current packet loss rate is greater than the high packet loss threshold, a downward adjustment coefficient is obtained. When the current packet loss rate is less than the low packet loss threshold, an upward adjustment coefficient is obtained. The complement of the downward adjustment coefficient or the upward adjustment coefficient plus one is used as the rate adjustment coefficient. The new transmission rate is obtained by multiplying the current transmission rate by the rate adjustment coefficient. The new transmission rate and the transmission rate at the previous moment are then multiplied by the smoothing factor and the complement of the smoothing factor, respectively, and added together to obtain the smoothed transmission rate at the current moment. Multiply the adjusted predicted bandwidth value by the complement of the network jitter compensation coefficient to obtain the feedback bandwidth value; take the minimum value of the current smoothed transmission rate, the feedback bandwidth value, and the maximum rate limit, and take the maximum value of the minimum value and the minimum rate limit to obtain the final transmission rate.

7. A deep reinforcement learning bandwidth estimation system based on signal enhancement in mobile scenarios, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire signal bandwidth data and initial RSSI values ​​at a preset sampling interval; The initial RSSI value is converted into a normalized RSSI value through a linear mapping, and the normalized RSSI value is then subjected to a sliding window mean filter to obtain the RSSI value. The second unit is used to construct the signal bandwidth data into network transmission state features, reshape multiple recent historical bandwidth prediction samples into time-series features, and construct a state space based on the RSSI value, the network transmission state features, and the time-series features. The third unit is used to input the state space into a deep reinforcement learning model. The deep reinforcement learning model adopts an actor network-critic network structure. The actor network outputs the probability distribution of the bandwidth prediction interval, and the critic network evaluates the long-term benefits of the current state. The fourth unit is used to calculate the difference between the current RSSI value and the RSSI value at the previous moment. When the difference is less than the dynamic threshold, the bandwidth protection mechanism is triggered to obtain the adjusted predicted bandwidth value. The adjusted predicted bandwidth value is fed back to the sending end through a real-time transmission control protocol, and the sending end adaptively adjusts the transmission rate according to the current packet loss rate. The fifth unit is used to output the adjusted predicted bandwidth value and the minimum transmission rate as the final target bit rate to guide network resource allocation.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images