Bit rate adjustment method, device, electronic device and storage medium

Abstract

The present invention discloses a rate adjustment method, device, electronic device and storage medium, which belongs to the field of communication technology. The method includes: obtaining a communication data packet within the most recent preset interval time period, parsing the communication data packet to obtain packet-level information and frame-level information; obtaining a reward value corresponding to the most recent preset interval time period according to the packet-level information and frame-level information; forming a state vector according to the packet-level information, frame-level information and system information, inputting the state vector and reward value into a pre-trained neural network model to obtain a neural network model output; using a mapping function to map the neural network model output to a rate adjustment parameter; adjusting the encoder rate of the next preset interval time period according to the rate adjustment parameter. The present invention can quickly respond to a changing network environment, while efficiently utilizing bandwidth, ensuring low latency, low frame loss rate and high quality of video transmission, so as to achieve excellent user experience quality (QoE) and have better performance in congestion control.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a code rate adjustment method, device, electronic device and storage medium. Background Art

[0002] According to a Cisco report, video traffic will account for 82% of all internet traffic by 2022. In addition to traditional video streaming, real-time communication (RTC), including video calls, live broadcasts, and interactive video, has become another hot multimedia service due to its low latency. RTC can provide users with peer-to-peer latency of less than 500ms, bringing users a smoother viewing experience and shortening the distance between them. Given the current domestic and international environment, the demand for RTC is higher than ever.

[0003] Because RTC requires low latency and high video quality, its congestion control design is more challenging than traditional congestion control. Existing RTC congestion control can be divided into three types: rule-based, RL-based, and hybrid. These types of congestion control have the following main problems: traditional rule-based methods have difficulty tracking rapidly changing network bandwidth, resulting in low bandwidth utilization. In addition, video bitrates have a saturation state. Pursuing only high bitrates and high bandwidth utilization will lead to bandwidth waste and even more severe network congestion. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a rate adjustment method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a method for bit rate adjustment is provided, the method comprising:

[0006] Acquire a communication data packet within a recent preset interval time period, and parse the communication data packet to obtain packet-level information and frame-level information;

[0007] Obtaining a reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information;

[0008] forming a state vector according to the packet-level information, the frame-level information, and the system information, and inputting the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output;

[0009] Mapping the neural network model output to a bit rate adjustment parameter using a mapping function;

[0010] The encoder bit rate of the next preset interval time period is adjusted according to the bit rate adjustment parameter.

[0011] Optionally, obtaining the reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information further includes:

[0012] According to the packet-level information of the communication data packet, the following packet parameters are obtained through statistical analysis: packet reception rate within a preset interval time period, average packet delay within a preset interval time period and / or maximum burst packet loss ratio within a preset interval time period;

[0013] According to the frame-level information of the communication data packet, the following frame parameters are obtained through statistical analysis: the average frame peak signal-to-noise ratio (PSNR) value output by the encoder within a preset interval time period, and the average frame delay within the preset interval time period;

[0014] A reward value corresponding to the latest preset interval time period is determined according to the packet parameter and the frame parameter.

[0015] Optionally, the reward value is calculated as follows:

[0016]

[0017] Among them, rate pkt The packet receiving rate within the preset interval time period, delay pkt is the average packet delay within the preset interval, loss pkt The maximum burst packet loss ratio within the preset interval time period, PSNR enc The average frame peak signal-to-noise ratio (PSNR) value output by the encoder within the preset interval time period, delay frame is the average frame delay within the preset interval time period, rate pkt_r For r frame When the value is 1, the rate pkt The value of PSNR enc_r For r frame PSNR when the value is 1 enc The value of r frame is the ratio of the maximum frame resolution width to the maximum video resolution width within a preset interval, and α, β, γ, and δ are system constants.

[0018] Optionally, the packet-level information includes: packet reception rate, packet delay and / or packet loss rate; the frame-level information includes: frame quality, frame resolution, frame delay and / or frame loss rate.

[0019] Optionally, the state vector includes: the average packet delay within a preset interval time period, the average packet delay gradient between preset interval time periods, the maximum burst packet loss ratio within a preset interval time period, the average frame quality within a preset interval time period, the ratio of the maximum frame resolution width to the maximum video resolution width within the preset interval time period, the average frame delay within the preset interval time period, the average frame delay gradient between preset interval time periods and / or the neural network model output of the previous preset interval time period.

[0020] Optionally, the neural network model includes: an actor and critic network structure, wherein the actor network and the critic network share multiple preceding network layers, the multiple preceding network layers include several fully connected layers and a gated recurrent network layer, and the actor network also includes a Sigmoid layer.

[0021] Optionally, the mapping function is:

[0022]

[0023] Among them, g is the bit rate adjustment parameter, action is the output of the neural network model, mid is the output value of the neural network model when the mapping result is 1, max_inc is the maximum value of the mapping result corresponding to the output of the neural network model, and max_dec is the minimum value of the mapping result corresponding to the output of the neural network model.

[0024] According to another aspect of the present invention, a bit rate adjustment device is provided, the device comprising:

[0025] an acquisition module adapted to acquire communication data packets within a recent preset interval time period, and parse the communication data packets to obtain packet-level information and frame-level information;

[0026] a reward module, adapted to obtain a reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information;

[0027] a computing module adapted to form a state vector based on the packet-level information, the frame-level information, and the system information, and input the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output;

[0028] a mapping module adapted to map the output of the neural network model to a bit rate adjustment parameter for the next preset interval time period using a mapping function;

[0029] The adjustment module is adapted to adjust the encoder bit rate of the next preset interval time period according to the bit rate adjustment parameter.

[0030] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0031] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned bit rate adjustment method.

[0032] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned bit rate adjustment method.

[0033] According to the present invention, the rate adjustment scheme obtains communication data packets within the most recent preset interval, parses and statistically analyzes the communication data packets to obtain packet-level information and frame-level information; obtains a reward value corresponding to the most recent preset interval based on the packet-level information and frame-level information, and introduces the frame-level information into the reward value to achieve more accurate adjustment of the training and output of the neural network model; forms a state vector based on the packet-level information, frame-level information, and system information, inputs the state vector into a pre-trained neural network model, and uses the reward value as an input to adjust the output of the neural network model; to further obtain more accurate rate adjustment parameters, a mapping function is set and used to map the output of the neural network model to the rate adjustment parameter; and adjusts the encoder bit rate for the next preset interval based on the rate adjustment parameter. Since frame-level information is key information for video transmission and an important indicator for evaluating video transmission quality, the rate adjustment scheme provided by the present invention introduces frame-level information and fully considers the optimization and adjustment of frame-level information during the process of adjusting the video bit rate, thereby optimizing parameters such as frame delay and frame loss rate during video transmission, thereby ensuring low delay, low frame loss rate, and high quality video transmission. Furthermore, using the mapping function to obtain more accurate bitrate adjustment parameters allows for further optimization of the neural network's results based on historical experience, thus avoiding excessive bitrate adjustments that could lead to a decrease in video quality. In other words, the bitrate adjustment solution provided by the present invention can ensure high video quality with low latency and low frame loss, maximizing the final QoE and providing better congestion control.

[0034] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0036] Figure 1 A flow chart of a bit rate adjustment method provided in one embodiment of the present invention is shown;

[0037] Figure 2 A diagram showing a network architecture applicable to a bit rate adjustment method provided in one embodiment of the present invention;

[0038] Figure 3 shows a structural diagram of a neural network model provided in one embodiment of the present invention;

[0039] Figure 4 A schematic structural diagram of a bit rate adjustment device provided in one embodiment of the present invention is shown;

[0040] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0042] Figure 1 A flowchart illustrating an embodiment of a rate adjustment method of the present invention is provided. The method is applied to an electronic device, which may be a network device including a transmitting device and a receiving device, or a user's smart terminal. The rate adjustment method disclosed in the embodiment of the present invention includes the following steps:

[0043] Step 110: Acquire communication data packets within a recent preset interval time period, and parse the communication data packets to obtain packet-level information and frame-level information.

[0044] Step 120: Obtain a reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information.

[0045] Step 130: Form a state vector based on the packet-level information, the frame-level information, and the system information, and input the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output.

[0046] Step 140: Map the neural network model output to a bit rate adjustment parameter using a mapping function.

[0047] Step 150: Adjust the encoder bit rate for the next preset interval time period according to the bit rate adjustment parameter.

[0048] Exemplarily, the above method is applicable to Figure 2 In the network architecture shown, the network architecture includes an RTC transmitter and an RTC receiver. The above method and corresponding device can be used at the transmitter or the receiver, wherein "obtaining the communication data packet within the most recent preset interval time period" in step 110 includes obtaining packet-level data and frame-level data through the media channel and the dedicated channel respectively. Alternatively, the entire data packet can be obtained first and then decomposed to obtain packet-level data and frame-level data, and then the packet-level information and frame-level information can be parsed from the above data respectively. The communication data packet can be an RTP data packet or a data packet generated by other protocols. Of course, Figure 2 The method is only an example and does not constitute a limitation on the scope of protection. All methods of obtaining packet-level information and frame-level information and then performing calculations and processing are within the scope of protection of this embodiment.

[0049] Specifically, through Figure 2 The packet-level receive buffer in the packet-level receive buffer obtains packet-level information from the RTP packets received within the most recent preset interval (200ms, for example). For example, the average packet delay is calculated using the packet receive and send timestamps, the packet reception rate during this period is calculated using the packet size and number, and the average packet loss rate is calculated using the packet sequence number.

[0050] The frame-level receive buffer organizes RTP packets received within the last 200ms into frame-level data based on their content and extracts frame-level information from them. For example, it calculates average frame latency using the frame's receive and send timestamps, average peak signal-to-noise ratio (PSNR) using the frame data, and average frame quality using the average peak signal-to-noise ratio (PSNR). It also calculates the maximum frame resolution of these frames as the maximum frame resolution, and average frame loss rate using the frame sequence number.

[0051] In order to obtain the correlation between packet-level information and frame-level information and to mine the deep correlation information in the existing data, a reward value function that can reflect more correlation information is set in step 120 of this embodiment. The output of the neural network model is adjusted through this reward value function to provide more correlation information for training.

[0052] In step 130 of this embodiment, in order to better adapt to the changing network environment and improve bandwidth utilization, many researchers have tried to combine reinforcement learning with congestion control. Existing computing models such as R3Net (RL-based Recurrent Network for RTC) use deep reinforcement learning to directly output the bit rate based on the packet-level information in the RTCP data packet; while Hybrid Receiver-side Congestion Control (HRCC) combines deep reinforcement learning with GCC (Google Congestion Control). HRCC uses a deep neural network to output a multiplication factor to periodically adjust the congestion control output by GCC. These two algorithms are designed to track bandwidth changes better than GCC, but the above algorithms (including GCC) have two unconsidered issues as described below:

[0053] On the one hand, the aforementioned congestion control methods only consider packet-level information, whereas frame-level information is of greater concern to RTC systems. While packet-level information and frame-level information have a certain correlation, packet-level information cannot fully represent the content of frame-level information. Therefore, if congestion control can adjust the bitrate based on frame-level information, it can ultimately optimize Quality of Experience (QoE) more directly. In traditional video transmission systems, the video encoder and congestion control are often far apart, and congestion control does not access frame-level information. However, in RTC systems, congestion control is already connected to the encoder to a certain extent by guiding the encoder's bitrate. In this case, there is a greater possibility to further integrate congestion control with the encoder to directly access frame-level information to maximize the ultimate QoE.

[0054] On the other hand, RTC congestion control should aim to optimize the ultimate quality of experience (QoE). Existing congestion control algorithms primarily aim to achieve low latency, low packet loss, and high bandwidth utilization. However, in reality, improved bandwidth utilization does not necessarily correspond to improved video quality. QoE measures the user's viewing experience and is primarily composed of video quality, frame latency, and frame loss rate. Experiments have shown that when the video bitrate reaches a certain level, the rate of improvement in video quality tends to decline, at which point the video reaches a state of bitrate saturation.

[0055] Due to the conflict between video latency and video quality, RTC uses lower video frame rates and resolutions to ensure real-time performance. Furthermore, RTC is often used in video conferencing scenarios, where video content typically features only slight motion and relatively static scenes. For these reasons, the bit rate required for video transmitted by RTC is typically relatively low. Therefore, constantly pursuing high bandwidth utilization may only lead to wasted bandwidth. Furthermore, when network conditions suddenly deteriorate, sending video at an unnecessarily high bit rate will only lead to further network congestion. Therefore, it is crucial for RTC to be able to sense whether the current video bit rate is sufficient.

[0056] Additionally, GCC uses a delay-based state machine model along with packet loss-based rules to calculate bitrates based on network statistics collected from RTCP packets. However, in a rapidly changing network environment, the fixed rules within the delay-based state machine often fail to track rapid bandwidth changes. GCC's packet loss-based rules treat all packet losses as a sign of congestion, which can lead to significant bitrate degradation, to the point where packet loss-based rules dominate GCC.

[0057] Therefore, in order to overcome the above-mentioned defects, in addition to using the neural network model to calculate the bit rate adjustment parameters for congestion control, this embodiment also provides a specific input vector, which includes a state vector formed by packet-level information, the frame-level information, and system information, and also includes the reward value function generated in step 120.

[0058] Preferably, the neural network model is a deep reinforcement learning model. Unlike the traditional data-driven learning optimization objective function and iterative cycle, the learning process of reinforcement learning does not focus on using a quantitatively analyzable objective function to optimize the model, but on setting a learning agent and an interactive environment with clear rules so that the learning agent can choose behavior through decision-making methods in the environment and obtain more and more rewards, thereby training the algorithm to make the most optimized behavior. Specifically, the deep reinforcement learning training process can be roughly divided into the following steps: 1) Initialize the model, environment, and reward. 2) Perform the first behavior and calculate the next environment and the reward brought by this action through the behavior. 3) Repeat this iterative process in the new step environment, and calculate the loss function of the ideal reward and real-time reward every N steps. 4) Optimize the model parameters, reset the environment and reward, and continue to repeat the above process. Until the loss function converges to a reasonable range or the environment enters the final state.

[0059] Furthermore, in order to avoid congestion caused by excessive bit rate saturation, a mapping function is further set in this embodiment, which adjusts the relationship between the output of the neural network model and the mapping result of the mapping function, and uses the mapping result value of the mapping function as a bit rate adjustment parameter to adjust the encoder bit rate of the next preset interval time period.

[0060] In summary, combined Figure 1 and Figure 2 As shown, the method disclosed in the embodiment of the present invention first obtains packet-level information and frame-level information from the received RTP packet through packet-level and frame-level buffers and delivers it to the receiving controller. Secondly, the receiving controller processes the received packet-level information, frame-level information, and system information into a state vector, and at the same time calculates a reward function value based on these packet-level information and frame-level information to adjust the output of the neural network model; wherein the neural network model inputs the state and reward value and outputs the neural network output; finally, the neural network output is mapped to a bit rate adjustment parameter (multiplication factor) to calculate the bit rate of the next state interval time, and the bit rate is sent to the transmitting encoder to adjust the encoder bit rate to ensure high video quality under low latency and low frame loss conditions to maximize the final QoE.

[0061] The above solution can quickly respond to changing network environments while efficiently utilizing bandwidth to ensure low latency, low frame loss rate, and high quality of video transmission, thereby achieving excellent user experience quality (QoE). In addition, the neural network model is used to detect the current video transmission bitrate demand. If it detects that it is not saturated, it will continue to try to increase the bitrate while ensuring low latency and low frame loss. If it detects that it is saturated, it will maintain the bitrate at a relatively flat state.

[0062] In one or more embodiments, obtaining a reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information further includes:

[0063] According to the packet-level information of the communication data packet, the following packet parameters are obtained through statistical analysis: packet reception rate within a preset interval time period, average packet delay within a preset interval time period and / or maximum burst packet loss ratio within a preset interval time period.

[0064] According to the frame-level information of the communication data packet, the following frame parameters are obtained through statistical analysis: an average frame peak signal-to-noise ratio (PSNR) value output by the encoder within a preset interval time period, and an average frame delay within the preset interval time period.

[0065] A reward value corresponding to a latest preset interval time period is determined according to the packet parameter and the frame parameter.

[0066] The reward value is specifically used to optimize and adjust the training and output of the neural network model, continuously approaching the optimal result through the reward value. It should be noted that incorporating frame-level information when determining the reward value for training the neural network model can optimize frame-level information. Compared with related technologies that only consider packet-level information, this example can more directly optimize video quality.

[0067] In an exemplary embodiment, the calculation formula of the reward value may be the following step function:

[0068]

[0069] Among them, rate pkt The packet receiving rate within the preset interval time period, delay pkt is the average packet delay within the preset interval, loss pkt The maximum burst packet loss ratio within the preset interval time period, PSNR enc The average frame peak signal-to-noise ratio (PSNR) value output by the encoder within the preset interval time period, delay frame is the average frame delay within the preset interval time period, rate pkt_r For r frame When the value is 1, the rate pkt The value of PSNR enc_r For r frame PSNR when the value is 1 enc The value of r frame is the ratio of the maximum frame resolution width to the maximum video resolution width within a preset interval, and α, β, γ, and δ are system constants.

[0070] Specifically, the α, β, γ and δ are constants determined according to the specific conditions of the communication system. For example, in a specific example, the value of α is 16, the value of β is 6, the value of γ is 10, and the value of δ is 4 / 15.

[0071] In a preferred embodiment, the packet-level information obtained in step 110 includes: packet reception rate, packet delay, and / or packet loss rate. Specifically, the average packet delay can be calculated based on the packet reception and transmission timestamps, the packet reception rate during the period can be calculated based on the packet size and number, and the average packet loss rate can be calculated based on the packet sequence number.

[0072] The frame level information includes: frame quality, frame resolution, frame delay and / or frame loss rate. Figure 2As shown, the frame-level receive buffer preferably organizes the RTP packets received within the last 200ms into frame-level data based on their content and obtains frame-level information from them. For example, the average frame delay is calculated using the frame receive and send timestamps, the average peak signal-to-noise ratio (PSNR) of the frame data is used to calculate the average frame quality, the maximum resolution of these frames is calculated as the maximum frame resolution, and the average frame loss rate is calculated using the frame sequence number.

[0073] The above frame-level information can provide richer model input features, making model training more efficient and the model output more accurate in formal use.

[0074] Furthermore, in one embodiment, forming a state vector according to the packet-level information, the frame-level information, and the system information includes:

[0075] A packet state vector is obtained by performing statistical analysis on the packet-level information, a frame state vector is obtained by performing statistical analysis on the frame-level information, and the state vector is obtained by combining the system state vector obtained from the system information.

[0076] Among them, the state vector formed according to the packet-level information, the frame-level information, and the system information specifically includes: the average packet delay within a preset interval time period, the average packet delay gradient between preset interval time periods, the maximum burst packet loss ratio within a preset interval time period, the average frame quality within a preset interval time period, the ratio of the maximum frame resolution width to the maximum video resolution width within the preset interval time period, the average frame delay within the preset interval time period, and the average frame delay gradient between preset interval time periods.

[0077] In order to characterize the influence of the network system on the calculation of the neural network model, the vector formed by the system information includes the output of the neural network model in the last preset interval time period. Of course, the system information can also be characterized by methods such as vector addition and fusion.

[0078] Specifically, the above state vector is the following 8-dimensional vector:

[0079]

[0080] Among them, the package level information is d pkt is the average packet delay within the preset interval time period, is the average packet delay gradient between the preset interval time periods, l pkt is the maximum burst packet loss ratio within the preset interval time period; the frame level information is q frame The average frame quality within the preset interval time period, the frame quality is specifically the peak signal-to-noise ratio PSNR obtained in the encoder, r frameThe ratio of the maximum frame resolution width to the maximum video resolution width within the preset interval time period, d frame is the average frame delay within the preset interval time period, The average frame delay gradient between the preset interval time periods of the state; the system information is [a last ] is the previous output of the neural network model. Peak Signal-to-Noise Ratio (PSNR), short for "Peak Signal to Noise Ratio," is an objective metric for image evaluation. The state vector determined above considers packet-level information, frame-level information, and system information, providing a more comprehensive picture of the video transmission process.

[0081] In one embodiment, the neural network model is a neural network including a convolutional layer and a fully connected layer. Figure 3 As shown, the neural network model is an actor-critic network structure, wherein the actor network and the critic network share multiple preceding network layers, wherein the multiple preceding network layers include multiple fully connected layers FC, wherein at least some of the FC fully connected layers include activation functions Relu, and the preceding network layer also includes a gated recurrent network layer GRU, wherein GRU (Gate Recurrent Unit) is a type of recurrent neural network, and like LSTM (Long-Short Term Memory), is also proposed to solve problems such as long-term memory and gradient in back propagation.

[0082] In addition to the above network layers, the actor network also includes a Sigmoid layer. The Sigmoid function is an S-shaped function commonly seen in biology, also known as an S-shaped growth curve. In information science, due to its monotonic increasing and inverse monotonic increasing properties, the Sigmoid function is often used as an activation function of a neural network to map variables between (0, 1).

[0083] In a preferred embodiment, the mapping function is:

[0084]

[0085] Among them, g is the bit rate adjustment parameter, action is the output of the neural network model, mid is the output value of the neural network model when the mapping result is 1, max_inc is the maximum value in the mapping result corresponding to the output of the neural network model, and max_dec is the minimum value in the mapping result corresponding to the output of the neural network model. The mapping result involved in this embodiment is the bit rate adjustment parameter obtained by mapping the neural network output. By using the above-mentioned mapping function to convert the neural network output into the bit rate adjustment parameter, the result obtained by the neural network can be further optimized based on historical experience, thereby avoiding excessive adjustment of the bit rate and resulting in a decrease in video quality. That is, high video quality is guaranteed under low latency and low frame loss conditions to maximize the final QoE.

[0086] In a specific example, max_inc is the maximum value of the neural network output mapping result, and its specific value range is 1-1.5; max_dec is the minimum value of the neural network output mapping result, and its specific value range is 0.5-0.8.

[0087] Figure 4 FIG. 1 shows a schematic diagram of the structure of an embodiment of a bit rate adjustment device in the present invention. Figure 4 As shown, the apparatus 400 includes:

[0088] An acquisition module 410 is adapted to acquire communication data packets within a recent preset interval, and parse the communication data packets to obtain packet-level information and frame-level information;

[0089] A reward module 420, adapted to obtain a reward value corresponding to the most recent preset interval time period according to the packet-level information and the frame-level information;

[0090] a calculation module 430 adapted to form a state vector based on the packet-level information, the frame-level information, and the system information, and input the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output;

[0091] A mapping module 440 is adapted to map the output of the neural network model to a bit rate adjustment parameter for the next preset interval time period using a mapping function;

[0092] The adjustment module 450 is adapted to adjust the encoder bit rate of the next preset interval time period according to the bit rate adjustment parameter.

[0093] In one or more embodiments, the reward module 420 is further adapted to:

[0094] According to the packet-level information of the communication data packet, the following packet parameters are obtained through statistical analysis: packet reception rate within a preset interval time period, average packet delay within a preset interval time period and / or maximum burst packet loss ratio within a preset interval time period;

[0095] According to the frame-level information of the communication data packet, the following frame parameters are obtained through statistical analysis: the average frame peak signal-to-noise ratio (PSNR) value output by the encoder within a preset interval time period, and the average frame delay within the preset interval time period;

[0096] A reward value corresponding to the latest preset interval time period is determined according to the packet parameter and the frame parameter.

[0097] Preferably, the calculation formula of the reward value is:

[0098]

[0099] Among them, rate pkt The packet receiving rate within the preset interval time period, delay pkt is the average packet delay within the preset interval, loss pkt The maximum burst packet loss ratio within the preset interval time period, PSNR enc The average frame peak signal-to-noise ratio (PSNR) value output by the encoder within the preset interval time period, delay frame is the average frame delay within the preset interval time period, rate pkt_r For r frame When the value is 1, the rate pkt The value of PSNR enc_r For r frame PSNR when the value is 1 enc The value of r frame is the ratio of the maximum frame resolution width to the maximum video resolution width within a preset interval, and α, β, γ, and δ are system constants.

[0100] In one or some embodiments, the packet-level information includes: packet reception rate, packet delay and / or packet loss rate; the frame-level information includes: frame quality, frame resolution, frame delay and / or frame loss rate.

[0101] Forming a state vector according to the packet-level information, the frame-level information, and the system information includes:

[0102] Performing statistical analysis on the packet-level information in the calculation module 430 to obtain a packet state vector, performing statistical analysis on the frame-level information to obtain a frame state vector, and combining the system state vector obtained from the system information to obtain the state vector;

[0103] Among them, the state vector includes: the average packet delay within the preset interval time period, the average packet delay gradient between the preset interval time periods, the maximum burst packet loss ratio within the preset interval time period, the average frame quality within the preset interval time period, the ratio of the maximum frame resolution width to the maximum video resolution width within the preset interval time period, the average frame delay within the preset interval time period, the average frame delay gradient between the preset interval time periods and / or the neural network model output of the previous preset interval time period.

[0104] In a preferred embodiment, the neural network model includes: an actor and critic network structure, wherein the actor network and the critic network share multiple preceding network layers, the multiple preceding network layers include several fully connected layers and a gated recurrent network layer, and the actor network also includes a Sigmoid layer.

[0105] Preferably, the mapping function is:

[0106]

[0107] Among them, g is the bit rate adjustment parameter, action is the output of the neural network model, mid is the output value of the neural network model when the mapping result is 1, max_inc is the maximum value of the mapping result corresponding to the output of the neural network model, and max_dec is the minimum value of the mapping result corresponding to the output of the neural network model.

[0108] It should be pointed out that:

[0109] The rate adjustment method disclosed in this embodiment of the present invention is represented by E3C and compared with other advanced methods (GCC and HRCC) in the same simulated network environment. Mahimahi is used as its local network simulator. Mahimahi is a lightweight and composable network simulation tool that can simulate different cellular network traces using a given trace file. Several random network paths and cellular network paths were selected from the database, where the cellular network paths include different scenarios, such as driving, walking, and stationary.

[0110] Table 1 Performance comparison results of the present invention on random network paths and other methods

[0111]

[0112] Table 2 Performance comparison results of the present invention in cellular network path and other methods

[0113]

[0114]

[0115] Comparisons show that the reinforcement learning-based inter-layer fusion congestion control method E3C proposed in this paper reduces latency compared to existing GCC and HRCC, and also outperforms both GCC and HRCC in terms of video quality peak signal-to-noise ratio (PSNR) and VMAF scores. Although the average packet acceptance rate is the lowest among the three, this is because the method itself targets the highest possible video quality, not the highest possible throughput. Therefore, once the video bitrate reaches a certain saturation point, E3C maintains a relatively stable bitrate instead of increasing it. This results in a lower average packet acceptance rate compared to the other two methods, but in return, it achieves very good video quality, latency, and frame loss performance.

[0116] The reinforcement learning-based inter-layer fusion congestion control method and device provided in the above embodiments of the present invention provide very low frame delay and frame loss rate for video transmission in real-time communication systems, and on this basis maximize video quality while maintaining robustness to network mutations, thereby improving adaptability and feasibility in practical applications.

[0117] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the bit rate adjustment method in any of the above method embodiments.

[0118] Figure 5 A schematic structural diagram of an electronic device embodiment of the present invention is shown. The electronic device includes a communication network device, specifically a sending device or a receiving device. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0119] like Figure 5 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0120] Processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other devices, such as clients or other server network elements. Processor 502 is used to execute program 510, which may specifically perform the steps described in the aforementioned embodiment of the bit rate adjustment method for an electronic device.

[0121] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0122] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in an electronic device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0123] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0124] The program 510 may be specifically configured to enable the processor 502 to execute operations corresponding to the bit rate adjustment method provided in the above embodiment.

[0125] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems may also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0126] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0127] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0128] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0129] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0130] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0131] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A bit rate adjustment method, the method comprising: Acquire a communication data packet within a recent preset interval time period, and parse the communication data packet to obtain packet-level information and frame-level information; A reward value corresponding to the most recent preset interval is obtained based on the packet-level information and the frame-level information; wherein, based on the packet-level information of the communication data packet, the following packet parameters are obtained through statistical analysis: a packet reception rate within the preset interval, an average packet delay within the preset interval, and / or a maximum burst packet loss ratio within the preset interval; and based on the frame-level information of the communication data packet, the following frame parameters are obtained through statistical analysis: an average frame peak signal-to-noise ratio (PSNR) value output by an encoder within the preset interval, and an average frame delay within the preset interval; and a reward value corresponding to the most recent preset interval is determined based on the packet parameters and the frame parameters. forming a state vector according to the packet-level information, the frame-level information, and the system information, and inputting the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output; Mapping the neural network model output to a bit rate adjustment parameter using a mapping function; The encoder bit rate of the next preset interval time period is adjusted according to the bit rate adjustment parameter.

2. The method according to claim 1, characterized in that The calculation formula of the reward value is: Among them, rate pkt The packet receiving rate within the preset interval time period, delay pkt is the average packet delay within the preset interval, loss pkt The maximum burst packet loss ratio within the preset interval time period, PSNR enc The average frame peak signal-to-noise ratio (PSNR) value output by the encoder within the preset interval time period, delay frame is the average frame delay within the preset interval time period, rate pkt_r For r frame When the value is 1, the rate pkt The value of PSNR enc_r For r frame PSNR when the value is 1 enc The value of r frame is the ratio of the maximum frame resolution width to the maximum video resolution width within a preset interval, and α, β, γ, and δ are system constants.

3. The method according to claim 1, characterized in that The packet-level information includes: packet reception rate, packet delay and / or packet loss rate; the frame-level information includes: frame quality, frame resolution, frame delay and / or frame loss rate.

4. The method according to claim 3, characterized in that The state vector includes: the average packet delay within a preset interval time period, the average packet delay gradient between preset interval time periods, the maximum burst packet loss ratio within a preset interval time period, the average frame quality within a preset interval time period, the ratio of the maximum frame resolution width to the maximum video resolution width within the preset interval time period, the average frame delay within the preset interval time period, the average frame delay gradient between preset interval time periods and / or the neural network model output of the previous preset interval time period.

5. The method according to any one of claims 1 to 4, characterized in that The neural network model includes: an actor and critic network structure, wherein the actor network and the critic network share multiple preceding network layers, the multiple preceding network layers include several fully connected layers and a gated recurrent network layer, and the actor network also includes a Sigmoid layer.

6. The method according to any one of claims 1 to 4, characterized in that The mapping function is: Among them, g is the bit rate adjustment parameter, action is the output of the neural network model, mid is the output value of the neural network model when the mapping result is 1, max_inc is the maximum value of the mapping result corresponding to the output of the neural network model, and max_dec is the minimum value of the mapping result corresponding to the output of the neural network model.

7. A bit rate adjustment device, comprising: an acquisition module adapted to acquire communication data packets within a recent preset interval time period, and parse the communication data packets to obtain packet-level information and frame-level information; A reward module adapted to obtain a reward value corresponding to the most recent preset interval time period based on the packet-level information and the frame-level information; wherein, based on the packet-level information of the communication data packet, the following packet parameters are obtained through statistical analysis: a packet reception rate within the preset interval time period, an average packet delay within the preset interval time period, and / or a maximum burst packet loss ratio within the preset interval time period; and based on the frame-level information of the communication data packet, the following frame parameters are obtained through statistical analysis: an average frame peak signal-to-noise ratio (PSNR) value output by an encoder within the preset interval time period, and an average frame delay within the preset interval time period; and the reward value corresponding to the most recent preset interval time period is determined based on the packet parameters and the frame parameters. a computing module adapted to form a state vector based on the packet-level information, the frame-level information, and the system information, and input the state vector and the reward value into a pre-trained neural network model to obtain a neural network model output; a mapping module adapted to map the output of the neural network model to a bit rate adjustment parameter for the next preset interval time period using a mapping function; The adjustment module is adapted to adjust the encoder bit rate of the next preset interval time period according to the bit rate adjustment parameter.

8. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the bit rate adjustment method according to any one of claims 1 to 6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to perform operations corresponding to the bit rate adjustment method according to any one of claims 1 to 6.

Images