Video frame rendering frequency adjustment method, device, equipment and storage medium

Abstract

The application discloses a video frame rendering frequency adjustment method and device, equipment and storage medium, in the video frame rendering frequency adjustment process, when receiving the video frame sent by the sending end, the basic parameters of the video frame and the network transmission parameters of the sending end when transmitting the video frame are acquired, and the state parameter prediction value of the video frame received in the next time slice is predicted according to the basic parameters and the network transmission parameters, so that the rendering frequency of the video frame in the next time slice can be adjusted according to the state parameter prediction value of the video frame. Since the prediction is performed according to the basic parameters and the network transmission parameters, reliable, robust and real-time parameter adjustment basis can be provided for video frame playing, and after receiving each video frame, the rendering frame rate of the video frame is adaptively and dynamically adjusted according to the state parameter prediction value of the video frame, so that the real-time performance of the video frame rendering can be ensured, and the fluency of the video playing can be improved in a weak network environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the video processing technology, and particularly relates to a video frame rendering frequency adjustment method, device, equipment and storage medium. BACKGROUND

[0002] In the 5G remote driving technology, although the 5G network can provide the ability of large bandwidth and low latency network, when at the edge of the base station or between the base stations switching, the network edge bandwidth decreases, the time delay fluctuates and the network packet loss problem occurs, so the environment at the edge of the base station or between the base stations switching is called a weak network environment. In the weak network environment, in order to guarantee the fluency and low latency of the video, the existing technology usually uses the NACK (Non-Acknowledge) retransmission technology and the FEC (Forward Error Correction) technology to resist the weak network, so as to improve the fluency of the video playing in the weak network environment and reduce the transmission time delay of the video transmission in the weak network environment.

[0003] However, in the weak network scenario, the network packet loss and the network time delay coexist, when the network time delay fluctuates, the frequency of receiving the data packet also fluctuates. When the network state is poor, the video data packet is delayed to be received, which causes the picture freezing phenomenon of the client when playing the video; when the network state is good, multiple data packets are concentrated to be received, which causes the data packet buffer (used for temporarily storing data) to be accumulated, and the video time delay is increased. Although the single NACK retransmission and the FEC technology can resist the network packet loss to a certain extent and improve the video freezing phenomenon, the data packet transmission time delay is increased, even greater transmission time delay fluctuation is caused, the time for the sending end to send the data packet to the receiving end to receive the data packet is increased, the video fluency is poor, and the phenomena such as playing picture freezing, delay lag or rapid flashing are prone to occur, and the subjective experience is poor. SUMMARY

[0004] The purpose of the embodiment of the present application is to provide a video frame rendering frequency adjustment method, device, equipment and storage medium, which can reduce the transmission time delay in the weak network environment, guarantee the real-time of the video frame rendering, and improve the fluency of the video playing.

[0005] To achieve the above purpose, the embodiment of the present application provides a video frame rendering frequency adjustment method, comprising:

[0006] receiving the video frame sent by the sending end;

[0007] obtaining the basic parameters of the video frame and the network transmission parameters when the video frame is transmitted; wherein the network transmission parameters include the network time delay, the time delay fluctuation and the network packet loss rate.

[0008] predicting a state parameter prediction value of a video frame received in a next time slice according to the basic parameter and the network transmission parameter;

[0009] adjusting a video frame rendering frequency in the next time slice according to the state parameter prediction value.

[0010] As an improvement of the above scheme, the state parameter prediction value includes a predicted video frame number and a predicted receiving delay in the next time slice; and the adjusting a video frame rendering frequency in the next time slice according to the state parameter prediction value includes:

[0011] calculating a target rendering frequency in the next time slice according to the predicted video frame number and the predicted receiving delay;

[0012] adjusting a video frame rendering frequency in the next time slice according to the target rendering frequency.

[0013] As an improvement of the above scheme, the sending end sends the video frame according to a preset weak network resistance mode; wherein the weak network resistance mode is determined according to the network transmission parameter and a bandwidth parameter.

[0014] As an improvement of the above scheme, the weak network resistance mode includes an anti-packet loss mode, a low code rate mode and a low frame rate mode; wherein the anti-packet loss mode is divided into different anti-packet loss strengths according to selective application of non-acknowledgement retransmission technology and forward error correction technology, the encoding code rate in the low code rate mode is calculated according to a historical maximum receiving code rate and an estimated code rate in a current period, and the packet sending frequency in the low frame rate mode is calculated according to a packet sending frequency in a previous period.

[0015] As an improvement of the above scheme, the basic parameter includes a real-time transmission number and a size of the video frame, and the network transmission parameter includes a network delay, a delay jitter and a network packet loss rate.

[0016] As an improvement of the above scheme, the predicting a state parameter prediction value of a video frame received in a next time slice according to the basic parameter and the network transmission parameter includes:

[0017] generating an input control quantity at a current time according to the basic parameter and the network transmission parameter;

[0018] inputting the input control quantity into a video frame state parameter prediction model, so that the video frame state parameter prediction model outputs a state parameter prediction value of a video frame received in a next time slice.

[0019] As an improvement of the above scheme, the video frame state parameter prediction model includes a state equation and a state update equation, wherein,

[0020] The output of the state equation is the state parameter prediction value, and the input of the state equation includes the input control quantity, the state parameter estimation value of the current time slice calculated through the state update equation and the process noise;

[0021] The output of the state update equation is the updated state parameter estimation value of the current time slice, and the input of the state update equation is the state parameter estimation value of the current time slice, the video frame state measurement value of the current time slice, the measurement matrix and the Kalman gain.

[0022] As an improvement of the above scheme, the video frame state parameter prediction model further includes a covariance equation of the predicted state and a covariance update equation; wherein,

[0023] The output of the covariance equation is the covariance matrix of the state parameter prediction value, and the input of the covariance equation is the covariance matrix corresponding to the state parameter estimation value of the current time slice calculated through the covariance update equation and the covariance matrix corresponding to the process noise;

[0024] The output of the covariance update equation is the updated covariance matrix corresponding to the state parameter estimation value of the current time slice, and the input of the covariance update equation is the unit matrix and the covariance matrix corresponding to the state parameter estimation value of the current time slice.

[0025] As an improvement of the above scheme, the target rendering frequency of the next time slice is calculated according to the predicted video frame quantity and the predicted receiving time delay, including:

[0026] Obtaining buffer cache parameters and a current rendering frequency;

[0027] Calculating a frequency adjustment value according to the buffer cache parameters, the predicted video frame quantity and the predicted receiving time delay;

[0028] Calculating the ratio of the frequency adjustment value and a preset smoothing factor;

[0029] The sum of the current rendering frequency and the ratio is the target rendering frequency of the next time slice.

[0030] To achieve the above object, the embodiment of the present application further provides a video frame rendering frequency adjustment device, which comprises:

[0031] A video frame receiving module is configured to receive video frames sent by a sending end;

[0032] A parameter obtaining module is configured to obtain basic parameters of the video frames and network transmission parameters during transmission of the video frames; wherein, the network transmission parameters include network time delay, time delay jitter and network packet loss rate;

[0033] a prediction module configured to predict a state parameter prediction value of a video frame received in a next time slice according to the basic parameter and the network transmission parameter;

[0034] a rendering frequency adjustment module configured to adjust a video frame rendering frequency in the next time slice according to the state parameter prediction value.

[0035] To achieve the above object, the embodiment of the present application further provides a video frame rendering frequency adjustment device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to realize the video frame rendering frequency adjustment method as described in any of the above embodiments.

[0036] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the video frame rendering frequency adjustment method as described in any of the above embodiments when the computer program runs.

[0037] Compared with the prior art, the video frame rendering frequency adjustment method, device, equipment and storage medium disclosed by the present application, when receiving a video frame sent by a sending end, acquire a basic parameter of the video frame and a network transmission parameter of the sending end when transmitting the video frame, and predict a state parameter prediction value of a video frame received in a next time slice according to the basic parameter and the network transmission parameter, so as to adjust a rendering frequency of the video frame in the next time slice according to the state parameter prediction value of the video frame. Since the prediction is performed according to the basic parameter and the network transmission parameter, a reliable, robust and real-time parameter adjustment basis can be provided for video frame playing, and after receiving each video frame, the rendering frame rate of the video frame is adaptively and dynamically adjusted according to the state parameter prediction value of the video frame, so that the real-time performance of video frame rendering can be ensured and the fluency of video playing can be improved in a weak network environment. In addition, the sending end can match a corresponding working mode according to a current network state, intelligently select an anti-weak network algorithm, can fully utilize network resources and has stronger anti-weak network capability. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a video processing flowchart of the sending end and the receiving end provided by the embodiment of the present application;

[0039] Figure 2 is a flowchart of a video frame rendering frequency adjustment method provided by the embodiment of the present application;

[0040] Figure 3 is a structural block diagram of a video frame rendering frequency adjustment device provided by the embodiment of the present application;

[0041] Figure 4 The structural block diagram of the rendering frequency adjustment module in the video frame rendering frequency adjustment device provided by the embodiment of the present application is shown in FIG. 1.

[0042] Figure 5 The structural block diagram of the video frame rendering frequency adjustment device provided by the embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0044] Referring to Figure 1 , Figure 1 The video processing flowchart of the sending end and the receiving end provided by the embodiment of the present application is shown in FIG. 2. The sending end is used to collect video image data and encode the image data to send the data to the receiving end. The receiving end receives the data and decodes the data to render and play. In order to solve the phenomena of playing lag, delay lag, rapid flashing and the like caused by transmitting video data in a weak network scenario, the present application proposes a video rendering frequency adjustment method based on adaptive anti-weak network technology. An adaptive anti-weak network algorithm module is added to the sending end. The network state is estimated according to the sending and receiving conditions of the video frame during image encoding, and the anti-weak network mode is adaptively selected. At the same time, a video state prediction module and a frame rate dynamic adjustment module are added to the receiving end. The video state prediction module predicts the state parameters of the video frame of the next time slice, and provides a basis for the dynamic adjustment of the video rendering frame rate by the frame rate dynamic adjustment module. The receiving end predicts the receiving condition of the video frame of the next time slice according to the network transmission parameters and the basic parameters of the current video frame, generates a state parameter prediction value, and then adaptively adjusts the video frame rendering frame rate according to the state parameter prediction value. In a weak network environment, the real-time performance of video frame rendering can be ensured, and the fluency of video playing can be improved.

[0045] The sending end sequentially sends data packets, the data packets are transmitted by the network, and the data packets are received by the receiving end. The sending end and the receiving end periodically send information interaction packets. According to the condition of receiving the data packets and the condition of the periodic information interaction between the sending end and the receiving end, the network state is estimated, and network transmission parameters and bandwidth parameters are obtained. The network transmission parameters include network delay, delay jitter and network packet loss rate, and the bandwidth parameter is a code rate. The sending end sends the video frame to the receiving end according to a preset anti-weak network mode. The anti-weak network mode is determined according to the network transmission parameters and the bandwidth parameters. It is worth noting that the network transmission parameters and the bandwidth parameters are estimated values, which are calculated according to the following calculation method.

[0046] Specifically, the network latency refers to the network RTT (Round-Trip Time) latency, which is the round-trip time of a data packet during network transmission, i.e., the time elapsed from when the sender begins sending data to when the sender receives an acknowledgment from the receiver. A packet interaction process includes sender A sending data to receiver B, and receiver B returning an acknowledgment to sender A. The network RTT latency can be estimated simultaneously at both ends A and B.

[0047] Specifically, the network latency estimation method includes: obtaining the current timestamp T of the data packet sent by the sending end. A2 And obtain the last reception time T of the last information packet returned by the sender to the receiver. A1 ; Calculate the transmission delay DT based on the current transmission time and the previous reception time. A1 ; Obtain the last transmission time T of the information packet sent by the receiver, as mentioned in the last information packet received by the sending end. B1 The sending end will send the current timestamp T A2 Sending delay DT A1 and the previous sending time T B1 The information is sent to the receiving end in the form of a packet; the current reception time T of the receiving end receiving the information packet is obtained. B2 According to the previous transmission time T B1 Current receiving time T B2 and transmission delay DT A1 Calculate network latency.

[0048] For example, taking end A as the sender and end B as the receiver, the current timestamp T of the information packet sent by end A is recorded. A2 Get the last received time T of the information packet sent by B that A last received. A1 Calculate the transmission delay DT A1 It satisfies the following formula:

[0049] DT A1 =T A2 -T A1 (1).

[0050] If end A has never received a data packet from end B, then DT A1 =0. Furthermore, endpoint A also records the last time endpoint A received a packet from endpoint B, specifically the time T when endpoint B sent that packet. B1 If no information packet has ever been received from the B party, then T B1 =0. A terminal will T A2 T B1 DTA1 Packing to information package, sending, B end receives information package.

[0051] B end records the current receiving time T of receiving information package B2 , takes out T A2 , T B1 , DT A1 information in information package, calculates network delay RTT B , satisfies the following formula:

[0052] RTT B =T B2 -T B1 -DT A1 (2)。

[0053] Wherein, if T B1 =0 or DT A1 =0, this RTT time is invalid.

[0054] It is worth mentioning that when B end is as a receiving end and A end is as a sending end, B end sends information package in the next cycle according to the above mode, and when A end receives information package of B end, calculates network delay RTT A according to formula (2). The receiving end and the sending end will periodically send information package with time stamp, through the above method, in addition to the first round of information interaction, any end can estimate network delay once every time receiving information package.

[0055] Specifically, the delay jitter refers to the difference between the data packet time interval of a pair of data packets at the receiving end and the sending end, and the delay jitter can be obtained by estimating the statistical variance of network transmission time. In order to simplify the calculation, the packet group is used for jitter estimation, that is, the packets with a time interval between received packets within a threshold of milliseconds (for example, 5 milliseconds) are a packet group, the delay jitter is estimated at the receiving end, and the sending end is transmitted through the information package, the delay jitter between the packet groups is calculated, and the following calculation formula is satisfied:

[0056] D i,j =(R j -R i )-(S j -S i ) (3);

[0057]

[0058] Wherein, S i is the sending time of packet group i; S j is the sending time of packet group j; R i is the receiving time of packet group i; R j is the receiving time of packet group j; D i,jis the delay gradient of the packet group i relative to the packet group j; D i,j-1 is the delay gradient of the packet group i relative to the last packet group j; J i is the network transmission jitter estimation at the time of the packet group i; J i-1 is the network transmission jitter estimation at the time of the last packet group i.

[0059] Specifically, the network packet loss rate refers to the number of lost packets divided by the expected number of packets in a certain period, and the network packet loss rate is estimated at the receiving end and transmitted to the sending end through information packets. The calculation formula is as follows:

[0060] Exp i = sn i - sn0+ 1 (5) ;

[0061] Exp Δi = Exp i - Exp i-1 (6) ;

[0063] N Δi = N i - N i-1 (7) ;

[0064]

[0065] wherein, sn i is the maximum packet sequence number at the current time; sn0 is the sequence number of the first packet; Exp i is the total number of expected packets received up to the time of period i; Exp Δi is the number of expected packets received during the period i-1 to the period i; N i is the total number of actual packets received up to the time of period i, in order to exclude the influence of retransmitted packets, retransmitted packets are not counted in the statistics of the received packet number N; N Δi is the number of actual packets received during the period i-1 to the period i; LR i is the network packet loss rate at the time of the period i.

[0066] Specifically, the code rate estimation method comprises: the receiving end feeds back the delay gradient data of the calculated network delay to the sending end, the sending end caches the delay gradient data, the sending end takes a segment of the cached data, and a linear regression is used to fit to obtain a delay gradient trend T L , the delay gradient trend T L is compared with a preset first threshold T N1 , a second threshold T N2 , T N1 < T N2, to obtain the overload condition of the current network, and the target code rate is calculated according to the overload condition. The code rate estimation is calculated at the sending end and transmitted to the receiving end through the information packet, satisfying the following formula:

[0067]

[0068] wherein Bt i is the estimated code rate value at time i; R i is the maximum received code rate in the past 500 milliseconds; and a and β are multiplicative coefficients, a can take an empirical value of 0.85, and β can take an empirical value of 1.08. When T L > T N2 , it indicates that the network is overloaded; when T L < T N1 , it indicates that the network is underloaded; and when T N1 ≤ T L ≤ T N2 , it indicates that the network is normal.

[0069] Specifically, the sending end sends the video frame according to a preset weak network resistance mode; wherein the weak network resistance mode is determined according to the network delay, delay jitter, network packet loss rate and code rate calculated by the above estimation method. The weak network resistance mode includes an anti-packet loss mode, a low code rate mode and a low frame rate mode; wherein the anti-packet loss mode is divided into different anti-packet loss strengths according to the selective application of the non-acknowledgement retransmission technology and the forward error correction technology (i.e. turning on the non-acknowledgement retransmission technology + turning on / off the forward error correction technology), the encoding code rate in the low code rate mode is calculated according to the historical maximum received code rate and the estimated code rate of the current period, and the packet sending frequency in the low frame rate mode is calculated according to the packet sending frequency of the last period.

[0070] For example, the anti-packet loss mode includes the following three modes:

[0071] ① Low-intensity anti-packet loss mode, in which the FEC algorithm function is turned off and only the NACK retransmission mechanism is turned on. In this mode, the NACK retransmission is completely relied on to resist data packet loss.

[0072] ② Medium-intensity anti-packet loss mode, in which the FEC algorithm function is turned on and low-intensity FEC and NACK retransmission are used to resist packet loss. In this mode, the low-intensity FEC algorithm resists a part of the packet loss, reduces the packet loss rate, reduces the probability of NACK retransmission, and reduces the video transmission delay; the remaining part of the packet loss rate is handed over to the NACK retransmission algorithm to avoid too much bandwidth waste caused by FEC. The intensity of the FEC algorithm is adjusted by adjusting the number of redundant packets in the FEC algorithm. For example, the intensity of the FEC algorithm is determined by the algorithm parameters of the FEC, and the number of FEC redundant packets is adjusted so that the packet loss rate is reduced to a certain value T L1 , such as T L1= 5%. The number of redundant packets is adjusted according to the following formula:

[0073]

[0074] m = r-n (11);

[0075] P n,m <T L1 (12).

[0076] where n is the number of original packets; m is the number of redundant packets; r is the redundant packet ratio; p is the original packet loss rate; T L1 is the packet loss rate threshold; P n,m is the probability that the data cannot be recovered after encoding m redundant packets based on n original packets, i.e., the packet loss rate. That is, the "remaining packet loss rate" after the FEC algorithm, which needs to be compensated by other means. A redundant packet ratio list R list is defined, and the remaining packet loss rate after the FEC algorithm is calculated according to formulas (10)-(12) by traversing the list. From R list , a redundant packet ratio r1 value closest to T L1 is found, and the number of redundant packets m is calculated accordingly. k is a variable, and the range is to m-n+m.

[0077] ③ High-strength anti-packet loss mode, FEC algorithm function is enabled, high-strength FEC and NACK retransmission are used to resist packet loss. In this mode, the FEC algorithm is mainly used to resist packet loss, and the probability of NACK retransmission is maximized to ensure video latency. According to formulas (10)-(12), a redundant packet ratio r2 value closest to T list is selected from R L2 , and the number of redundant packets m is calculated to achieve high anti-packet loss, T L2 needs to satisfy the following formula:

[0078] T L2 <T L1 (13).

[0079] For example, the low code rate mode needs to reduce the video encoding code rate, which can be adjusted according to formula (9).

[0080] For example, the low frame rate mode needs to reduce the packet sending frequency of the sending end, and each adjustment is Y times the last cycle, 0<Y<1, such as Y=0.85 times.

[0081] Specifically, according to the network transmission parameter and the bandwidth parameter, different anti-weak network mode combinations are adaptively selected to comprehensively optimize the data packet transmission, the sending end needs to apply at least one of three different intensity modes in the anti-packet loss mode, and the low code rate mode and the low frame rate mode are selectively applied, and the selection rule of the anti-weak network mode can refer to Table 1:

[0082] Table 1 shows the anti-weak network mode combination selection of the sending end

[0083]

[0084]

[0085] When the packet loss rate is less than the threshold T, only a small amount of data packet retransmission is needed, under this condition, the FEC does not need to be started, only the NACK retransmission algorithm (low intensity anti-packet loss mode) is used, which can resist packet loss and will not cause bandwidth waste.

[0086] When the packet loss rate is greater than the threshold T, only the NACK retransmission algorithm will have the phenomenon of repeated loss of the same packet, which will cause the transmission delay of the packet to increase linearly, under this condition, the FEC function needs to be started to reduce the probability of packet retransmission, and the intensity of the FEC is determined by the network RTT delay, if the RTT delay is high, we should try to reduce the probability of data packet retransmission, that is, under the condition of bandwidth, the intensity of the FEC is increased, if the RTT delay is low, the data packet retransmission probability is controlled within a certain range, the probability of repeated loss of the same packet is reduced, and the transmission delay is controlled within an acceptable range, while avoiding too much bandwidth waste caused by high intensity FEC.

[0087] When the bandwidth estimation value is less than the current coding code rate, the low code rate mode is started, the coding code rate is actively reduced, and the other anti-weak network modes are cooperated to work. It is worth noting that under this condition, the high intensity FEC mode will cause the network bandwidth occupation to increase, the network environment to deteriorate, the network congestion to cause, and the packet loss and delay to be greater, therefore, the high intensity FEC algorithm should be avoided under the condition of limited bandwidth.

[0088] When the delay jitter is greater than the video acquisition period, the low frame rate mode is started, the packet sending frequency is reduced, the data packet disorder phenomenon is alleviated, and the other anti-weak network modes are cooperated to work to achieve better anti-weak network effect.

[0089] In the embodiment of the application, a plurality of anti-weak network modes are defined in advance, the sending end can match the corresponding working mode according to the current network state, intelligently select the anti-weak network algorithm, fully utilize the network resources, and has stronger anti-weak network capability.

[0090] Referring to Figure 2 , Figure 2is a flow chart of a video frame rendering frequency adjustment method provided by an embodiment of the present application, the video frame rendering frequency adjustment method is implemented by a receiving end, and the video frame rendering frequency adjustment method comprises the following steps:

[0091] S1, receiving a video frame sent by a sending end;

[0092] S2, acquiring basic parameters of the video frame and network transmission parameters during transmission of the video frame;

[0093] S3, predicting a state parameter prediction value of a video frame received in a next time slice according to the basic parameters and the network transmission parameters;

[0094] S4, adjusting a video frame rendering frequency in the next time slice according to the state parameter prediction value.

[0095] Specifically, in step S1, the sending end collects images at a fixed frame rate, encodes the images into a plurality of video data packets, transmits the video data packets through a network, and receives the video data packets by the receiving end. The receiving end must recombine and frame the plurality of data packets and send the data packets to a decoder for decoding and playing. Unlike the delay and jitter estimation of a single data packet, the transmission state of a complete video frame is modeled and predicted, which is more conducive to the decision of the playing end.

[0096] Specifically, in step S2, the basic parameters of the video frame and the network transmission parameters during transmission of the video frame are acquired, wherein the basic parameters include the real-time transmission quantity and size of the video frame, and the network transmission parameters include network delay, delay jitter and network packet loss rate.

[0097] Specifically, in step S3, the state parameter prediction value of the video frame received in the next time slice is predicted according to the basic parameters and the network transmission parameters, which comprises: generating an input control quantity at the current time according to the basic parameters and the network transmission parameters; inputting the input control quantity into a video frame state parameter prediction model, so that the video frame state parameter prediction model outputs the state parameter prediction value of the video frame received in the next time slice.

[0098] For example, the transmission state of the video frame is Z=[N, D] T , N is the number of video frames, D is the delay of the video frame, the transmission state of the video frame is related to the frame rate F of the camera (the number of video frames I can be obtained), the size S of the video frame, the network delay RTT, the network jitter J, and the network packet loss rate LR. Let the input control quantity be U=[I, S, RTT, J, LR] T , which is input into a video frame state parameter prediction model, and the video frame state parameter prediction model is a Kalman prediction mathematical model.

[0099] Specifically, the video frame state parameter prediction model comprises a state equation and a state update equation, wherein an output of the state equation is the state parameter prediction value, inputs of the state equation comprise the input control quantity, the state parameter estimation value of the current time slice calculated through the state update equation and process noise, and the state equation can refer to formula (14); an output of the state update equation is the updated state parameter estimation value of the current time slice, and inputs of the state update equation are the state parameter estimation value of the current time slice, the video frame state measurement value of the current time slice, a measurement matrix and Kalman gain, and the state update equation can refer to formula (17).

[0100] Further, the video frame state parameter prediction model further comprises a covariance equation of a predicted state and a covariance update equation; wherein an output of the covariance equation is a covariance matrix of the state parameter prediction value, inputs of the covariance equation are the covariance matrix corresponding to the state parameter estimation value of the current time slice calculated through the covariance update equation and the covariance matrix corresponding to the process noise, and the covariance equation can refer to formula (16); an output of the covariance update equation is the updated covariance matrix corresponding to the state parameter estimation value of the current time slice, and inputs of the covariance update equation are a unit matrix and the covariance matrix corresponding to the state parameter estimation value of the current time slice, and the covariance update equation can refer to formula (19).

[0101] The video frame state parameter prediction model specifically comprises the following equations:

[0102] (1) State equation:

[0103]

[0104] wherein, is the state parameter prediction value of the video frame at t+1 moment; is the state parameter estimation value of the video frame at t moment updated through formula (17) below; A is a state transition matrix, which is a 2x2 matrix; B is a control matrix, which is a 2x5 matrix; U t is the system input at t moment, U t satisfies: I t is the number of complete video frames input between time slices t-1 and t, and is fixed as 1 (exactly one video coding period), is the average video frame size of time slice t, The calculation method is as follows:

[0105]

[0106] wherein, S tThe value is the size of a complete video frame formed by time slice t. If no complete video frame is successfully formed between time slices t-1 and t, the value is taken as the previous value S. t-1 And so on; φ is the sliding smoothing coefficient, taken as an empirical value of 0.9; J t Network jitter for time slice t; LR t Let W be the network packet loss rate for time slice t; W is the process noise, which follows a zero-mean Gaussian distribution W ~ N(0,R), where R is the standard deviation.

[0107] (2) Covariance equation for predicted state:

[0108]

[0109] in, It is the predicted value of the video frame state parameters at time t+1. The covariance matrix; The estimated value of the video frame state parameters at time t Q is the covariance matrix of W; Q is the covariance matrix of W.

[0110] (3) State update equation:

[0111]

[0112] in, It is the estimated value of the video frame state parameters for the current time slice before the state update, based on the prediction completed in the previous time slice t-1; Z t H is the state measurement value of the video frame at time t; H is the measurement matrix, which transforms the state variables into predicted observations, and is a 2x2 matrix; K g It is the Kalman gain, which satisfies the following formula:

[0113]

[0114] (4) Covariance update equation for the state:

[0115]

[0116] Where E is the identity matrix, It is the predicted value of the video frame state at time t. The covariance matrix.

[0117] For example, with the camera's frame rate F as the time slice t, the following conditions are met: The network delay (RTT) for the current time slice t is calculated using formula (2). t The time delay jitter value J for the current time slice t is calculated using formula (4). t The packet loss rate LR for time slice t is calculated using formula (8). tThe average video frame size of the time slice t is calculated by formula (15) The input control quantity U of the current time slice t t is input into the video frame state parameter prediction model. The actual video frame quantity N t received between the time when the previous video frame was received and the current time slice t t and the delay D experienced in between t are the actual video frame state Z of the current time slice t = [N t , D T ] t The error between the video state parameter estimation value of the current time slice and the actual video frame state Z of the current time slice and The video frame state parameter prediction value of the next time slice is predicted according to formula (14) by the input U of the current time slice t t and the estimation value of the video frame state

[0118] Specifically, in step S4, the state parameter prediction value includes the predicted video frame quantity and the predicted receiving delay of the next time slice; then, the adjusting the video frame rendering frequency in the next time slice according to the state parameter prediction value includes:

[0119] S41, calculating the target rendering frequency of the next time slice according to the predicted video frame quantity and the predicted receiving delay;

[0120] S42, adjusting the video frame rendering frequency in the next time slice according to the target rendering frequency.

[0121] For example, since the actual video frame state of the current time slice satisfies Z t = [N t , D t ] T , the video frame state parameter prediction value output by the video frame state parameter prediction model includes the predicted video frame quantity N t+1 and the predicted receiving delay D t+1 of the next time slice. According to the predicted video frame quantity N t+1 and the predicted receiving delay D t+1The target rendering frequency of the next time slice is calculated according to the predicted video frame quantity and the predicted receiving time delay, and after the video frame is rendered (played) according to the current rendering frequency, the rendering frequency is adjusted to the target rendering frequency, so that after the video frame is received in the next time slice, the video frame is directly rendered according to the target rendering frequency, thereby predicting the target rendering frequency of the next time receiving video frame every time the video frame is received after the first time, and realizing dynamic adjustment of the rendering frequency.

[0122] Specifically, the target rendering frequency of the next time slice is calculated according to the predicted video frame quantity and the predicted receiving time delay in step S41, comprising:

[0123] S411, obtaining buffer cache parameters and a current rendering frequency;

[0124] S412, calculating a frequency adjustment value according to the buffer cache parameters, the predicted video frame quantity and the predicted receiving time delay;

[0125] S413, calculating a ratio of the frequency adjustment value and a preset smoothing factor; and taking a sum of the current rendering frequency and the ratio as the target rendering frequency of the next time slice.

[0126] For example, after obtaining the predicted video frame quantity N t+1 and the predicted receiving time delay D t+1 , and the current buffer cache value M t , the target rendering frequency F t+1 is dynamically adjusted, and the adjustment rule is as follows:

[0127] (20);

[0129]

[0130] F t+1 ≤F max (22);

[0131] F t+1 ≥F min (23);

[0132] Wherein, M avg is the average value of the buffer cache in the previous 500 milliseconds; σ is a smoothing factor, that is, how many cycles are needed to offset the buffer fluctuations, the greater the value, the slower the response, and the more stable the frame rate adjustment, and the preferred experience value is 3; F max is the maximum value of the limited rendering frame rate, and the preferred experience value is 50fps; F min is the minimum value of the limited rendering frame rate, and the preferred experience value is 15fps.

[0133] In the embodiment of the application, if the buffer is empty, no video frame can be rendered, and the screen is stuck; if the buffer is full, the playing screen is delayed. The method can predict the receiving state of the video frame in the future, dynamically adjust the rendering period in advance, optimize the number of video frames in the buffer, maintain the number at a suitable value, reduce the probability of buffer emptying and buffer accumulation, guarantee the video delay, and improve the smoothness of the video.

[0134] Further, Table 2 is experimental data using the present application, and the experimental object is a video source with a resolution of 1080p and a frame rate of 30 frames. The experimental scene is real-time video transmission under different network (delay + packet loss rate) conditions, and the video playback stall rate and video transmission bandwidth occupancy are tested by the video playback end. From the experimental results, in a common weak network environment (<50% packet loss rate, <60ms network delay), using the video frame rendering frequency adjustment method of the present application, the video can be played smoothly (<1% stall rate) while reducing the network bandwidth occupancy (<=1.7Mbps), effectively resisting the influence of weak network.

[0135] Table 2 Video playback stall rate and video transmission bandwidth occupancy under different network conditions

[0136]

[0137]

[0138] Compared with the prior art, the video frame rendering frequency adjustment method disclosed by the present application, when receiving the video frame sent by the sending end, obtains the basic parameters of the video frame and the network transmission parameters of the sending end when transmitting the video frame, and predicts the state parameter prediction value of the video frame received in the next time slice according to the basic parameters and the network transmission parameters, so as to adjust the rendering frequency of the video frame in the next time slice according to the state parameter prediction value of the video frame. Since the prediction is made according to the basic parameters and the network transmission parameters, reliable, robust, and real-time parameter adjustment basis can be provided for video frame playing, and after receiving each video frame, the rendering frame rate of the video frame is adaptively and dynamically adjusted according to the state parameter prediction value of the video frame, which can guarantee the real-time performance of the video frame rendering while improving the smoothness of the video playing in a weak network environment. In addition, the sending end can match the corresponding working mode according to the current network state, intelligently select the anti-weak network algorithm, fully utilize the network resources, and has stronger anti-weak network capability.

[0139] Reference Figure 3 , Figure 3A structural block diagram of a video frame rendering frequency adjustment device 100 is provided in an embodiment of the present application, and the video frame rendering frequency adjustment device 100 comprises:

[0140] A video frame receiving module 11 is configured to receive a video frame sent by a sending end;

[0141] A parameter obtaining module 12 is configured to obtain a basic parameter of the video frame and a network transmission parameter when the video frame is transmitted, wherein the network transmission parameter comprises a network time delay, a time delay jitter and a network packet loss rate;

[0142] A prediction module 13 is configured to predict a state parameter prediction value of a video frame received in a next time slice according to the basic parameter and the network transmission parameter;

[0143] A rendering frequency adjustment module 14 is configured to adjust a video frame rendering frequency in the next time slice according to the state parameter prediction value.

[0144] Specifically, the prediction module 13 is specifically configured to generate an input control amount at a current moment according to the basic parameter and the network transmission parameter, and input the input control amount into a video frame state parameter prediction model, so that the video frame state parameter prediction model outputs a state parameter prediction value of a video frame received in a next time slice.

[0145] Specifically, the video frame state parameter prediction model comprises a state equation and a state update equation, wherein,

[0146] an output of the state equation is the state parameter prediction value, and inputs of the state equation comprise the input control amount, a state parameter estimation value of the current time slice calculated through the state update equation and process noise;

[0147] an output of the state update equation is an updated state parameter estimation value of the current time slice, and inputs of the state update equation are the state parameter estimation value of the current time slice, a video frame state measurement value of the current time slice, a measurement matrix and a Kalman gain.

[0148] Specifically, the video frame state parameter prediction model further comprises a covariance equation of a predicted state and a covariance update equation; wherein,

[0149] an output of the covariance equation is a covariance matrix of the state parameter prediction value, and inputs of the covariance equation are a covariance matrix corresponding to the state parameter estimation value of the current time slice calculated through the covariance update equation and a covariance matrix corresponding to the process noise;

[0150] The output of the covariance update equation is a covariance matrix corresponding to the updated state parameter estimation value of the current time slice, and the input of the covariance update equation is a unit matrix and a covariance matrix corresponding to the state parameter estimation value of the current time slice.

[0151] Specifically, the state parameter prediction value includes a predicted video frame number and a predicted receiving time delay of the next time slice. Figure 4 The rendering frequency adjustment module 14 includes:

[0152] The calculation unit 141 is configured to calculate a target rendering frequency of the next time slice according to the predicted video frame number and the predicted receiving time delay.

[0153] The rendering frequency adjustment unit 142 is configured to adjust the video frame rendering frequency of the next time slice according to the target rendering frequency.

[0154] Specifically, the calculation unit 141 is specifically configured to: obtain a buffer cache parameter and a current rendering frequency; calculate a frequency adjustment value according to the buffer cache parameter, the predicted video frame number and the predicted receiving time delay; calculate a ratio of the frequency adjustment value and a preset smoothing factor; and take a sum of the current rendering frequency and the ratio as the target rendering frequency of the next time slice.

[0155] Specifically, the basic parameters include a real-time transmission number and a size of the video frame, and the network transmission parameters include a network time delay, a time delay jitter and a network packet loss rate.

[0156] Specifically, the sending end sends the video frame according to a preset weak network resistance mode; wherein, the weak network resistance mode is determined according to the network transmission parameters and bandwidth parameters.

[0157] Specifically, the weak network resistance mode includes an anti-packet loss mode, a low code rate mode and a low frame rate mode; wherein, the anti-packet loss mode is divided into different anti-packet loss strengths according to selective application of non-acknowledgement retransmission technology and forward error correction technology, the encoding code rate in the low code rate mode is calculated according to a historical maximum receiving code rate and an estimated code rate of the current period, and the packet sending frequency in the low frame rate mode is calculated according to a packet sending frequency of the last period.

[0158] It is worth noting that the working processes of the various modules in the video frame rendering frequency adjustment device 100 described in the embodiments of the present application can refer to the working processes of the video frame rendering frequency adjustment method described in the above embodiments, which will not be described here.

[0159] Compared with the prior art, the video frame rendering frequency adjustment device 100 disclosed by the application can obtain the basic parameters of the video frame and the network transmission parameters of the sending terminal when transmitting the video frame, and predict the state parameter prediction value of the video frame received in the next time slice according to the basic parameters and the network transmission parameters, so as to adjust the rendering frequency of the video frame in the next time slice according to the state parameter prediction value of the video frame. Since the prediction is performed according to the basic parameters and the network transmission parameters, reliable, robust and real-time parameter adjustment basis can be provided for video frame playing, and after each video frame is received, the rendering frame rate of the video frame is adaptively and dynamically adjusted according to the state parameter prediction value of the video frame, so that the real-time performance of video frame rendering can be ensured and the fluency of video playing can be improved in a weak network environment. In addition, the sending terminal can match the corresponding working mode according to the current network state, intelligently select the anti-weak network algorithm, and fully utilize the network resources, thereby having stronger anti-weak network capability.

[0160] Reference Figure 5 , Figure 5 is a structural block diagram of a video frame rendering frequency adjustment device 200 provided by an embodiment of the application. The video frame rendering frequency adjustment device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. The processor 21 implements the steps in each of the video frame rendering frequency adjustment method embodiments described above when executing the computer program, such as steps S1-S4.

[0161] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the video frame rendering frequency adjustment device 200.

[0162] The video frame rendering frequency adjustment device 200 can include, but is not limited to, the processor 21 and the memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the video frame rendering frequency adjustment device 200 and does not constitute a limitation on the video frame rendering frequency adjustment device 200, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the video frame rendering frequency adjustment device 200 can also include an input / output device, a network access device, a bus, etc.

[0163] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor 21 is a control center of the video frame rendering frequency adjustment device 200, and is connected with various parts of the video frame rendering frequency adjustment device 200 through various interfaces and lines.

[0164] The memory 22 can be used to store computer programs and / or modules. The processor 21 realizes various functions of the video frame rendering frequency adjustment device 200 by running or executing the computer programs and / or modules stored in the memory 22, and calling data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a nonvolatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0165] The modules / units integrated in the video frame rendering frequency adjustment device 200 can be stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. Based on this understanding, all or part of the processes in the above-mentioned embodiments can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0166] The above is the preferred embodiment of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A method for adjusting the rendering frequency of video frames, characterized in that, The method comprises the following steps: receiving a video frame sent by a sending end; obtaining basic parameters of the video frame and network transmission parameters during transmission of the video frame; wherein the network transmission parameters comprise network delay, delay jitter and network packet loss rate, and the basic parameters comprise real-time transmission quantity and size of the video frame; predicting state parameter prediction values of a video frame received in a next time slice according to the basic parameters and the network transmission parameters; adjusting a video frame rendering frequency in the next time slice according to the state parameter prediction values; wherein the state parameter prediction values comprise predicted video frame quantity and predicted receiving delay of the next time slice; and the adjusting of the video frame rendering frequency in the next time slice according to the state parameter prediction values comprises: obtaining buffer cache parameters and a current rendering frequency; calculating a frequency adjustment value according to the buffer cache parameters, the predicted video frame quantity and the predicted receiving delay; calculating a ratio of the frequency adjustment value and a preset smoothing factor; taking a sum of the current rendering frequency and the ratio as a target rendering frequency of the next time slice; adjusting the video frame rendering frequency in the next time slice according to the target rendering frequency.

2. The video frame rendering frequency adjustment method of claim 1, wherein, The sending end sends the video frame according to a preset anti-weak network mode; wherein the anti-weak network mode comprises an anti-packet loss mode, a low code rate mode and a low frame rate mode; the anti-packet loss mode is divided into different anti-packet loss strengths according to selective application of non-acknowledgement retransmission technology and forward error correction technology, the encoding code rate in the low code rate mode is calculated according to a historical maximum receiving code rate and an estimated code rate of a current period, and the packet sending frequency in the low frame rate mode is calculated according to a packet sending frequency of a previous period.

3. The video frame rendering frequency adjustment method of claim 1, wherein, The network transmission parameters comprise network delay, delay jitter and network packet loss rate.

4. The video frame rendering frequency adjustment method of claim 1, wherein, The predicting of the state parameter prediction values of the video frame received in the next time slice according to the basic parameters and the network transmission parameters comprises: generating an input control quantity at a current moment according to the basic parameters and the network transmission parameters; inputting the input control quantity into a video frame state parameter prediction model, so that the video frame state parameter prediction model outputs the state parameter prediction values of the video frame received in the next time slice.

5. The video frame rendering frequency adjustment method of claim 4, wherein, The video frame state parameter prediction model further comprises a covariance equation of a predicted state and a covariance update equation; wherein an output of the covariance equation is a covariance matrix of the state parameter prediction values, and inputs of the covariance equation are a covariance matrix corresponding to state parameter estimation values of a current time slice after calculation by the covariance update equation and a covariance matrix corresponding to process noise; an output of the covariance update equation is an updated covariance matrix corresponding to the state parameter estimation values of the current time slice, and inputs of the covariance update equation are a unit matrix and the covariance matrix corresponding to the state parameter estimation values of the current time slice.

6. An apparatus for adjusting a video frame rendering frequency, the apparatus comprising: a video frame rendering frequency adjuster configured to adjust a video frame rendering frequency based on a video frame rendering frequency adjustment value. The method comprises the following steps: a video frame receiving module is configured to receive a video frame sent by a sending end; The parameter obtaining module is configured to obtain basic parameters of the video frame and network transmission parameters in transmitting the video frame; wherein, the network transmission parameters include network time delay, time delay jitter and network packet loss rate, and the basic parameters include real-time transmission quantity and size of the video frame; The prediction module is configured to predict a state parameter prediction value of a video frame received in a next time slice according to the basic parameters and the network transmission parameters; The rendering frequency adjusting module is configured to adjust a video frame rendering frequency in the next time slice according to the state parameter prediction value; The state parameter prediction value includes predicted video frame quantity and predicted receiving time delay of the next time slice; and the rendering frequency adjusting module includes: The calculation unit is configured to obtain buffer cache parameters and a current rendering frequency; calculate a frequency adjustment value according to the buffer cache parameters, the predicted video frame quantity and the predicted receiving time delay; calculate a ratio of the frequency adjustment value and a preset smoothing factor; and take a sum of the current rendering frequency and the ratio as a target rendering frequency of the next time slice; The rendering frequency adjusting unit is configured to adjust a video frame rendering frequency in the next time slice according to the target rendering frequency.

7. A video frame rendering frequency adjustment device, characterized by, The computer readable storage medium includes a stored computer program, wherein, when the computer program runs, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the video frame rendering frequency adjusting method.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored computer program, wherein, when the computer program runs, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the video frame rendering frequency adjusting method.

Images