Video frame rendering frequency adjustment method and device, equipment and storage medium

Abstract

The invention discloses a video frame rendering frequency adjustment method and device, equipment and a storage medium, and the method comprises the steps: obtaining a basic parameter of a video frame and a network transmission parameter when the transmitting end transmits the video frame in a video frame rendering frequency adjustment process when the video frame transmitted by the transmitting end is received; and predicting a state parameter prediction value of the video frame received in the next time slice according to the basic parameter and the network transmission parameter, so that the rendering frequency of the video frame in the next time slice can be adjusted according to the state parameter prediction value of the video frame. As prediction is carried out according to the basic parameters and the network transmission parameters, a reliable, robust and real-time parameter adjustment basis can be provided for video frame playing, and after each video frame is received, the rendering frame rate of the video frame is adaptively and dynamically adjusted according to the video frame state parameter prediction value, so that the video frame rendering efficiency can be improved in a weak network environment. And the video playing fluency is improved while the video frame rendering real-time performance is ensured.

Description

Technical Field

[0001] The present invention relates to video processing technology, and in particular to a method, device, equipment and storage medium for adjusting video frame rendering frequency. Background Art

[0002] In 5G remote driving technology, although 5G networks can provide large bandwidth and low latency network capabilities, at the edge of the base station or when switching between base stations, network edge bandwidth will decrease, latency jitter and network packet loss will occur. Therefore, the environment at the edge of the base station or when switching between base stations is called a weak network environment. In a weak network environment, in order to ensure the smoothness and low latency of the video, the existing technology often uses NACK (Non-Acknowledge) retransmission technology and FEC (Forward Error Correction) technology to achieve the purpose of resisting weak networks, so as to improve the smoothness of video playback in a weak network environment and reduce the transmission delay of video transmission in a weak network environment.

[0003] However, in weak network scenarios, network packet loss and network delay coexist. When the network delay fluctuates, the frequency of receiving data packets also fluctuates. When the network status is poor, the reception of video data packets will be delayed, causing the client to experience screen freezes during video playback; when the network status improves, multiple data packets will be received in a concentrated manner, resulting in the accumulation of data packet buffers (used to temporarily store data), which increases video delay. Although single NACK retransmission and FEC technology can resist network packet loss and improve video freezes to a certain extent, they will increase data packet transmission delay (transmission delay refers to the total time required for a site to start sending a data packet and complete the transmission of the data packet), and even cause greater transmission delay fluctuations. The transmission delay increases, and the time from the sender sending a data packet to the receiver receiving the data packet increases, resulting in poor video fluency, and prone to screen freezes, delays, or rapid flashes, resulting in poor subjective experience. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, equipment and storage medium for adjusting the frequency of video frame rendering, which can reduce the transmission delay in a weak network environment, ensure the real-time performance of video frame rendering, and improve the smoothness of video playback.

[0005] To achieve the above object, an embodiment of the present invention provides a method for adjusting a video frame rendering frequency, comprising:

[0006] Receive video frames sent by the sender;

[0007] Obtaining basic parameters of the video frame and network transmission parameters when transmitting the video frame; wherein the network transmission parameters include network delay, delay jitter and network packet loss rate;

[0008] Predicting a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameter and the network transmission parameter;

[0009] The video frame rendering frequency in the next time slice is adjusted according to the predicted value of the state parameter.

[0010] As an improvement of the above solution, the state parameter prediction value includes the predicted number of video frames and the predicted receiving delay of the next time slice; then, adjusting the video frame rendering frequency in the next time slice according to the state parameter prediction value includes:

[0011] Calculate the target rendering frequency of the next time slice according to the predicted number of video frames and the predicted receiving delay;

[0012] The video frame rendering frequency in the next time slice is adjusted according to the target rendering frequency.

[0013] As an improvement of the above solution, the sending end sends the video frame according to a preset anti-weak network mode; wherein the anti-weak network mode is determined according to the network transmission parameters and bandwidth parameters.

[0014] As an improvement of the above-mentioned scheme, the anti-weak network mode includes an anti-packet loss mode, a low bit rate mode and a low frame rate mode; wherein, the anti-packet loss mode is divided into different anti-packet loss strengths according to the selective application of non-confirmation retransmission technology and forward error correction technology, the encoding bit rate in the low bit rate mode is calculated according to the historical maximum received bit rate and the estimated bit rate of the current period, and the packet sending frequency in the low frame rate mode is calculated according to the packet sending frequency of the previous period.

[0015] As an improvement of the above solution, the basic parameters include the real-time transmission quantity and size of the video frames, and the network transmission parameters include network delay, delay jitter and network packet loss rate.

[0016] As an improvement of the above solution, predicting a predicted value of a state parameter of a video frame received in the next time slice according to the basic parameter and the network transmission parameter includes:

[0017] Generate an input control amount at the current moment according to the basic parameters and the network transmission parameters;

[0018] The input control amount is input into a video frame state parameter prediction model so that the video frame state parameter prediction model outputs a predicted value of the state parameter of a video frame received in the next time slice.

[0019] As an improvement of the above solution, the video frame state parameter prediction model includes a state equation and a state update equation, wherein:

[0020] The output of the state equation is the predicted value of the state parameter, and the input of the state equation includes the input control quantity, the estimated value of the state parameter of the current time slice calculated by the state update equation, and the process noise;

[0021] The output of the state update equation is the updated state parameter estimate of the current time slice, and the input of the state update equation is the state parameter estimate of the current time slice, the video frame state measurement value of the current time slice, the measurement matrix and the Kalman gain.

[0022] As an improvement of the above solution, the video frame state parameter prediction model also includes a covariance equation and a covariance update equation of the predicted state; wherein,

[0023] The output of the covariance equation is the covariance matrix of the state parameter prediction value, and the input of the covariance equation is the covariance matrix corresponding to the state parameter estimation value of the current time slice calculated by the covariance update equation and the covariance matrix corresponding to the process noise;

[0024] The output of the covariance update equation is the covariance matrix corresponding to the updated state parameter estimate of the current time slice, and the input of the covariance update equation is the identity matrix and the covariance matrix corresponding to the state parameter estimate of the current time slice.

[0025] As an improvement to the above solution, the target rendering frequency of the next time slice is calculated according to the predicted number of video frames and the predicted receiving delay, including:

[0026] Get buffer cache parameters and current rendering frequency;

[0027] Calculating a frequency adjustment value according to the buffer cache parameter, the predicted number of video frames and the predicted receiving delay;

[0028] Calculating a ratio of the frequency adjustment value to a preset smoothing factor;

[0029] The sum of the current rendering frequency and the ratio is used as the target rendering frequency for the next time slice.

[0030] To achieve the above object, an embodiment of the present invention further provides a video frame rendering frequency adjustment device, comprising:

[0031] A video frame receiving module, used for receiving video frames sent by a sending end;

[0032] A parameter acquisition module, used to acquire basic parameters of the video frame and network transmission parameters when transmitting the video frame; wherein the network transmission parameters include network delay, delay jitter and network packet loss rate;

[0033] A prediction module, configured to predict a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameters and the network transmission parameters;

[0034] The rendering frequency adjustment module is used to adjust the video frame rendering frequency in the next time slice according to the predicted value of the state parameter.

[0035] To achieve the above-mentioned purpose, an embodiment of the present invention also provides a video frame rendering frequency adjustment device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the video frame rendering frequency adjustment method as described in any of the above embodiments.

[0036] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the video frame rendering frequency adjustment method as described in any of the above embodiments.

[0037] Compared with the prior art, the video frame rendering frequency adjustment method, device, equipment and storage medium disclosed in the present invention obtain the basic parameters of the video frame and the network transmission parameters of the transmitting end when transmitting the video frame when receiving the video frame sent by the transmitting end, and predict the state parameter prediction value of the video frame received in the next time slice according to the basic parameters and the network transmission parameters, so that the rendering frequency of the video frame in the next time slice can be adjusted according to the state parameter prediction value of the video frame. Since the prediction is made according to the basic parameters and the network transmission parameters, a reliable, robust and real-time parameter adjustment basis can be provided for the video frame playback, and after receiving each video frame, the rendering frame rate of the video frame can be adaptively and dynamically adjusted according to the predicted value of the video frame state parameter, which can ensure the real-time rendering of the video frame in a weak network environment while improving the smoothness of the video playback. In addition, the transmitting end can match the corresponding working mode according to the current network status, intelligently select the anti-weak network algorithm, make full use of network resources, and have a stronger anti-weak network capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a video processing flow chart of a transmitting end and a receiving end provided by an embodiment of the present invention;

[0039] Figure 2 is a flow chart of a method for adjusting video frame rendering frequency provided by an embodiment of the present invention;

[0040] Figure 3 It is a structural block diagram of a video frame rendering frequency adjustment device provided by an embodiment of the present invention;

[0041] Figure 4 A structural block diagram of a rendering frequency adjustment module in a video frame rendering frequency adjustment device provided by an embodiment of the present invention;

[0042] Figure 5 It is a structural block diagram of a video frame rendering frequency adjustment device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 , Figure 1 It is a video processing flow chart of a transmitting end and a receiving end provided by an embodiment of the present invention, wherein the transmitting end is used to collect video image data, and after image encoding, the data is sent to the receiving end, and the receiving end receives the data, and renders and plays after decoding the data. In order to solve the phenomenon of playback jamming, delay lag, and fast flash caused by transmitting video data in a weak network scenario, this application proposes a video rendering frequency adjustment method based on adaptive anti-weak network technology. An adaptive anti-weak network algorithm module is added to the transmitting end, and the network state is estimated according to the sending and receiving conditions of the video frame during image encoding, and the anti-weak network mode is adaptively selected. At the same time, a video state prediction module and a frame rate dynamic adjustment module are added to the receiving end. The video state prediction module predicts the state parameters of the video frame of the next time slice, and provides a basis for the frame rate dynamic adjustment module to dynamically adjust the video rendering frame rate. At the receiving end, the network transmission parameters and the basic parameters of the current video frame are used to predict the reception of the video frame of the next time slice, and the state parameter prediction value is generated. Then, the video frame rendering frame rate is adaptively adjusted according to the state parameter prediction value, which can ensure the real-time performance of the video frame rendering in a weak network environment while improving the smoothness of the video playback.

[0045] The transmitting end sends data packets sequentially, which are transmitted over the network and received by the receiving end. The transmitting end and the receiving end periodically send information exchange packets. According to the situation of receiving data packets and the situation of periodic information exchange between the transmitting end and the receiving end, the network state is estimated to obtain network transmission parameters and bandwidth parameters. The network transmission parameters include network delay, delay jitter and network packet loss rate, and the bandwidth parameter is the bit rate. The transmitting end sends the video frame to the receiving end according to the preset anti-weak network mode; wherein the anti-weak network mode is determined according to the network transmission parameters and bandwidth parameters. It is worth noting that the network transmission parameters and the bandwidth parameters are both estimated values, which are calculated according to the following calculation method.

[0046] Specifically, the network delay is the network RTT (Round-Trip Time) delay, which refers to the round-trip delay of a data packet in network transmission, that is, the time from when the sender sends data to when the sender receives confirmation from the receiver. A packet interaction process includes the sender A sending data to the receiver B, and the receiver B returning confirmation information to the sender A. The network RTT delay can be estimated at both ends A and B.

[0047] Specifically, the network delay estimation method includes: obtaining the current timestamp T of the information packet sent by the sender A2 , and obtain the last receiving time T of the last packet received by the sender from the receiver A1 Calculate the sending delay DT based on the current sending time and the last receiving time A1 ; Get the last sending time T of the receiving end sending the information packet in the last information packet received by the sending end B1 ; The sender sends the current timestamp T A2 , Send delay DT A1 and the last sending time T B1 Send it to the receiving end in the form of an information packet; obtain the current receiving time T when the receiving end receives the information packet B2 ; According to the last sending time T B1 , current receiving time T B2 And send delay DT A1 Calculate network latency.

[0048] For example, end A is used as the sending end and end B is used as the receiving end to record the current timestamp T of the information packet sent by end A. A2 , get the last receiving time T of the information packet sent by end B received by end A A1 , calculate the sending delay DT A1 , satisfying the following formula:

[0049] DT A1 =T A2 -T A1 (1).

[0050] If A has never received a packet from B, then DT A1 = 0. Furthermore, the A end also records the last time the A end received the B end information packet and the last time the B end sent the information packet T B1 , if the information packet from end B has never been received, then T B1 =0. A terminal will T A2 , T B1 , DTA1 Pack it into an information packet, send it, and end B receives the information packet.

[0051] Terminal B records the current receiving time T of the received information packet B2 , take out T from the information packet A2 , T B1 , DT A1 Information, calculate network delay RTT B , satisfying the following formula:

[0052] RTT B =T B2 -T B1 -DT A1 (2).

[0053] Among them, if T B1 =0 or DT A1 =0, the current RTT time is invalid.

[0054] It is worth noting that when end B is the receiving end and end A is the sending end, when end B sends a packet in the next cycle, it sends it in the above manner. When end A receives the packet from end B, it calculates the network delay RTT according to formula (2): A Both the receiving end and the sending end will periodically send information packets with timestamps. Through the above method, in addition to the first round of information exchange, each time either end receives an information packet, it can estimate the network delay once.

[0055] Specifically, delay jitter refers to the difference in the time interval between a pair of data packets at the receiving end and the sending end. The delay jitter can be obtained by estimating the statistical variance of the network transmission time. In order to simplify the calculation, the jitter is estimated using packet groups, that is, the time interval between received packets is within a threshold milliseconds (for example, 5 milliseconds) as a packet group. The delay jitter is estimated at the receiving end and transmitted to the sending end through the information packet. The delay jitter between the packet groups is calculated to meet the following calculation formula:

[0056] D i,j =(R j -R i )-(S j -S i ) (3);

[0057]

[0058] Among them, S i is the sending time of packet group i; S j is the sending time of packet group j; R i is the receiving time of packet group i; R j is the receiving time of packet group j; D i,jis the delay gradient of packet group i relative to packet group j; D i,j-1 is the delay gradient of packet group i relative to the previous packet group j; J i is the estimated network transmission jitter to packet group i at time; J i-1 is the estimated network transmission jitter up to the last packet group i.

[0059] Specifically, the network packet loss rate refers to the number of lost packets divided by the expected number of data packets in a certain period. The network packet loss rate is estimated at the receiving end and transmitted to the sending end through information packets. The calculation formula is as follows:

[0060] Exp i =sn i -sn0+1 (5);

[0061] Exp Δi =Exp i -Exp i-1 (6);

[0063] N Δi =N i -N i-1 (7);

[0064]

[0065] Among them, sn i is the maximum data packet sequence number at the current moment; sn0 is the sequence number of the first data packet; Exp i Exp is the total number of packets expected to be received by the end of cycle i; Δi N is the number of packets expected to be received from period i-1 to period i; i N is the total number of data packets actually received by the end of period i. In order to eliminate the impact of retransmitted packets, retransmitted packets are not included in the statistics of the number of received packets N. Δi is the number of data actually received from cycle i-1 to cycle i; LR i is the network packet loss rate at time i in period.

[0066] Specifically, the bit rate estimation method includes: the receiving end feeds back the delay gradient data of the calculated network delay to the sending end, the sending end caches the delay gradient data, the sending end takes a section of the cached data, and uses linear regression fitting to obtain the delay gradient trend T L , the delayed gradient trend T L and the preset first threshold T N1 , the second threshold T N2 Compare, T N1 <T N2, get the current network overload status, and calculate the target bit rate based on the overload status. The bit rate estimation is calculated at the sender and transmitted to the receiver through information packets, satisfying the following formula:

[0067]

[0068] Among them, Bt i is the estimated bit rate value at time i in cycle; R i is the maximum received bit rate in the past 500 milliseconds; α and β are multiplicative coefficients, α can take the empirical value of 0.85, and β can take the empirical value of 1.08. L >T N2 When T L <T N1 When T N1 ≤T L ≤T N2 , it indicates the network is normal.

[0069] Specifically, the transmitting end sends the video frame according to a preset anti-weak network mode; wherein the anti-weak network mode is determined according to the network delay, delay jitter, network packet loss rate and bit rate calculated by the above estimation method. The anti-weak network mode includes an anti-packet loss mode, a low bit rate mode and a low frame rate mode; wherein the anti-packet loss mode is divided into different anti-packet loss strengths according to the selective application of non-confirmation retransmission technology and forward error correction technology (i.e., turning on non-confirmation retransmission technology + turning on / off forward error correction technology), the encoding bit rate in the low bit rate mode is calculated according to the historical maximum receiving bit rate and the estimated bit rate of the current cycle, and the packet sending frequency in the low frame rate mode is calculated according to the packet sending frequency of the previous cycle.

[0070] Exemplarily, the anti-packet loss mode includes the following three modes:

[0071] ① Low-intensity packet loss resistance mode, in which the FEC algorithm function is turned off and only the NACK retransmission mechanism is turned on. In this mode, NACK retransmission is fully relied upon to resist data packet loss.

[0072] ② Medium-intensity anti-packet loss mode, turn on the FEC algorithm function, and use low-intensity FEC and NACK retransmission to jointly resist packet loss. In this mode, the low-intensity FEC algorithm resists part of the packet loss, reduces the packet loss rate, reduces the probability of NACK retransmission, and reduces the video transmission delay; the remaining part of the packet loss rate is handed over to the NACK retransmission algorithm to avoid too much bandwidth waste caused by FEC. Among them, the strength of the FEC algorithm is adjusted by adjusting the number of redundant packets in the FEC algorithm. Exemplarily, the strength of the FEC algorithm is determined by the algorithm parameters of the FEC, and the number of FEC redundant packets is adjusted to reduce the packet loss rate to a certain value T. L1 , for example T L1= 5%. The formula for adjusting the number of redundant packets is as follows:

[0073]

[0074] m = r·n (11);

[0075] P n,m <T L1 (12);

[0076] Where, n is the number of original packets; m is the number of redundant packets; r is the redundant packet ratio; p is the original packet loss rate; T L1 is the packet loss rate threshold; P n,m is the probability that data cannot be recovered after encoding m redundant packets based on n original packets, that is, the packet loss rate. That is, the "remaining packet loss rate" after the FEC algorithm. This part of the packet loss rate needs to be compensated by other means. Define a list R list of redundant packet ratios, traverse the list, calculate the remaining packet loss rate after the FEC algorithm according to formulas (10) - (12), and find a value r1 of the redundant packet ratio closest to T list from R L1 and calculate the number of redundant packets m therefrom. k is a variable, and its range is from m to n + m.

[0077] ③ High-intensity anti-packet loss mode, enable the FEC algorithm function, and use high-intensity FEC and NACK retransmission to jointly resist packet loss. In this mode, the FEC algorithm is mainly used to resist packet loss, maximizing the reduction of the probability of NACK retransmission and ensuring video latency. According to formulas (10) - (12), select a value r2 of the redundant packet ratio closest to T list from R L2 and calculate the number of redundant packets m therefrom to achieve the purpose of high anti-packet loss. T L2 needs to satisfy the following formula:

[0078] T L2 <T L1 (13).

[0079] Exemplarily, the low bitrate mode needs to reduce the video encoding bitrate, and the bitrate can be adjusted according to formula (9).

[0080] Exemplarily, the low frame rate mode requires the sender to reduce the packet sending frequency, and each adjustment is Y times that of the previous cycle, 0 < Y < 1, such as Y = 0.85 times.

[0081] Specifically, according to the network transmission parameters and the bandwidth parameters, different combinations of anti-weak network modes are adaptively selected to comprehensively optimize the transmission of data packets. The sending end needs to apply at least one of the three different strength modes in the anti-packet loss mode, and selectively apply the low bit rate mode and the low frame rate mode. The specific selection rules of the anti-networking mode can be referred to Table 1:

[0082] Table 1 Schematic diagram of weak network resistance mode combination selection at the transmitter

[0083]

[0084]

[0085] When the packet loss rate is less than the threshold T, only a small number of data packets need to be retransmitted. Under this condition, FEC does not need to be enabled, and only the NACK retransmission algorithm (low-intensity anti-packet loss mode) is used, which can resist packet loss without causing bandwidth waste.

[0086] When the packet loss rate is greater than the threshold T, the NACK retransmission algorithm alone will cause the same packet to be lost repeatedly, resulting in a linear increase in the transmission delay of the packet. Under this condition, the FEC function needs to be enabled to reduce the probability of packet retransmission. The strength of FEC is determined by the network RTT delay. If the RTT delay is high, we should try to reduce the probability of data packet retransmission, that is, increase the strength of FEC if bandwidth permits; if the RTT delay is low, we control the probability of data packet retransmission within a certain range and reduce the probability of repeated loss of the same packet. While controlling the transmission delay within an acceptable range, we can avoid excessive bandwidth waste caused by high-intensity FEC.

[0087] When the bandwidth estimate is less than the current encoding bit rate, the low bit rate mode is turned on to actively reduce the encoding bit rate, and work with other weak network resistance modes. It is worth noting that under this condition, the high-intensity FEC mode will increase the network bandwidth usage, deteriorate the network environment, cause network congestion, and cause greater packet loss and delay. Therefore, the high-intensity FEC algorithm should be avoided when the bandwidth is limited.

[0088] When the delay jitter is greater than the video acquisition cycle, the low frame rate mode is turned on to reduce the packet sending frequency and alleviate the phenomenon of data packet disorder. At the same time, it works with other anti-weak network modes to achieve better anti-weak network effect.

[0089] In the embodiment of the present invention, several weak network resistance modes are predefined, and the sending end can match the corresponding working mode according to the current network status, intelligently select the weak network resistance algorithm, and can fully utilize network resources and have stronger weak network resistance capability.

[0090] See also Figure 2 , Figure 2: is a flow chart of a method for adjusting the video frame rendering frequency provided by an embodiment of the present invention. The method for adjusting the video frame rendering frequency is implemented by a receiving end. The method for adjusting the video frame rendering frequency includes:

[0091] S1, receiving the video frame sent by the transmitter;

[0092] S2. Acquire basic parameters of the video frame and network transmission parameters when transmitting the video frame;

[0093] S3, predicting a predicted value of a state parameter of a video frame received in the next time slice according to the basic parameter and the network transmission parameter;

[0094] S4. Adjust the video frame rendering frequency in the next time slice according to the predicted value of the state parameter.

[0095] Specifically, in step S1, the transmitter collects images at a fixed frame rate, generates several video data packets through video encoding, and transmits them through the network to the receiver, who must reassemble and divide the multiple data packets into frames and send them to the decoder for decoding and playback. Different from the delay and jitter estimation of a single data packet, a video frame consists of multiple data packets. Modeling and predictive analysis of the transmission status of the complete video frame is more conducive to the decision-making of the player.

[0096] Specifically, in step S2, basic parameters of the video frame and network transmission parameters when transmitting the video frame are obtained; wherein the basic parameters include the real-time transmission quantity and size of the video frame, and the network transmission parameters include network delay, delay jitter and network packet loss rate.

[0097] Specifically, in step S3, the predicted values ​​of the state parameters of the video frame received at the next time slice are predicted based on the basic parameters and the network transmission parameters, including: generating an input control amount at the current moment based on the basic parameters and the network transmission parameters; inputting the input control amount into a video frame state parameter prediction model, so that the video frame state parameter prediction model outputs the predicted values ​​of the state parameters of the video frame received at the next time slice.

[0098] Exemplarily, let the transmission state of the video frame be Z = [N, D] T , N is the number of video frames, D is the delay of the video frame, the transmission status of the video frame is related to the camera acquisition frame rate F (the number of video frames I can be obtained), the video frame size S, the network delay RTT, the network jitter J, and the network packet loss rate LR. Let the input control quantity be U = [I, S, RTT, J, LR] T , input into the video frame state parameter prediction model, and the video frame state parameter prediction model is a Kalman prediction mathematical model.

[0099] Specifically, the video frame state parameter prediction model includes a state equation and a state update equation, wherein the output of the state equation is the state parameter prediction value, the input of the state equation includes the input control quantity, the state parameter estimation value of the current time slice calculated by the state update equation and the process noise, and the state equation can refer to formula (14); the output of the state update equation is the updated state parameter estimation value of the current time slice, and the input of the state update equation is the state parameter estimation value of the current time slice, the video frame state measurement value of the current time slice, the measurement matrix and the Kalman gain, and the state update equation can refer to formula (17).

[0100] Furthermore, the video frame state parameter prediction model also includes a covariance equation and a covariance update equation for the predicted state; wherein, the output of the covariance equation is the covariance matrix of the state parameter prediction value, and the input of the covariance equation is the covariance matrix corresponding to the state parameter estimate of the current time slice after calculation by the covariance update equation and the covariance matrix corresponding to the process noise, and the covariance equation may refer to formula (16); the output of the covariance update equation is the covariance matrix corresponding to the updated state parameter estimate of the current time slice, and the input of the covariance update equation is the unit matrix and the covariance matrix corresponding to the state parameter estimate of the current time slice, and the covariance update equation may refer to formula (19).

[0101] The video frame state parameter prediction model specifically includes the following equation:

[0102] (1) Equation of state:

[0103]

[0104] in, is the predicted value of the state parameter of the video frame at time t+1; is the estimated value of the state parameter of the video frame at time t after being updated by the following formula (17); A is the state transfer matrix, which is a 2x2-dimensional matrix; B is the control matrix, which is a 2x5-dimensional matrix; U t is the system input at time t, U t satisfy: I t is the number of complete video frames input between time slices t-1 and t, fixed to 1 (just one video encoding cycle), is the average video frame size of time slice t, The calculation method is as follows:

[0105]

[0106] Among them, S tis the size of a complete video frame composed of time slice t. If no complete video frame is successfully formed between time slices t-1 to t, the previous value S is taken. t-1 , and so on; φ is the sliding smoothness coefficient, which is taken as the empirical value of 0.9; J t is the network jitter of time slice t; LR t is the network packet loss rate of time slice t; W is the process noise, which obeys the zero-mean Gaussian distribution W~N(0,R), and R is the standard deviation.

[0107] (2) Covariance equation of the predicted state:

[0108]

[0109] in, is the predicted value of the video frame state parameter at time t+1 The covariance matrix of is the estimated value of the video frame state parameter at time t ; Q is the covariance matrix of W.

[0110] (3) State update equation:

[0111]

[0112] in, is the estimated value of the video frame state parameter of the current time slice before the state update, which is predicted by the previous time slice t-1; Z t is the state measurement value of the video frame at time t; H is the measurement matrix, which converts the state variables into predicted observation values ​​and is a 2x2 dimensional matrix; K g is the Kalman gain, which satisfies the following formula:

[0113]

[0114] (4) State covariance update equation:

[0115]

[0116] Where E is the identity matrix, is the predicted value of the video frame state at time t The covariance matrix of .

[0117] For example, the camera capture frame rate F is used as the time slice t, satisfying The network delay RTT of the current time slice t is calculated by formula (2): t , calculate the delay jitter value J of the current time slice t through formula (4) t , calculate the packet loss rate LR of time slice t by formula (8) t, calculate the average video frame size of time slice t by formula (15) The input control quantity U that constitutes the current time slice t t , which is input into the video frame state parameter prediction model. The actual number of video frames N received from the time when the previous video frame was received to the current time slice t t , and the delay D experienced in between t , is the actual video frame state Z of the current time slice t =[N t ,D t ] T , and the estimated value of the video state parameter of the current time slice The error between them is used to update the video frame state estimation value according to formulas (17), (18), and (19) and The input U of the current time slice t t and an estimate of the video frame state Together, according to formula (14), the video frame state parameter prediction value of the next time slice is predicted

[0118] Specifically, in step S4, the state parameter prediction value includes the predicted number of video frames and the predicted receiving delay of the next time slice; then, adjusting the video frame rendering frequency in the next time slice according to the state parameter prediction value includes:

[0119] S41, calculating a target rendering frequency for the next time slice according to the predicted number of video frames and the predicted receiving delay;

[0120] S42: Adjust the video frame rendering frequency in the next time slice according to the target rendering frequency.

[0121] Exemplarily, since the actual video frame state of the current time slice satisfies Z t =[N t ,D t ] T , so the video frame state parameter prediction value output by the video frame state parameter prediction model is Contains the number of predicted video frames N in the next time slice t+1 and the predicted receiving delay D t+1 According to the predicted number of video frames N t+1 and the predicted receiving delay D t+1Calculate the target rendering frequency for the next time slice, and after rendering (playing) the video frame according to the current rendering frequency, adjust the rendering frequency to the target rendering frequency, so that after receiving the video frame in the next time slice, directly render it according to the target rendering frequency. Therefore, except for the first time receiving the video frame using the default rendering frequency, each time a video frame is received thereafter, the target rendering frequency of the next received video frame can be predicted, thereby realizing dynamic adjustment of the rendering frequency.

[0122] Specifically, in step S41, the target rendering frequency of the next time slice is calculated according to the predicted number of video frames and the predicted receiving delay, including:

[0123] S411, obtaining buffer cache parameters and current rendering frequency;

[0124] S412, calculating a frequency adjustment value according to the buffer cache parameter, the predicted number of video frames and the predicted receiving delay;

[0125] S413, calculating a ratio of the frequency adjustment value to a preset smoothing factor; and taking the sum of the current rendering frequency and the ratio as a target rendering frequency for the next time slice.

[0126] Exemplarily, after obtaining the number N of predicted video frames that can be received next time t+1 and the predicted receiving delay D t+1 After that, the current buffer cache value M t , dynamically adjust the target rendering frequency F t+1 , the adjustment rules are as follows:

[0127] (20);

[0129]

[0130] F t+1 ≤F max (twenty two);

[0131] F t+1 ≥F min (twenty three);

[0132] Among them, M avg is the average value of the buffer cache in the previous 500 milliseconds; σ is the smoothing factor, that is, the minimum number of cycles required to offset these buffer fluctuations. The larger the value, the smoother the frame rate adjustment, but the slower the response. The empirical value is 3; F max The maximum rendering frame rate is limited, and the empirical value is 50fps; F min The minimum rendering frame rate is limited, and the empirical value can be 15fps.

[0133] In an embodiment of the present invention, at the video playback end, if the buffer is empty, no video frames can be rendered, and the screen freezes; when the buffer is accumulated, the playback screen has a large delay. This method can optimize the number of buffer caches by predicting the reception status of video frames in the future and dynamically adjusting the rendering cycle in advance, so as to maintain it at an appropriate value, reduce the probability of buffer emptying and buffer accumulation, ensure video delay, and improve video fluency.

[0134] Furthermore, Table 2 shows the experimental data performance of this proposal. The experimental object is a video source with a resolution of 1080p and a frame rate of 30 frames. The experimental scenario is real-time video transmission under different network (delay + packet loss rate) conditions, and the video playback stuttering rate and video transmission bandwidth occupancy obtained by the video player test. From the experimental results, it can be concluded that in a common weak network environment (<50% packet loss rate, <60ms network delay), the use of the video frame rendering frequency adjustment method of this application can ensure smooth video playback (<1% stuttering rate) while reducing network bandwidth occupancy (<=1.7Mbps), effectively resisting the impact of weak networks.

[0135] Table 2 Video playback freeze rate and video transmission bandwidth usage under different network conditions

[0136]

[0137]

[0138] Compared with the prior art, the video frame rendering frequency adjustment method disclosed in the present invention obtains the basic parameters of the video frame and the network transmission parameters of the transmitting end when transmitting the video frame when receiving the video frame sent by the transmitting end, and predicts the state parameter prediction value of the video frame received in the next time slice according to the basic parameters and the network transmission parameters, so that the rendering frequency of the video frame in the next time slice can be adjusted according to the state parameter prediction value of the video frame. Since the prediction is made according to the basic parameters and the network transmission parameters, a reliable, robust and real-time parameter adjustment basis can be provided for the video frame playback, and after receiving each video frame, the rendering frame rate of the video frame can be adaptively and dynamically adjusted according to the predicted value of the video frame state parameter, which can ensure the real-time rendering of the video frame in a weak network environment, while improving the fluency of the video playback. In addition, the transmitting end can match the corresponding working mode according to the current network state, intelligently select the anti-weak network algorithm, make full use of network resources, and have a stronger anti-weak network capability.

[0139] See also Figure 3 , Figure 31 is a structural block diagram of a video frame rendering frequency adjustment device 100 provided in an embodiment of the present invention, wherein the video frame rendering frequency adjustment device 100 comprises:

[0140] The video frame receiving module 11 is used to receive the video frame sent by the sending end;

[0141] A parameter acquisition module 12, used to acquire basic parameters of the video frame and network transmission parameters when transmitting the video frame; wherein the network transmission parameters include network delay, delay jitter and network packet loss rate;

[0142] A prediction module 13, configured to predict a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameters and the network transmission parameters;

[0143] The rendering frequency adjustment module 14 is used to adjust the video frame rendering frequency in the next time slice according to the predicted value of the state parameter.

[0144] Specifically, the prediction module 13 is specifically used to: generate an input control amount at the current moment according to the basic parameters and the network transmission parameters; input the input control amount into a video frame state parameter prediction model so that the video frame state parameter prediction model outputs a predicted value of the state parameter of the video frame received in the next time slice.

[0145] Specifically, the video frame state parameter prediction model includes a state equation and a state update equation, wherein:

[0146] The output of the state equation is the predicted value of the state parameter, and the input of the state equation includes the input control quantity, the estimated value of the state parameter of the current time slice calculated by the state update equation, and the process noise;

[0147] The output of the state update equation is the updated state parameter estimate of the current time slice, and the input of the state update equation is the state parameter estimate of the current time slice, the video frame state measurement value of the current time slice, the measurement matrix and the Kalman gain.

[0148] Specifically, the video frame state parameter prediction model also includes a covariance equation and a covariance update equation of the prediction state; wherein,

[0149] The output of the covariance equation is the covariance matrix of the state parameter prediction value, and the input of the covariance equation is the covariance matrix corresponding to the state parameter estimation value of the current time slice calculated by the covariance update equation and the covariance matrix corresponding to the process noise;

[0150] The output of the covariance update equation is the covariance matrix corresponding to the updated state parameter estimate of the current time slice, and the input of the covariance update equation is the identity matrix and the covariance matrix corresponding to the state parameter estimate of the current time slice.

[0151] Specifically, the state parameter prediction value includes the predicted number of video frames and the predicted receiving delay of the next time slice; then, see Figure 4 , the rendering frequency adjustment module 14 includes:

[0152] A calculation unit 141, configured to calculate a target rendering frequency of a next time slice according to the predicted number of video frames and the predicted receiving delay;

[0153] The rendering frequency adjustment unit 142 is configured to adjust the video frame rendering frequency in the next time slice according to the target rendering frequency.

[0154] Specifically, the calculation unit 141 is specifically used to: obtain buffer cache parameters and current rendering frequency; calculate a frequency adjustment value based on the buffer cache parameters, the predicted number of video frames and the predicted receiving delay; calculate the ratio of the frequency adjustment value and a preset smoothing factor; and use the sum of the current rendering frequency and the ratio as the target rendering frequency of the next time slice.

[0155] Specifically, the basic parameters include the real-time transmission quantity and size of the video frames, and the network transmission parameters include network delay, delay jitter and network packet loss rate.

[0156] Specifically, the transmitting end sends the video frame according to a preset weak network resistance mode; wherein the weak network resistance mode is determined according to the network transmission parameters and bandwidth parameters.

[0157] Specifically, the anti-weak network mode includes an anti-packet loss mode, a low bit rate mode and a low frame rate mode; wherein the anti-packet loss mode has different anti-packet loss strengths according to the selective application of non-confirmation retransmission technology and forward error correction technology, the encoding bit rate in the low bit rate mode is calculated based on the historical maximum received bit rate and the estimated bit rate of the current period, and the packet sending frequency in the low frame rate mode is calculated based on the packet sending frequency of the previous period.

[0158] It is worth noting that the working process of each module in the video frame rendering frequency adjustment device 100 described in the embodiment of the present invention can refer to the working process of the video frame rendering frequency adjustment method described in the above embodiment, and will not be repeated here.

[0159] Compared with the prior art, the video frame rendering frequency adjustment device 100 disclosed in the present invention obtains the basic parameters of the video frame and the network transmission parameters of the transmitting end when transmitting the video frame when receiving the video frame sent by the transmitting end, and predicts the state parameter prediction value of the video frame received in the next time slice according to the basic parameters and the network transmission parameters, so that the rendering frequency of the video frame in the next time slice can be adjusted according to the state parameter prediction value of the video frame. Since the prediction is made according to the basic parameters and the network transmission parameters, a reliable, robust and real-time parameter adjustment basis can be provided for the video frame playback, and after receiving each video frame, the rendering frame rate of the video frame can be adaptively and dynamically adjusted according to the predicted value of the video frame state parameter, which can ensure the real-time rendering of the video frame in a weak network environment while improving the smoothness of the video playback. In addition, the transmitting end can match the corresponding working mode according to the current network status, intelligently select the anti-weak network algorithm, make full use of network resources, and have a stronger anti-weak network capability.

[0160] See also Figure 5 , Figure 5 1 is a structural block diagram of a video frame rendering frequency adjustment device 200 provided in an embodiment of the present invention, and the video frame rendering frequency adjustment device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned video frame rendering frequency adjustment method embodiments are implemented, such as steps S1 to S4.

[0161] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the video frame rendering frequency adjustment device 200.

[0162] The video frame rendering frequency adjustment device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art may understand that the schematic diagram is merely an example of the video frame rendering frequency adjustment device 200 and does not constitute a limitation on the video frame rendering frequency adjustment device 200. The video frame rendering frequency adjustment device 200 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the video frame rendering frequency adjustment device 200 may also include input and output devices, network access devices, buses, etc.

[0163] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the video frame rendering frequency adjustment device 200, and uses various interfaces and lines to connect various parts of the entire video frame rendering frequency adjustment device 200.

[0164] The memory 22 can be used to store the computer program and / or module. The processor 21 implements various functions of the video frame rendering frequency adjustment device 200 by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0165] Wherein, if the module / unit integrated in the video frame rendering frequency adjustment device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 21. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0166] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for adjusting video frame rendering frequency, characterized in that: include: Receive video frames sent by the sender; Obtaining basic parameters of the video frame and network transmission parameters when transmitting the video frame; wherein the network transmission parameters include network delay, delay jitter and network packet loss rate; Predicting a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameter and the network transmission parameter; The video frame rendering frequency in the next time slice is adjusted according to the predicted value of the state parameter.

2. The video frame rendering frequency adjustment method according to claim 1, characterized in that: The state parameter prediction value includes the predicted number of video frames and the predicted receiving delay of the next time slice; then, adjusting the video frame rendering frequency in the next time slice according to the state parameter prediction value includes: Calculate the target rendering frequency of the next time slice according to the predicted number of video frames and the predicted receiving delay; The video frame rendering frequency in the next time slice is adjusted according to the target rendering frequency.

3. The video frame rendering frequency adjustment method according to claim 1, characterized in that: The transmitting end sends the video frame according to a preset weak network resistance mode; wherein the weak network resistance mode is determined according to the network transmission parameters and bandwidth parameters.

4. The video frame rendering frequency adjustment method according to claim 3, characterized in that: The anti-weak network mode includes an anti-packet loss mode, a low bit rate mode and a low frame rate mode; wherein the anti-packet loss mode has different anti-packet loss strengths according to the selective application of non-confirmation retransmission technology and forward error correction technology, the encoding bit rate in the low bit rate mode is calculated based on the historical maximum received bit rate and the estimated bit rate of the current period, and the packet sending frequency in the low frame rate mode is calculated based on the packet sending frequency of the previous period.

5. The video frame rendering frequency adjustment method according to claim 1, characterized in that: The basic parameters include the real-time transmission quantity and size of the video frames, and the network transmission parameters include network delay, delay jitter and network packet loss rate.

6. The video frame rendering frequency adjustment method according to claim 1, characterized in that: Predicting a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameter and the network transmission parameter includes: Generate an input control amount at the current moment according to the basic parameters and the network transmission parameters; The input control amount is input into a video frame state parameter prediction model so that the video frame state parameter prediction model outputs a predicted value of the state parameter of a video frame received in the next time slice.

7. The video frame rendering frequency adjustment method according to claim 6, characterized in that: The video frame state parameter prediction model includes a state equation and a state update equation, wherein: The output of the state equation is the predicted value of the state parameter, and the input of the state equation includes the input control variable, the estimated value of the state parameter of the current time slice calculated by the state update equation, and the process noise; The output of the state update equation is the updated state parameter estimate of the current time slice, and the input of the state update equation is the state parameter estimate of the current time slice, the video frame state measurement value of the current time slice, the measurement matrix and the Kalman gain.

8. The video frame rendering frequency adjustment method according to claim 7, characterized in that: The video frame state parameter prediction model also includes a covariance equation and a covariance update equation for the predicted state; wherein, The output of the covariance equation is the covariance matrix of the state parameter prediction value, and the input of the covariance equation is the covariance matrix corresponding to the state parameter estimation value of the current time slice calculated by the covariance update equation and the covariance matrix corresponding to the process noise; The output of the covariance update equation is the covariance matrix corresponding to the updated state parameter estimate of the current time slice, and the input of the covariance update equation is the identity matrix and the covariance matrix corresponding to the state parameter estimate of the current time slice.

9. The video frame rendering frequency adjustment method according to claim 2, characterized in that: Calculating a target rendering frequency for a next time slice according to the predicted number of video frames and the predicted receiving delay includes: Get buffer cache parameters and current rendering frequency; Calculating a frequency adjustment value according to the buffer cache parameter, the predicted number of video frames and the predicted receiving delay; Calculating a ratio of the frequency adjustment value to a preset smoothing factor; The sum of the current rendering frequency and the ratio is used as the target rendering frequency for the next time slice.

10. A video frame rendering frequency adjustment device, characterized in that: include: A video frame receiving module, used for receiving video frames sent by a sending end; A parameter acquisition module, used to acquire basic parameters of the video frame and network transmission parameters when transmitting the video frame; wherein the network transmission parameters include network delay, delay jitter and network packet loss rate; A prediction module, configured to predict a predicted value of a state parameter of a video frame received in a next time slice according to the basic parameters and the network transmission parameters; The rendering frequency adjustment module is used to adjust the video frame rendering frequency in the next time slice according to the predicted value of the state parameter.

11. A video frame rendering frequency adjustment device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for adjusting the video frame rendering frequency as described in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video frame rendering frequency adjustment method according to any one of claims 1 to 9.

Images