Video data processing method, device and equipment, and computer storage medium

Abstract

The present application discloses a video data processing method, device, equipment and computer storage medium, which are used to reduce the jamming of video playback and improve the smoothness of video playback. The method includes: a video receiving end receives a video stream sent by a video sending end, and stores each video frame of the video stream in a frame buffer; when the frame buffer has stored the complete data of the kth video frame, based on the sending time estimator state after the most recent update and the sending time measurement value of the kth video frame, determine the sending time estimator state corresponding to the kth video frame; based on the sending time estimator state corresponding to the kth video frame, determine the sending time smoothing value corresponding to the kth video frame; based on the sending time smoothing value corresponding to the kth video frame, determine the estimated rendering time of the kth video frame; based on the estimated rendering time, read the kth video frame from the frame buffer, and render the kth video frame.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of video processing technology, and provides a video data processing method, device and equipment, and a computer storage medium. Background Art

[0002] Generally speaking, current video data needs to be transmitted through the network, so the network conditions have a greater impact on the video experience, especially in real-time video scenarios such as video conferencing, screen sharing or live broadcast, which are more dependent on network conditions. Therefore, the impact of network conditions on video viewing effects is particularly prominent. When the network is normal, video data transmission can be maintained in a good state and the video viewing effect is better. However, in an environment where network transmission is unstable, video playback will have various problems, such as video freeze and jitter, which seriously affect the video viewing effect.

[0003] Therefore, how to reduce the probability of video playback or jitter problems in poor network conditions and thus improve the video viewing experience is an urgent problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a video data processing method, apparatus, device and computer storage medium for reducing the jamming of video playback and improving the smoothness of video playback.

[0005] In one aspect, a method for processing video data is provided, the method comprising:

[0006] The video receiving end receives the video stream sent by the video sending end, and stores each video frame of the video stream in a frame buffer;

[0007] When the frame buffer has stored complete data of the kth video frame, the video receiving end performs the following operations:

[0008] Determine a sending time estimator state corresponding to the kth video frame based on a sending time estimator state after the most recent update and a sending time measurement value of the kth video frame; wherein k is a positive integer not less than 1, and the sending time estimator state is used to characterize a sending frame rate of the video stream;

[0009] Determine a sending time smoothing value corresponding to the kth video frame based on a sending time estimator state corresponding to the kth video frame;

[0010] Determining an estimated rendering time of the kth video frame based on a sending time smoothing value corresponding to the kth video frame;

[0011] Based on the estimated rendering time, the kth video frame is read from the frame buffer, and rendering processing is performed on the kth video frame.

[0012] In one aspect, a video data processing device is provided, the device comprising:

[0013] A receiving unit, configured to receive a video stream sent by a video sending end, and store each video frame of the video stream in a frame buffer;

[0014] A sending time estimation unit, configured to determine, when the frame buffer has stored complete data of the k-th video frame, a sending time estimator state corresponding to the k-th video frame based on the sending time estimator state after the last update and the sending time measurement value of the k-th video frame; wherein k is a positive integer not less than 1; wherein the sending time estimator state is used to characterize the sending frame rate of the video stream; and, based on the sending time estimator state corresponding to the k-th video frame, determine a sending time smoothing value corresponding to the k-th video frame;

[0015] A rendering time estimation unit, configured to determine an estimated rendering time of the k-th video frame based on a sending time smoothing value corresponding to the k-th video frame;

[0016] A decoding and rendering unit is used to read the kth video frame from the frame buffer based on the estimated rendering time, and perform rendering processing on the kth video frame.

[0017] Optionally, the device further includes a network delay estimation unit, wherein the network delay estimation unit is configured to:

[0018] Determine the network delay estimator state corresponding to the kth video frame based on the most recently updated network delay estimator state and the measured value of the sending time and the measured value of the receiving time of the kth video frame; wherein the network delay estimator state is used to characterize the linear mapping relationship between the sending time and the receiving time of each video frame;

[0019] Determine a network delay estimation value corresponding to the kth video frame based on a network delay estimator state corresponding to the kth video frame;

[0020] The rendering time estimation unit is specifically used to determine the estimated rendering time based on the sending time smoothing value corresponding to the kth video frame and the network delay estimation value.

[0021] Optionally, the device further includes a network jitter estimation unit, wherein the network jitter estimation unit is configured to:

[0022] When the rendering of the k-1th video frame ends, updating the network jitter estimation parameter based on the sending time measurement value and the receiving time measurement value of the k-1th video frame;

[0023] Determine a network jitter estimation value corresponding to the k-1th video frame based on the updated network jitter estimation parameter;

[0024] The rendering time estimation unit is specifically used to determine the estimated rendering time based on the sending time smoothing value corresponding to the kth video frame, the network delay estimation value, and the network jitter estimation value corresponding to the k-1th video frame.

[0025] Optionally, the sending time estimation unit is specifically used to:

[0026] Determining a first a priori state corresponding to the kth video frame based on the most recently updated state of the sending time estimator;

[0027] Based on the first priori state corresponding to the kth video frame, determine a first weight value corresponding to the kth video frame; wherein the larger the first weight value is, the higher the confidence of the transmission time measurement value is, and the smaller the first weight value is, the higher the confidence of the transmission time smoothing value determined based on the first priori state is;

[0028] Based on the first weight value and the measured value of the sending time of the k-th video frame, a first priori state corresponding to the k-th video frame is updated to obtain a sending time estimator state corresponding to the k-th video frame.

[0029] Optionally, the sending time estimation unit is specifically used to:

[0030] Based on the value of the transmission time estimator state after the most recent update, obtaining the value of the first priori state corresponding to the k-th video frame; and,

[0031] Based on the probability distribution of the value of the sending time estimator state after the most recent update, the probability distribution of the value of the first priori state corresponding to the k-th video frame is determined.

[0032] Optionally, the sending time estimation unit is specifically used to:

[0033] Determine a first priori state corresponding to the k-th video frame to obtain an estimated value of a sending time corresponding to the k-th video frame, and determine a difference between the estimated value of the sending time corresponding to the k-th video frame and a measured value of the sending time corresponding to the k-th video frame;

[0034] Obtaining a value of a transmission time estimator state corresponding to the kth video frame based on a value of a first priori state corresponding to the kth video frame, and performing a weighted summation on the difference value and the first weight value;

[0035] Based on the probability distribution of the value of the first priori state corresponding to the kth video frame and the first weight value, the probability distribution of the value of the sending time estimator state corresponding to the kth video frame is determined.

[0036] Optionally, the sending time estimation unit is specifically used to:

[0037] Determine a target sending time corresponding to the kth video frame based on a sending frame rate represented by a sending time estimator state corresponding to the kth video frame and a frame number of the kth video frame;

[0038] The target sending time is determined as a sending time smoothing value of the kth video frame.

[0039] Optionally, the device further includes an abnormality detection unit, configured to:

[0040] Determine whether the set of sending time errors corresponding to the video stream meets the zero mean condition; wherein each sending time error in the sending time error set is the difference between the sending time smoothing value and the sending time measurement value of a video frame;

[0041] When it is determined that the sending time error set does not satisfy the zero mean condition, the sending time estimator state corresponding to the k-th video frame is reset.

[0042] Optionally, the network delay estimation unit is specifically used to:

[0043] Determine a second priori state corresponding to the k-th video frame based on the most recently updated network delay estimator state;

[0044] Based on the second priori state corresponding to the kth video frame, determine a second weight value corresponding to the kth video frame; wherein the larger the second weight value, the higher the confidence of the network delay measurement value, and the smaller the second weight value, the higher the confidence of the network delay estimation value obtained based on the second priori state corresponding to the kth video frame;

[0045] Based on the second weight value and the sending time measurement value and the receiving time measurement value of the kth video frame, the second priori state corresponding to the kth video frame is updated to obtain the network delay estimator state corresponding to the kth video frame.

[0046] Optionally, the network delay estimation unit is specifically used to:

[0047] Based on the value of the network delay estimator state after the most recent update and random noise, obtaining the value of the second priori state corresponding to the k-th video frame; and,

[0048] Based on the probability distribution of the value of the network delay estimator state after the most recent update, the probability distribution of the value of the second prior state corresponding to the k-th video frame is determined.

[0049] Optionally, the anomaly detection unit is further used to:

[0050] Determine whether the network delay error set corresponding to the video stream satisfies a zero mean condition; wherein each network delay error in the network delay error set is a difference between a network delay estimation value and a network delay measurement value of a video frame;

[0051] When it is determined that the network delay error set does not satisfy the zero mean condition, the probability distribution of the value of the network delay estimator state corresponding to the k-th video frame is reset.

[0052] Optionally, the sending time estimation unit is specifically used to:

[0053] When the frame buffer has stored complete data of the k+nth video frame, the video receiving end determines the sending time estimator state corresponding to the k+nth video frame based on the sending time estimator state after the most recent update and the sending time measurement value of the k+nth video frame; wherein n is a positive integer not less than 1;

[0054] Using the sending time estimator state corresponding to the k+n-th video frame as the sending time estimator state corresponding to the k-th video frame;

[0055] When the decoding of the k-1th video frame is finished, a sending time smoothing value of the kth video frame is determined based on a sending time estimator state corresponding to the kth video frame.

[0056] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0057] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0058] In one aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above methods.

[0059] In the embodiment of the present application, the state of the sending time estimator corresponding to the current video frame is updated by using the previous state of the sending time estimator and the measured value of the sending time of the currently processed video frame, and then the sending time smoothing value of the current video frame can be determined based on the updated sending time estimator state, and then the estimated rendering time of the current video frame can be determined, so as to take the frame for rendering processing based on the estimated rendering time. It can be seen that the embodiment of the present application takes into account that the sending time in actual applications is not strictly equidistant, but has certain fluctuations. Therefore, the embodiment of the present application iteratively updates the state of the sending time estimator, and realizes the smoothing of the sending time through the updated sending time estimator state, and uses the smoothed sending time to estimate the rendering time, so as to reduce the problem of uneven rendering time caused by sending time jitter, thereby improving the smoothness of the rendering time of the video frame, and correspondingly improving the video viewing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0061] Figure 1 A schematic diagram of the sending time smoothing process provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of the processing flow of the real-time video solution provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a scenario provided for an embodiment of the present application;

[0064] Figure 4 A schematic diagram of the structure of a video receiving module 10 provided in an embodiment of the present application;

[0065] Figure 5 A flowchart of a multimedia information recommendation method provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of different states of a sending time estimator provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of an update process of a sending time estimator provided in an embodiment of the present application;

[0068] Figure 8 A schematic flow chart of a video data processing method combined with a network delay estimator provided in an embodiment of the present application;

[0069] Fig. 9 A state diagram of a network delay estimator provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of an update process of a network delay estimator provided in an embodiment of the present application;

[0071] Fig.11 A schematic flow chart of a video data processing method combined with a network jitter estimator provided in an embodiment of the present application;

[0072] Fig.12 A schematic diagram of estimated rendering time provided by an embodiment of the present application;

[0073] Fig.13 A schematic diagram of the structure of a video data processing device provided in an embodiment of the present application;

[0074] Fig.14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0076] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0077] Video stream: Video stream refers to video data transmitted using streaming transmission technology. Because it is based on streaming transmission technology, the video receiving end (or video playing end) can display video data before the entire video file is transmitted. It is the basic technology for real-time video solutions. Real-time video solutions can be, for example, video calls, video conferencing, or live broadcasts, etc., which require real-time transmission of video data.

[0078] Frame Buffer: Also known as frame buffer memory, the frame buffer stores undecoded video data packets and distinguishes them according to video frames to manage video data packets.

[0079] Framing: When a video frame is transmitted, the video transmitter usually divides it into multiple data packets for transmission, and the video receiver will receive multiple data packets accordingly. The process of integrating the data included in the data packet of each video frame is called framing.

[0080] Jitter buffer: Jitter refers to the change in data packet transmission time caused by network congestion, timing drift, or route changes. Jitter buffer can be used to alleviate problems such as video freeze caused by jitter. The core idea of ​​jitter buffer is to increase the delay in exchange for the smoothness of video calls. When the network is unstable (jitter occurs), increase the length of the buffer and cache more data to cope with possible jitter in the future. When the network stabilizes, reduce the length of the buffer and cache less data, reduce the end-to-end delay of the video, and improve real-time performance. Therefore, the operation of jitter buffer is a process of dynamically adjusting the buffer length according to jitter.

[0081] Sending time: refers to the time when the video transmitter sends the data packet of the video frame. The sending time involved in the embodiments of the present application includes a sending time measurement value and a sending time smoothing value. Among them, the sending time measurement value refers to the time when the video transmitter actually sends the data packet, that is, the sending timestamp carried in the data packet. The sending time smoothing value is the sending time estimated by the sending time estimator, but because the sending time estimator is to obtain the optimal linear change relationship of the sending time of each video frame through continuous iterative updates, that is to say, the final estimated sending time tends to a certain linear relationship, then for each video frame, the sending time is a smooth transition, and therefore, the sending time estimator is equivalent to smoothing the sending time measurement value to obtain the sending time smoothing value.

[0082] Send time estimator status: refers to the status of the send time estimator. A status of the send time estimator can reflect the send frame rate of the video stream. Figure 1 As shown in FIG. 1 , the encoding speed difference between the i-frame and the p-frame at the video transmitter and the system jitter at the video transmitter make the actual transmission time interval of the video frames not completely equal, such as Figure 1 As shown by the line La of the transmission time measurement value in the figure, the variation relationship of the transmission time is not smooth, and the purpose of the transmission time estimator is to fit a smooth linear variation relationship of the transmission time, that is, to obtain Figure 1 The connecting line La' of the smoothed sending time value is shown, so that the rendering time finally obtained based on the smoothed sending time can be smoothed.

[0083] Rendering time: Rendering time refers to the time it takes to render a video frame for display. Ideally, the rendering time should change linearly, which means that the rendering time should be smooth, and the difference between any two frames should be equal. In this way, there will be no freezes or playback problems when the video is played. However, in reality, due to factors such as network transmission and device processing power, the actual rendering time will not change strictly linearly. However, the goal of the video player is to use a certain anti-jitter solution to make the rendering time change linearly and the video playback transition smoothly.

[0084] Network delay: The time difference between when the video transmitter sends the first data packet of the video frame and when the video receiver receives the last data packet of the video frame is the network delay.

[0085] Network jitter: Due to the instability of network transmission conditions, the network delay of different video frames may also be different. The difference between network delays is jitter. For example, the transmission time of the first video frame is T a1 , the receiving time is T b1 , the sending time of the second video frame is T a2 , the receiving time is T b2 , then the network jitter is:

[0086] U=(T b2 -T a2 )-(T b1 -T a1 )

[0087] The jitter buffer takes network jitter into consideration and adaptively adjusts the buffer length based on the size of the network jitter to improve the smooth playback of the video.

[0088] Zero mean: For a data set, zero mean means that the difference between each data in the set and the mean of the set tends to zero. In other words, zero mean can reflect the distribution of a data set, that is, there will be no sudden changes in the data set, so it can be used to assist in judging whether the data is abnormal. For example, when a data set suddenly does not meet the zero mean, then the newly added data must have mutations or anomalies.

[0089] Currently, real-time video such as live video and video calls are becoming increasingly popular, but real-time video is greatly affected by network transmission conditions. Figure 2 As shown, it is a schematic diagram of the processing flow of the real-time video solution, wherein, after the video transmitter sends the video data packet, it is transmitted through the network, and the video receiver receives the video data packet. Since the data packets of each video frame may be out of order, and in order to ensure the data integrity of each video frame, a framing operation is required at the video receiver. The jitterbuffer is located between framing and decoding, and is used to cache video frames and smoothly output video frames to the decoder. After the decoder decodes the video frame, it is output to the renderer for rendering processing to display the video frame.

[0090] In relevant real-time video solutions, Jitter Buffer is an important module in the real-time video architecture. Jitter Buffer can be used to suppress video problems caused by network instability. For example, it can handle packet loss, disorder, and delayed arrival, smoothly output video frames to the decoding module, and resist the jamming problem caused by a weak network environment.

[0091] However, in current real-time video solutions, when estimating the rendering time, the frames that arrive in sequence are first used to calculate the mapping relationship from the sending time to the receiving time, thereby calculating the smooth receiving time, and using the retransmission module to calculate the average retransmission time, and using the network jitter estimator to calculate the average network jitter, and then the smooth receiving time, the average retransmission time and the average network jitter are added together to obtain the estimated rendering time. However, related real-time video solutions do not take into account the problem of unstable sending time intervals, resulting in fluctuations in the final calculated rendering time, which results in poor video playback. In addition, in related real-time video solutions, there is a coupling between network delay estimation and retransmission time estimation, that is, the network delay needs to be estimated based on the average retransmission time output by the retransmission module, which increases the error.

[0092] In view of this, an embodiment of the present application provides a video data processing method, in which the state of the sending time estimator corresponding to the current video frame is updated by using the previous state of the sending time estimator and the sending time measurement value of the currently processed video frame, and then the sending time smoothing value of the current video frame can be determined based on the updated sending time estimator state, and then the estimated rendering time of the current video frame can be determined, so as to take the frame for rendering processing based on the estimated rendering time. It can be seen that the embodiment of the present application takes into account that the sending time in actual applications is not strictly equidistant, but has certain fluctuations. Therefore, the embodiment of the present application iteratively updates the state of the sending time estimator, and realizes the smoothing of the sending time through the updated sending time estimator state, and uses the smoothed sending time to estimate the rendering time, so as to reduce the problem of uneven rendering time caused by sending time jitter, thereby improving the smoothness of the rendering time of the video frame, and correspondingly improving the video viewing effect.

[0093] In addition, in the embodiment of the present application, when estimating the network delay, the network delay estimator state of the current frame is updated by the network delay measurement value and the network delay estimator state obtained by the last update to calculate the network delay of the current frame, so that there is no need to calculate the retransmission time, and the network delay change is estimated quickly and accurately. In addition, the embodiment of the present application can achieve high-sensitivity anomaly detection, and adopt a lightweight reset operation to correct the abnormal parameter state of the transmission time and network delay, while ensuring that the estimation of each parameter does not fluctuate too much.

[0094] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0095] The solution provided in the embodiment of the present application can be applied to most video scenarios, especially to real-time video scenarios, such as video calls, live video broadcasts, and video conferences. Figure 3 As shown, it is a schematic diagram of a scenario provided by an embodiment of the present application, in which a plurality of terminal devices 101 and a server 102 may be included, wherein: Figure 3 Although only a limited number of terminal devices 101 are shown, namely Figure 3 The terminal devices 101 of users A to D are shown, but it should be noted that the number of terminal devices 101 can be expanded to more, and the embodiment of the present application does not limit this.

[0096] The terminal device 101 may be, for example, a mobile phone, a tablet computer (PAD), a personal computer (PC), a smart TV, a smart vehicle device, a wearable device, etc. The terminal device 101 may be installed with a video application, which may be, for example, a real-time video application such as a video conferencing application, a live video application, or an instant messaging application. Of course, the application may also be any other application capable of playing videos online.

[0097] Each terminal device 101 may include one or more processors 1011, a memory 1012, and an I / O interface 1013 for interacting with the terminal. The memory 1012 of the server 101 may also store program instructions of the video data processing method provided in the embodiment of the present application, and these program instructions, when executed by the processor 1011, can be used to implement the steps of the video data processing method provided in the embodiment of the present application to play the corresponding video on the terminal device.

[0098] Server 102 can be a background server corresponding to the video application installed on the terminal device 101. For example, it can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but is not limited to these.

[0099] Taking the video conference in the real-time video application as an example, when conducting a video conference, any terminal device 101 can be used as a video transmitter or a video receiver. Specifically, user A can create a video conference through the video conference application installed on its terminal device 101. Correspondingly, the server 102 can respond to the operation of user A and create a new video conference. Other users can also request to join the video conference from the server 102. Similarly, the server 102 will process the joining request of each user. For example, if user B joins the video conference, the server will create a transmission channel between the terminal devices 101 of user A and user B, so that the terminal devices 101 of user A and user B can transmit their respective video data to each other. For example, after the terminal device 101 of user A collects the video data of user A, it transmits the video stream carrying the video data to user B. Correspondingly, after the terminal device 101 of user B receives the video stream, it can use the video data processing method provided in the embodiment of the present application to process the video frame and display it. Similarly, other real-time video scenes are similar, so they will not be described in detail.

[0100] Taking online video playback as an example, after user A selects the video to be played in the video application on the terminal device 101, he will request the corresponding video data from the server 102. The server 102 returns the video stream to the terminal device 101 of user A. After the terminal device 101 of user A receives the video stream, it can use the video data processing method provided in the embodiment of the present application to process the video frame and then display it.

[0101] The terminal device 101 and the server 102, as well as each terminal device 101, can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network, for example, a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course other possible networks, which are not limited in the embodiments of the present application.

[0102] Of course, the method provided in the embodiment of the present application is not limited to Figure 3 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 3 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0103] The method of the embodiment of the present application mainly involves the video reception related content in the terminal device 101, see Figure 4 , which is a schematic diagram of the structure of the video receiving module 10 provided in an embodiment of the present application. The video receiving module 10 may include a packet buffer 401 , a jitter buffer 402 and a decoder 403 .

[0104] (1) Packet buffer 401

[0105] The video data packets sent by the video sending end arrive at the video receiving end after being transmitted through the network. The packet buffer 401 is used to store the received video data packets and plays a buffering role between the receiving thread and the framing thread.

[0106] (2) jitter buffer 402

[0107] The jitter buffer 402 may further include a control module 4021, a transmission time estimator 4022, a network delay estimator 4023, and a network jitter estimator 4024. The jitter buffer 402 is used to suppress video problems caused by network instability. For example, it can handle packet loss, disorder, and delayed arrival, smoothly output video frames to the decoding module, and resist the jamming problem caused by a weak network environment.

[0108] The control module 4021 plays the role of overall control, controlling the update timing and action timing of each estimator. Figure 4 (not shown) takes a packet from packet buffer 401, parses the frame number, and stores the parsed frame data in a frame buffer (frame buffer) ( Figure 4 (not shown), the control module 4021 updates the sending time estimator 4022 and the network delay estimator 4023 once when the frame buffer has cached the complete data of a video frame, and when the frame buffer has cached the complete data of a video frame and the frame is the next frame of the video frame that has just been decoded, the sending time estimator 4022 and the network delay estimator 4023 that have been updated most recently are used to calculate the sending time smoothing value and the network delay estimation value respectively, and the rendering time is estimated using the obtained sending time smoothing value, the network delay estimation value and the network jitter estimated for the previous frame.

[0109] (3) Decoder 403

[0110] When the rendering time of the frame is about to arrive, the decoder 403 takes the frame data from the frame buffer for decoding and outputs it to the renderer for rendering and display. Among them, the control module 4021 controls the update of the network jitter estimator 4024 after rendering, and uses the updated network jitter estimator 4024 to estimate the current network jitter for the calculation of the next rendering time.

[0111] The method steps executed by the above-mentioned functional modules will be specifically introduced in the subsequent embodiments, and will not be described in detail here.

[0112] In a possible application scenario, various video-related data in the embodiments of the present application can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (or storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0113] In a possible application scenario, in order to reduce communication latency, servers 102 may be deployed in various regions, or different servers 102 may serve the regions corresponding to each terminal device 101 for load balancing. Multiple servers 102 share data through blockchain, and multiple servers 102 are equivalent to a data sharing system composed of multiple servers 102. For example, if the terminal device 101 is located at location a, and its corresponding server 102 is the first server 102, then the first server 102 provides recommendation services for the terminal device 101 at location a; if the terminal device 101 is located at location b, and its corresponding server 102 is the second server 102, then the second server 102 provides recommendation services for the terminal device 101 at location b.

[0114] Each server 102 in the data sharing system has a node identifier corresponding to the server 102. Each server 102 in the data sharing system can store the node identifiers of other servers 102 in the data sharing system, so that the generated blocks can be broadcast to other servers 102 in the data sharing system according to the node identifiers of other servers 102. Each server 102 can maintain a node identifier list as shown in the following table, and store the server 102 name and the node identifier in the node identifier list. Among them, the node identifier can be the Internet Protocol (IP) address of the interconnection between networks and any other information that can be used to identify the node. Table 1 only takes the IP address as an example for explanation.

[0115] Server Name Node ID Node 1 119.115.151.174 Node 2 118.116.189.145 … … Node N 119.124.789.258

[0116] Table 1

[0117] The following is an introduction to the method flow provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0118] See also Figure 5 , is a flow chart of a video data processing method provided in an embodiment of the present application, which can be performed by Figure 3 The method is executed by the terminal device 101, and the process of the method is described as follows.

[0119] Step 501: The video receiving end receives the video stream sent by the video sending end, and stores each video frame of the video stream into a frame buffer.

[0120] In an embodiment of the present application, the video sending end packages the video data to be transmitted, and transmits the video data to the video receiving end in the form of a video stream. Correspondingly, after receiving the video stream, the video receiving end stores the received video data packets in a packet buffer, and takes out the data packets from the packet buffer for framing operations, that is, parsing the frame number, and storing the parsed frame data in the frame buffer.

[0121] In actual applications, for real-time video scenarios, the video sender is the terminal device of each party participating in the real-time video. For example, when user A and user B have a video call, the terminal devices of user A and user B can both serve as video senders. When user A's terminal device serves as the video sender, the video sender collects user A's video data through a camera and sends it to user B. Correspondingly, user B's terminal device serves as a video receiver and receives the video data packets transmitted by user A's terminal device.

[0122] For the scenario of online video playback, the video sender can be the background server corresponding to the video application, or the content delivery network (CDN) server that stores video data. When the user triggers the video playback operation in the video application, the background server is requested to obtain the video stream of the video. Correspondingly, the video receiver receives the video stream data returned by the background server or CDN server.

[0123] After the video receiving end receives the video data packet, it stores the video data packet in the packet buffer area, and then the framing module reads the data packet from the packet buffer area, parses the frame number, integrates the data of the same video frame, and stores it in the frame buffer area, that is, the frame buffer area can cache the data of one or more video frames.

[0124] Step 502: When the frame buffer has stored complete data of the kth video frame, the video receiving end determines the sending time estimator state corresponding to the kth video frame based on the sending time estimator state after the most recent update and the sending time measurement value of the kth video frame, where k is a positive integer not less than 1.

[0125] In the embodiment of the present application, the sending time can be estimated by a sending time estimator, and the state of the sending time estimator, that is, the parameter of the sending time estimator, is used to characterize the sending frame rate of the video stream.

[0126] Among them, the embodiment of the present application can be combined with the Kalman filtering method to realize the sending time estimation task, wherein the Kalman filtering method is an algorithm that uses the linear system state equation to optimally estimate the system state through the system input and output observation data. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process. Data filtering is a data processing technology that removes noise and restores real data. Kalman filtering can estimate the state of a dynamic system from a series of data with measurement noise when the measurement variance is known.

[0127] Specifically, the state equation and observation equation of the sending time estimator can be expressed as follows:

[0128] θ k+1 =θ k

[0129] s k =[i k ,1]·θ k +η k

[0130] The state of the sending time estimator is a process of continuous iterative update, θ k+1 represents the state of the estimator after the k+1th update, that is, the state of the sending time estimator after the k+1th update, θ k Represents the state of the estimator after the last update, that is, the state of the estimator after the kth update, θ k is a vector whose two components are denoted by θ k [0] and θ k [1], since θ k That is, the state of the sending time estimator, and the sending time estimator state is used to characterize the sending frame rate of the video stream, then θ k In fact, on the coordinate axis (such as Figure 1 (shown) is a straight line showing the change of the sending time with the frame number, θ k is the slope of the straight line, which can reflect the frame rate of the video stream. k [1] is the intercept of the line; s k Represents the sending time measurement value, that is, the sending timestamp carried in the video data packet; i k represents the frame number; η k represents the observation noise. In practical applications, η k Gaussian noise may be used.

[0131] It should be noted that, in the embodiment of the present application, the update of the transmission time estimator is performed when each video frame data is complete, that is, the k+1th update is performed after the k+1th video frame is completely received, so θk+1 It can also be understood as the estimator state corresponding to the k+1th video frame.

[0132] Specifically, when the frame buffer has stored the complete data of the kth video frame, the control module in the jitter buffer controls the update of the transmission time estimator. Referring to the above state equation, when updating, the transmission time estimator state corresponding to the kth video frame is determined based on the transmission time estimator state after the last update and the transmission time measurement value of the kth video frame. In an ideal case, that is, the transmission time of each video frame is smooth, then the transmission time estimator state after the last update should be the same as the current time smoothing parameter.

[0133] Step 503: Based on the state of the sending time estimator corresponding to the k-th video frame, determine the sending time smoothing value corresponding to the k-th video frame.

[0134] In an embodiment of the present application, after obtaining the state of the sending time estimator corresponding to the kth video frame, that is, after knowing the current state of the sending time estimator, the sending time estimator state can be used to determine the sending time smoothing value corresponding to the kth video frame.

[0135] See also Figure 6 As shown in the figure, it is a schematic diagram of different states of the sending time estimator. Among them, La1' and La2' are the estimator states after two consecutive updates, for example, La1' is the estimator state after the last update, and La2' is the estimator state corresponding to the kth video frame. It can be seen that when k-1 video frames are received completely and when k video frames are received completely, the smoothed sending time change relationship is different, and the sending time estimator makes the state of the sending time estimator converge through the process of continuous iterative update, so that the obtained estimator state can perform more optimized smoothing processing on the sending time.

[0136] Specifically, after obtaining the sending time estimator state corresponding to the kth video frame, the target sending time corresponding to the kth video frame can be determined based on the sending frame rate represented by it and the frame number of the kth video frame, and the target sending time is determined as the sending time smoothing value of the kth video frame. Figure 6 For example, when k is 12, the time point corresponding to the 12th frame can be determined from La2' as 900, and the sending time smoothing value corresponding to the 12th frame is 900. Of course, the time axis here is based on the coordinate axis with the first frame as the starting point at time 0, and it can also be converted to real time in actual application, such as Unix timestamp.

[0137] Specifically, the sending time smoothing value of the kth video frame may be determined when the previous video frame of the kth video frame, that is, the k-1th video frame, is decoded.

[0138] Step 504: Determine an estimated rendering time of the kth video frame based on the sending time smoothing value corresponding to the kth video frame.

[0139] In an embodiment of the present application, after obtaining the sending time smoothing value corresponding to the k-th video frame, the estimated rendering time of the k-th video frame can be determined based on the sending time smoothing value.

[0140] Step 505: Based on the estimated rendering time, read the kth video frame from the frame buffer, and perform rendering processing on the kth video frame.

[0141] When the estimated rendering time of the kth video frame is about to arrive, the frame data is taken out from the frame buffer and input into the decoder for decoding. After the decoding is completed, the renderer performs rendering and display.

[0142] In the embodiment of the present application, the updating process of the sending time estimator is an iterative updating process. Below, taking one updating process as an example, the updating process of the sending time estimator is introduced. Figure 7 As shown, it is a schematic diagram of the update process of the sending time estimator.

[0143] Step 5021: Based on the most recently updated state of the sending time estimator, determine the first priori state corresponding to the kth video frame.

[0144] The first a priori state refers to the a priori state of the transmission time estimator.

[0145] Due to the influence of many factors such as network or device capabilities, the order of receiving video frames is not necessarily strictly in the order of frame numbers. In the embodiment of the present application, the sending time estimator can be updated once each time a complete video frame is received. Therefore, the timing of the most recent status update is not necessarily the previous frame of the current frame. For example, if the current frame is the kth frame, the most recent update of the estimator status may be when the k-1th frame data is complete, or it may be when the k-2th frame data is complete.

[0146] When using the Kalman filter method, only the last estimator state and the current measurement value of the transmission time estimator need to be considered, without paying attention to the historical estimator state, so it is particularly simple and effective for linearly changing systems. Specifically, the Kalman filter method is a process of using the last estimator state to determine the current prior state, and then combining the measurement value to correct the current prior state to obtain the current posterior state.

[0147] The sending time estimator state includes the value of the sending time estimator state and the probability distribution of the value. Then, determining the first priori state corresponding to the k-th video frame based on the sending time estimator state after the most recent update may also include two parts:

[0148] (1) Based on the value of the sending time estimator state after the most recent update, the value of the first prior state corresponding to the k-th video frame is obtained, which is expressed as follows:

[0149] θ(k|k-1)=θ(k-1|k-1)

[0150] Wherein, θ(k|k-1) represents the value of the first priori state obtained based on the value of the send time estimator state after the most recent update, and θ(k-1|k-1) is the value of the send time estimator state after the most recent update.

[0151] (2) Based on the probability distribution of the value of the sending time estimator state after the most recent update, determine the probability distribution of the value of the first prior state corresponding to the kth video frame. Here, taking the covariance as an example to characterize the probability distribution, it can be expressed as follows:

[0152] P a (k|k-1)=P a (k-1|k-1)

[0153] Among them, P a (k|k-1) is the covariance corresponding to θ(k|k-1) obtained based on θ(k-1|k-1), P a (k-1|k-1) is the covariance corresponding to θ(k-1|k-1).

[0154] Step 5022: Determine a first weight value corresponding to the kth video frame based on a first priori state corresponding to the kth video frame.

[0155] Among them, the first weight value is expressed as follows:

[0156] K g1 (k) = P a (k|k-1)[i k ,1]′ / [i k ,1](P a (k|k-1)[i k ,1]′+R)

[0157] Among them, K g1 (k) is the first weight value corresponding to the kth video frame, also known as Kalman gain (KalmanGain), R is η k The corresponding covariance, [i k,1]′ means [i k ,1] is the transpose; the larger the first weight value is, the higher the confidence of the measured value of the sending time is, and the smaller the first weight value is, the higher the confidence of the smoothed value of the sending time determined based on the first prior state corresponding to the kth video frame is.

[0158] Step 5023: Based on the first weight value and the transmission time measurement value of the k-th video frame, the first priori state corresponding to the k-th video frame is updated to obtain the transmission time estimator state corresponding to the k-th video frame.

[0159] It can be seen that the prior values ​​of the current estimator state, namely θ(k|k-1) and P a (k|k-1), and the current transmission time measurement value is known, then the prior value and the observed value can be combined to obtain the transmission time estimator state corresponding to the current k-th video frame.

[0160] Specifically, the estimated value of the sending time corresponding to the kth video frame can be obtained based on the first priori state corresponding to the kth video frame, and the difference between the two values ​​of the sending time estimated value and the sending time measured value can be determined, and then based on the value of the first priori state corresponding to the kth video frame, and the difference value and the first weight value are weighted summed to obtain the value of the sending time estimator state corresponding to the kth video frame, that is, the optimized estimated value of the current estimator state, which can be specifically expressed as follows:

[0161] θ(k|k)=θ(k|k-1)+K g1 (k)(s k -[i k ,1]θ(k|k-1)

[0162] Wherein, θ(k|k) is the value of the state of the sending time estimator corresponding to the kth video frame.

[0163] In addition, the probability distribution of the value of the sending time estimator state corresponding to the kth video frame can be determined based on the probability distribution of the value of the first prior state corresponding to the kth video frame and the first weight value. That is to say, up to now, the optimal estimation value θ(k|k) at the kth video frame has been obtained, but in order for the sending time estimator to be able to continuously iterate until the system process ends, it is also necessary to update the probability distribution at the kth video frame. Here, the covariance is also taken as an example:

[0164] P a (k|k)=(IK g1 (k)[i k ,1])P a (k|k-1)

[0165] Where I is a matrix of 1. Generally speaking, the value of I can be 1.

[0166] When the frame data of the k+1th video frame is completely received, that is, the frame buffer has cached all the frame data of the k+1th video frame, the above process can be used to calculate the sending time estimator state corresponding to the k+1th video frame. In this way, as the video reception progresses, the sending time estimator can be continuously iteratively updated and the estimator state is continuously optimized.

[0167] In the embodiment of the present application, some abnormal situations may exist, such as changes in the sending frame rate, which may cause the sending time interval to change. In this case, the previously obtained sending time estimator is obviously not applicable to the changed frame rate. Of course, there are other abnormal situations that may make the sending time estimator unable to estimate normally. For such abnormal situations, the embodiment of the present application has carried out corresponding abnormality detection means.

[0168] Specifically, after each determination of the sending time smoothing value, it is determined whether the sending time error set corresponding to the video stream satisfies the zero mean condition. When it is determined that the sending time error set does not satisfy the zero mean condition, it is considered that an abnormality has occurred, and the sending time estimator is reset. For example, if the current state of the sending time estimator is the sending time estimator state corresponding to the k-th video frame, then the sending time estimator state corresponding to the k-th video frame will be reset to initialize the sending time estimator and perform subsequent iterative updates again.

[0169] The sending time error set includes sending time errors corresponding to multiple video frames, and each sending time error is a difference between a sending time smoothing value and a sending time measurement value of a video frame.

[0170] In an embodiment of the present application, in order to accurately estimate the network delay, a network delay estimator is also designed. Before calculating the estimated rendering time, the network delay estimator needs to be updated, and the updated network delay estimator is used to determine the network delay estimate.

[0171] See also Figure 8 As shown, it is a flow chart of a video data processing method combined with a network delay estimator provided in an embodiment of the present application. It should be noted that in the subsequent introduction, Figure 8 The embodiment shown is Figure 5 Similar steps in the illustrated embodiment will not be described in detail again, and reference may be made to the introduction of the aforementioned embodiment.

[0172] Step 801: Receive a video stream sent by a video transmitter, and store each video frame of the video stream in a frame buffer.

[0173] Step 802: Determine whether the frame buffer has stored complete data of a video frame.

[0174] If the frame buffer has stored complete data of the kth video frame, jump to step 803 and step 805 .

[0175] Step 803: Determine the send time estimator state corresponding to the kth video frame based on the send time estimator state after the most recent update and the send time measurement value of the kth video frame.

[0176] Step 804: Determine a sending time smoothing value corresponding to the kth video frame based on a sending time estimator state corresponding to the kth video frame.

[0177] Step 805: Determine the network delay estimator state corresponding to the kth video frame based on the most recently updated network delay estimator state and the sending time measurement value and the receiving time measurement value of the kth video frame.

[0178] In an embodiment of the present application, network delay can be estimated by a network delay estimator, and the network delay estimator state is the parameter of the network delay estimator, which is used to characterize the linear mapping relationship between the sending time and the receiving time of each video frame.

[0179] Similarly, the Kalman filter method can also be combined to implement the network delay estimation task. The network delay estimator state of the network delay estimator is the state in the Kalman filter method, where the state equation and observation equation of the network delay estimator are as follows:

[0180] ω k+1 =ω k +ξ k

[0181] s k =[r k ,1]·ω k +η k

[0182] Similar to the sending time estimator, the state of the network delay estimator is also a process of continuous iterative update. k+1 represents the state of the estimator after the k+1th update, that is, the state of the network delay estimator after the k+1th update, ω k represents the state of the estimator after the last update (k+1), that is, the state of the estimator after the kth update, ω k is a vector, since ω k That is, the network delay estimator state, and the network delay estimator state is used to characterize the linear mapping relationship between the sending time and the receiving time of each video frame, then its component ω k[1] is the network delay; s k Represents the sending time measurement value, that is, the network delay stamp carried in the video data packet, r k Represents the received time measurement value, that is, the received timestamp.

[0183] Among them, ξ k represents the state transition noise, η k represents the observation noise. In practical applications, ξ k and η k Gaussian noise may be used. It should be noted that, by introducing the state transition noise, the dynamic response capability of the network delay estimator is improved, and the network delay estimator can converge quickly when the network delay changes.

[0184] Similarly, the network delay estimator is updated when each video frame data is complete, that is, the k+1th update is performed after the k+1th video frame is received completely, then ω k+1 It can also be understood as the estimator state corresponding to the k+1th video frame.

[0185] Specifically, when the frame buffer has stored the complete data of the kth video frame, the control module in the jitter buffer controls the update of the network delay estimator. Referring to the above state equation, when updating, the network delay estimator state corresponding to the kth video frame is determined based on the network delay estimator state after the most recent update, and the transmission time measurement value and the reception time measurement value of the kth video frame. In an ideal situation, that is, the network delay of each video frame is smooth, then the network delay estimator state after the last update should be the same as the current network delay estimator state, that is, the network delay is the same, but in actual applications, there will inevitably be deviations, so the network delay estimator simulates the actual situation and introduces noise.

[0186] Step 806: Determine a network delay estimation value corresponding to the kth video frame based on a network delay estimator state corresponding to the kth video frame.

[0187] In an embodiment of the present application, after obtaining the state of the network delay estimator corresponding to the kth video frame, that is, after knowing the current state of the network delay estimator, the network delay estimator state can be used to determine the network delay estimation value corresponding to the kth video frame.

[0188] Specifically, knowing the current state of the network delay estimator ω k After that, the estimated network delay is ω k The component ω k [1]. See Fig. 9The figure shows the state diagram of the network delay estimator in an ideal environment without jitter. The state of the network delay estimator represents the mapping relationship between the sending time and the receiving time. La' represents the change relationship of the sending time, and Lb' represents the change relationship of the receiving time. Based on the current mapping relationship, it can be seen that the network delay is equivalent to the change relationship of the receiving time based on the sending time change relationship and the above mapping relationship, that is, the distance of La' being shifted to the right, that is, the network delay is 800ms.

[0189] Step 807: Determine the estimated rendering time of the kth video frame based on the sending time smoothing value and the network delay estimation value corresponding to the kth video frame.

[0190] Step 808: Based on the estimated rendering time, read the kth video frame from the frame buffer, and perform rendering processing on the kth video frame.

[0191] In the embodiment of the present application, taking into account the instability of the network conditions, there may be a situation where the cache length of the jitter buffer is long. In this case, it is possible that the rendering time calculation timing of the k-th video frame has not yet arrived, and the caching of other video frames has been completed in the meantime. In this case, the sending time estimator and the network delay estimator are updated using the information of the newly cached video frames. When it is necessary to calculate the rendering time of the k-th video frame, the sending time estimator and the network delay estimator after the most recent update can be used to estimate the sending time and network delay.

[0192] Specifically, after the sending time estimator state corresponding to the kth video frame is obtained, but the time for calculating the sending time smoothing value has not yet arrived, if the frame buffer has stored complete data of the k+nth video frame, then the sending time estimator state corresponding to the k+n-1th video frame can be determined based on the sending time estimator state corresponding to the k+n-1th video frame and the sending time measurement value of the k+nth video frame; wherein n is a positive integer not less than 1, and then when it is necessary to calculate the sending time smoothing value corresponding to the kth video frame, the sending time estimator state corresponding to the k+nth video frame is used as the sending time estimator state corresponding to the kth video frame to participate in the calculation, so that it can be ensured that the sending time estimator after the last update is always used to calculate the sending time.

[0193] Of course, the same is true for the estimation of network delay, that is, after the network delay estimator state corresponding to the kth video frame is obtained, but before the time to calculate the network delay estimate, if the frame buffer has stored the complete data of the k+nth video frame, then the network delay estimator state corresponding to the k+n-1th video frame can be determined based on the network delay estimator state corresponding to the k+n-1th video frame, and the sending time measurement value and receiving time measurement value of the k+nth video frame; then, when it is necessary to calculate the network delay estimate corresponding to the kth video frame, the network delay estimator state corresponding to the k+nth video frame is used as the network delay estimator state corresponding to the kth video frame to participate in the calculation. In this way, it can be ensured that the network delay estimator after the last update is always used to calculate the network delay.

[0194] In the embodiment of the present application, similar to the transmission time estimator, the updating process of the network delay estimator is also an iterative updating process. Below, taking one updating process as an example, the updating process of the network delay estimator is introduced. Fig.10 As shown, it is a schematic diagram of the update process of the network delay estimator.

[0195] Step 8051: Based on the most recently updated network delay estimator state, determine the second prior state corresponding to the kth video frame.

[0196] The second priori state refers to the priori state of the network delay estimator.

[0197] In an embodiment of the present application, the network delay estimator can be updated every time a complete video frame is received. Similar to the sending time estimator, when using the Kalman filtering method, it is only necessary to pay attention to the previous estimator state and the current measurement value of the network delay estimator, without paying attention to the historical estimator state. Therefore, it is particularly simple and effective for linear changing systems.

[0198] The network delay estimator state includes the value of the network delay estimator state and the probability distribution of the value. Then, determining the second prior state corresponding to the k-th video frame based on the most recently updated network delay estimator state may also include two parts:

[0199] (1) Based on the value of the network delay estimator state after the most recent update, the value of the second prior state corresponding to the kth video frame is obtained, which is expressed as follows:

[0200] ω(k|k-1)=ω(k-1|k-1)

[0201] Among them, ω(k|k-1) represents the value of the second prior state corresponding to the kth video frame obtained based on the value of the network delay estimator state after the most recent update, and ω(k-1|k-1) is the value of the network delay estimator state after the most recent update.

[0202] (2) Based on the probability distribution of the value of the network delay estimator state after the most recent update, determine the probability distribution of the value of the second prior state corresponding to the kth video frame. Here, taking the covariance characterization probability distribution as an example, it can be expressed as follows:

[0203] P b (k|k-1)=P b (k-1|k-1)+Q

[0204] Among them, P b (k|k-1) is the covariance corresponding to ω(k|k-1) obtained based on ω(k-1|k-1), P b (k-1|k-1) is the covariance corresponding to ω(k-1|k-1), and Q is ξ k The corresponding covariance.

[0205] Step 8052: Based on the second priori state corresponding to the kth video frame, determine a second weight value corresponding to the kth video frame.

[0206] The second weight value is expressed as follows:

[0207] K g2 (k) = P b (k|k-1)[r k ,1]′ / [r k ,1](P b (k|k-1)[r k ,1]′+R)

[0208] Among them, K g1 (k) is the second weight value corresponding to the kth video frame, also known as Kalman gain (KalmanGain), R is η k The corresponding covariance, [i k ,1]′ means [i k ,1] is the transpose; the larger the second weight value is, the higher the confidence level of the network delay measurement value is, and the smaller the second weight value is, the higher the confidence level of the network delay estimation value determined based on the prior state corresponding to the kth video frame is.

[0209] Step 8053: Based on the second weight value and the sending time measurement value and the receiving time measurement value of the kth video frame, the prior state corresponding to the kth video frame is updated to obtain the network delay estimator state corresponding to the kth video frame.

[0210] It can be seen that the prior values ​​of the current estimator state, namely ω(k|k-1) and P b (k|k-1), and the current sending time measurement value and receiving time measurement value are known, then the prior value and the observed value can be combined to obtain the network delay estimator state corresponding to the current k-th video frame.

[0211] Specifically, the network delay estimation value corresponding to the kth video frame can be obtained based on the prior state corresponding to the kth video frame, and the network delay measurement value can be obtained based on the sending time measurement value and the receiving time measurement value, and then the difference between the network delay estimation value and the network delay measurement value can be determined, and then based on the value of the prior state corresponding to the kth video frame, and the difference value and the second weight value are weighted summed to obtain the value of the network delay estimator state corresponding to the kth video frame, that is, the optimized estimation value of the current estimator state, which can be specifically expressed as follows:

[0212] ω(k|k)=ω(k|k-1)+K g2 (k)(s k -[r k , 1]ω(k|k-1)

[0213] Wherein, ω(k|k) is the value of the network delay estimator state corresponding to the kth video frame.

[0214] In addition, the probability distribution of the value of the network delay estimator state corresponding to the kth video frame can be determined based on the probability distribution of the value of the prior state corresponding to the kth video frame and the second weight value. That is to say, up to now, the optimal estimation value ω(k|k) at the kth video frame has been obtained, but in order for the network delay estimator to be able to iterate continuously until the system process ends, it is also necessary to update the probability distribution at the kth video frame. Here, the covariance is also taken as an example:

[0215] P b (k|k)=(IK g2 (k)[i k ,1])P b (k|k-1)

[0216] Where I is a matrix of 1. Generally speaking, the value of I can be 1.

[0217] When the frame data of the k+1th video frame is completely received, that is, the frame buffer has cached all the frame data of the k+1th video frame, the above process can be used to calculate the network delay estimator state corresponding to the k+1th video frame. In this way, as the video reception progresses, the network delay estimator can perform autoregressive operations, continuously perform iterative updates, and continuously optimize the estimator state.

[0218] In the embodiment of the present application, considering that there may be some abnormal situations, which may make the network delay estimator unable to make accurate estimates, the embodiment of the present application implements corresponding abnormality detection means to address such abnormal situations.

[0219] Specifically, after each determination of the network delay estimation value, it is determined whether the network delay error set corresponding to the video stream satisfies the zero mean condition. When it is determined that the network delay error set does not satisfy the zero mean condition, that is, once the network delay estimation error deviates from the zero mean, it is considered that an abnormality has occurred, and the network delay estimator is reset. The network delay error set includes network delay errors corresponding to multiple video frames, and each network delay error is the difference between the network delay estimation value and the network delay measurement value of a video frame.

[0220] Specifically, in order to avoid serious fluctuations in the estimation results of the network delay estimator, a light reset method can be used for resetting. Light reset means resetting only part of the state of the network delay estimator, such as resetting the covariance of the network delay estimator. In this way, when the network delay suddenly changes, the network delay estimator can quickly track the sudden change and quickly and accurately estimate the network delay. For example, if the current state of the network delay estimator is the state of the network delay estimator corresponding to the kth video frame, then the covariance of the state of the network delay estimator corresponding to the kth video frame will be reset.

[0221] In the embodiment of the present application, before calculating the estimated rendering time, it is also necessary to update the network jitter estimator and use the updated network jitter estimator to determine the network jitter estimation value.

[0222] See also Fig.11 As shown, it is a flow chart of a video data processing method combined with a network jitter estimator provided in an embodiment of the present application. It should be noted that in the subsequent introduction, Fig.11 The embodiment shown is Figure 5 or Figure 8 Similar steps in the illustrated embodiment will not be described in detail again, and reference may be made to the introduction of the aforementioned embodiment.

[0223] Step 1101: based on the estimated rendering time of the k-1th video frame, read the k-1th video frame from the frame buffer, and perform rendering processing on the k-1th video frame.

[0224] Among them, the process of determining the estimated rendering time of the k-1th video frame is similar to the process of determining the estimated rendering time of the k-1th video frame, so it is not repeated here. After the estimated rendering time of the k-1th video frame is determined, when the estimated rendering time of the k-1th video frame is about to be reached, the k-1th video frame can be read from the frame buffer and rendered.

[0225] Step 1102: When the rendering of the k-1th video frame is finished, the network jitter estimation parameter is updated based on the sending time measurement value and the receiving time measurement value of the k-1th video frame.

[0226] In a weak network environment, it is necessary to take into account both the jam rate and the delay. By using network jitter as an added constant bias, the ability to cope with sudden network fluctuations can be enhanced. Adding a constant bias of network jitter to the rendering time of each video frame is equivalent to delaying the rendering time of the video frame. Although it will introduce a certain delay, in a weak network environment, a certain amount of time can be reserved for receiving video data. The role of the constant bias is to ensure the minimum water level of the jitter buffer. For example, if the minimum water level is 50ms, then sudden jitter within 50ms will not affect the rendering of the video frame, and will not cause problems such as video jams.

[0227] In addition, the embodiments of the present application also reduce the delay in severe weak network environments by reducing the upper quantile when estimating network jitter. The upper quantile refers to the upper quantile of the distribution of network jitter values. For example, for a normal distribution, if 95% is selected as the upper quantile, it means that for a frame, there is a 95% probability that the actual jitter is less than the estimated network jitter.

[0228] Step 1103: Based on the updated network jitter estimation parameters, determine the most recently updated network jitter estimation value.

[0229] Specifically, after the network jitter estimator is updated, the updated network jitter estimator may be used to determine the network jitter estimation value after the most recent update.

[0230] Step 1104: Determine whether the frame buffer has stored complete data of a video frame.

[0231] If the frame buffer has stored complete data of the kth video frame, jump to step 803 and step 805 .

[0232] Step 1105: Determine the send time estimator state corresponding to the kth video frame based on the send time estimator state after the most recent update and the send time measurement value of the kth video frame.

[0233] Step 1106: Determine a sending time smoothing value corresponding to the kth video frame based on the sending time estimator state corresponding to the kth video frame.

[0234] Step 1107: Determine the network delay estimator state corresponding to the kth video frame based on the most recently updated network delay estimator state and the sending time measurement value and the receiving time measurement value of the kth video frame.

[0235] Step 1108: Determine a network delay estimation value corresponding to the kth video frame based on a network delay estimator state corresponding to the kth video frame.

[0236] Step 1109: Determine the estimated rendering time based on the sending time smoothing value corresponding to the k-th video frame, the network delay estimation value, and the most recently updated network jitter estimation value.

[0237] See also Fig.12 As shown, it is a schematic diagram of the estimated rendering time under ideal conditions without jitter, wherein Lc' is the line connecting the estimated rendering times of each video frame, wherein each dot in the diagram represents a timestamp data, the dot on La represents the measured value of the sending time of a video frame, the dot on La' represents the smoothed value of the sending time of a video frame, the dot on Lb represents the measured value of the receiving time of a video frame, the dot on Lb' represents the estimated value of the receiving time of a video frame, the distance (time difference) between the dots on La' and Lb' corresponding to the same frame is the network delay, the distance between the dots on Lb' and Lc' corresponding to the same frame is the estimated value of the network jitter, that is, the estimated rendering time of each video frame is equal to the sum of the smoothed value of the sending time, the estimated value of the network delay and the estimated value of the network jitter.

[0238] For example, for the 12th frame, the sending time smoothing value is 900ms, the network delay estimated value is 800ms, and the network jitter estimated value calculated when the rendering of the 11th frame ends is 400ms. Then the estimated rendering time of the 12th frame is 2100ms.

[0239] based on Fig.12 It can be seen that in actual situations, the sending time and network delay are not smooth. After being processed by the method of the embodiment of the present application, the rendering time can be smoothed, so that the video can be played smoothly and the jamming can be reduced.

[0240] Step 1110: Based on the estimated rendering time, read the kth video frame from the frame buffer, and perform rendering processing on the kth video frame.

[0241] In an embodiment of the present application, when the rendering of the kth video frame is completed, the network jitter estimator is also updated based on the sending time measurement value and the receiving time measurement value of the k-1th video frame, and the updated network jitter estimator is used to calculate the current network jitter for the subsequent estimation of the rendering time, so that the process can continue to iterate.

[0242] In summary, the embodiment of the present application provides a sending time estimator, which can smoothly estimate the sending time interval on the receiving side to eliminate the problem of uneven rendering time caused by sending time jitter, and by performing high-sensitivity abnormal detection on the sending time, when a change in the sending frame rate or other abnormal scenarios is detected, the sending time estimator is reset in time; in addition, a network delay estimator is also provided, which can independently and quickly perform network delay estimation, and does not rely on the retransmission module in the related technology, and realizes high-sensitivity abnormal detection of network delay, realizes abnormal detection with low false alarm probability and lightweight reset operation, corrects the abnormal state of the filter, and ensures that the estimation of each parameter does not fluctuate too much; and, the embodiment of the present application also controls the update and estimation timing of each estimator, as well as the conditions for packets or frames to enter and exit the jitter buffer, that is, when the data of a frame is complete, the sending time estimator and the network delay estimator are updated in sequence, when the data of a frame is complete and it is the next frame of the frame just decoded, the sending time smoothing value and the network delay estimation value are calculated to calculate the estimated rendering time of the frame, when the estimated rendering time of a frame is reached, the frame is updated from the frame. The frame is taken out from the buffer, decoded, rendered, and the network jitter estimator is updated, and the current network jitter is calculated. In this way, the water level of the jitter buffer can be controlled at a reasonable level under different group of pictures (GOP) sizes and frames per second (FPS) conditions, so that most frames can always be rendered smoothly at the same time, because the delay added by the jitterbuffer will not be too long.

[0243] After processing by the embodiments of the present application, in a normal network scenario, the video can maintain a low delay without any freeze. In a slightly weak network scenario, freeze can be completely avoided by appropriately increasing the delay. In a severely weak network scenario, freeze cannot be completely avoided, but it can be minimized. By continuing to increase the delay, excessive delay can also be avoided.

[0244] See also Fig.13 Based on the same inventive concept, the embodiment of the present application further provides a video data processing device 130, which includes:

[0245] The receiving unit 1301 is used to receive the video stream sent by the video sending end, and store each video frame of the video stream into the frame buffer;

[0246] The sending time estimator unit 1302 is used to determine the sending time estimator state corresponding to the k-th video frame based on the sending time estimator state after the most recent update and the sending time measurement value of the k-th video frame when the frame buffer has stored the complete data of the k-th video frame; wherein k is a positive integer not less than 1, and the sending time estimator state is used to characterize the sending frame rate of the video stream; and determine the sending time smoothing value corresponding to the k-th video frame based on the sending time estimator state corresponding to the k-th video frame;

[0247] A rendering time estimation unit 1303, configured to determine an estimated rendering time of the kth video frame based on a sending time smoothing value corresponding to the kth video frame;

[0248] The decoding and rendering unit 1304 is used to read the kth video frame from the frame buffer based on the estimated rendering time, and perform rendering processing on the kth video frame.

[0249] Optionally, the device further includes a network delay estimation unit 1305, and the network delay estimation unit 1305 is used to:

[0250] Determine a network delay estimator state corresponding to the kth video frame based on the most recently updated network delay estimator state and the measured value of the sending time and the measured value of the receiving time of the kth video frame; wherein the network delay estimator state is used to characterize a linear mapping relationship between the sending time and the receiving time of each video frame;

[0251] Determine a network delay estimation value corresponding to the kth video frame based on a network delay estimator state corresponding to the kth video frame;

[0252] The rendering time estimation unit 1303 is specifically configured to determine the estimated rendering time based on the sending time smoothing value corresponding to the k-th video frame and the network delay estimation value.

[0253] Optionally, the device further includes a network jitter estimation unit 1306, and the network jitter estimation unit 130 is used to:

[0254] When the rendering of the k-1th video frame is finished, the network jitter estimation parameter is updated based on the sending time measurement value and the receiving time measurement value of the k-1th video frame;

[0255] Determine a network jitter estimation value corresponding to the k-1th video frame based on the updated network jitter estimation parameter;

[0256] The rendering time estimation unit 1303 is specifically used to determine the estimated rendering time based on the sending time smoothing value corresponding to the kth video frame, the network delay estimation value and the network jitter estimation value corresponding to the k-1th video frame.

[0257] Optionally, the sending time estimation unit 1302 is specifically configured to:

[0258] Determine a first priori state corresponding to the kth video frame based on the state of the sending time estimator after the most recent update;

[0259] Based on the first priori state corresponding to the kth video frame, determine a first weight value corresponding to the kth video frame; wherein, the larger the first weight value, the higher the confidence of the transmission time measurement value, and the smaller the first weight value, the higher the confidence of the transmission time smoothing value determined based on the first priori state;

[0260] Based on the first weight value and the transmission time measurement value of the k-th video frame, the first priori state corresponding to the k-th video frame is updated to obtain the transmission time estimator state corresponding to the k-th video frame.

[0261] Optionally, the sending time estimation unit 1302 is specifically configured to:

[0262] Based on the value of the transmission time estimator state after the most recent update, obtaining the value of the first priori state corresponding to the k-th video frame; and,

[0263] Based on the probability distribution of the value of the sending time estimator state after the most recent update, the probability distribution of the value of the first priori state corresponding to the kth video frame is determined.

[0264] Optionally, the sending time estimation unit 1302 is specifically configured to:

[0265] Determine a first priori state corresponding to the kth video frame to obtain an estimated value of a sending time corresponding to the kth video frame, and determine a difference between the estimated value of the sending time corresponding to the kth video frame and a measured value of the sending time corresponding to the kth video frame;

[0266] Obtaining a value of a transmission time estimator state corresponding to the kth video frame based on a value of a first priori state corresponding to the kth video frame, and performing a weighted summation on the difference value and the first weight value;

[0267] Based on the probability distribution of the value of the first priori state corresponding to the kth video frame and the first weight value, the probability distribution of the value of the sending time estimator state corresponding to the kth video frame is determined.

[0268] Optionally, the sending time estimation unit 1302 is specifically configured to:

[0269] Determine a target sending time corresponding to the kth video frame based on a sending frame rate represented by a sending time estimator state corresponding to the kth video frame and a frame number of the kth video frame;

[0270] The target sending time is determined as a smoothed value of the sending time of the kth video frame.

[0271] Optionally, the device further includes an abnormality detection unit 1307, which is used to:

[0272] Determine whether a set of sending time errors corresponding to the video stream satisfies a zero mean condition; wherein each sending time error in the sending time error set is a difference between a sending time smoothing value and a sending time measurement value of a video frame;

[0273] When it is determined that the sending time error set does not satisfy the zero mean condition, the sending time estimator state corresponding to the k-th video frame is reset.

[0274] Optionally, the network delay estimation unit 1305 is specifically configured to:

[0275] Determine a second prior state corresponding to the kth video frame based on the most recently updated network delay estimator state;

[0276] Based on the second priori state corresponding to the kth video frame, determine a second weight value corresponding to the kth video frame; wherein, the larger the second weight value, the higher the confidence of the network delay measurement value, and the smaller the second weight value, the higher the confidence of the network delay estimation value obtained based on the second priori state corresponding to the kth video frame;

[0277] Based on the second weight value and the sending time measurement value and the receiving time measurement value of the kth video frame, the second priori state corresponding to the kth video frame is updated to obtain the network delay estimator state corresponding to the kth video frame.

[0278] Optionally, the network delay estimation unit 1305 is specifically configured to:

[0279] Based on the value of the network delay estimator state after the most recent update and the random noise, obtain the value of the second prior state corresponding to the k-th video frame; and,

[0280] Based on the probability distribution of the value of the network delay estimator state after the most recent update, the probability distribution of the value of the second prior state corresponding to the kth video frame is determined.

[0281] Optionally, the anomaly detection unit 1307 is further configured to:

[0282] Determine whether a network delay error set corresponding to the video stream satisfies a zero mean condition; wherein each network delay error in the network delay error set is a difference between a network delay estimation value and a network delay measurement value of a video frame;

[0283] When it is determined that the network delay error set does not satisfy the zero mean condition, the probability distribution of the value of the network delay estimator state corresponding to the kth video frame is reset.

[0284] Optionally, the sending time estimation unit 1302 is specifically configured to:

[0285] When the frame buffer has stored complete data of the k+nth video frame, the video receiving end determines the state of the sending time estimator corresponding to the k+nth video frame based on the sending time estimator state after the most recent update and the sending time measurement value of the k+nth video frame; wherein n is a positive integer not less than 1;

[0286] Using the sending time estimator state corresponding to the k+n-th video frame as the sending time estimator state corresponding to the k-th video frame;

[0287] When the decoding of the k-1th video frame is finished, the sending time smoothing value of the kth video frame is determined based on the sending time estimator state corresponding to the kth video frame.

[0288] The device can be used to perform Figure 5 to Figure 12 Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 5 to Figure 12 The description of the illustrated embodiment is not repeated in detail.

[0289] See also Fig.14 Based on the same technical concept, an embodiment of the present application also provides a computer device 140, which may include a memory 1401 and a processor 1402.

[0290] The memory 1401 is used to store computer programs executed by the processor 1402. The memory 1401 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the computer device, etc. The processor 1402 may be a central processing unit (CPU), or a digital processing unit, etc. The specific connection medium between the above-mentioned memory 1401 and the processor 1402 is not limited in the embodiments of the present application. The embodiments of the present application are Fig.14 In the embodiment, the memory 1401 and the processor 1402 are connected via a bus 1403. The bus 1403 is connected to the processor 1402 via a bus 1403. Fig.14The connection between other components is shown by bold lines, which is only for schematic illustration and is not intended to be limiting. The bus 1403 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0291] The memory 1401 may be a volatile memory, such as a random-access memory (RAM); the memory 1401 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1401 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1401 may be a combination of the above memories.

[0292] The processor 1402 is used to execute the following when calling the computer program stored in the memory 1401: Figure 5 to Figure 12 The method executed by the device in the illustrated embodiment.

[0293] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 5 to Figure 12 The method executed by the device in the illustrated embodiment.

[0294] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0295] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0296] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video data processing method, characterized in that: The method comprises: The video receiving end receives the video stream sent by the video sending end, and stores each video frame of the video stream in a frame buffer; When the frame buffer has stored complete data of the kth video frame, the video receiving end performs the following operations: Determine a first a priori state corresponding to the k-th video frame based on the most recently updated sending time estimator state; wherein k is a positive integer not less than 1, and the sending time estimator state is used to characterize the sending frame rate of the video stream; Based on the first priori state corresponding to the kth video frame, determine a first weight value corresponding to the kth video frame; wherein the larger the first weight value is, the higher the confidence of the transmission time measurement value is, and the smaller the first weight value is, the higher the confidence of the transmission time smoothing value determined based on the first priori state is; Based on the first weight value and the measured value of the sending time of the k-th video frame, updating a first priori state corresponding to the k-th video frame to obtain a sending time estimator state corresponding to the k-th video frame; Determine a sending time smoothing value corresponding to the kth video frame based on a sending time estimator state corresponding to the kth video frame; Determining an estimated rendering time of the kth video frame based on a sending time smoothing value corresponding to the kth video frame; Based on the estimated rendering time, the kth video frame is read from the frame buffer, and rendering processing is performed on the kth video frame.

2. The method according to claim 1, characterized in that Before determining the estimated rendering time of the kth video frame based on the sending time smoothing value corresponding to the kth video frame, the method further includes: Determine the network delay estimator state corresponding to the kth video frame based on the most recently updated network delay estimator state and the measured value of the sending time and the measured value of the receiving time of the kth video frame; wherein the network delay estimator state is used to characterize the linear mapping relationship between the sending time and the receiving time of each video frame; Determine a network delay estimation value corresponding to the kth video frame based on a network delay estimator state corresponding to the kth video frame; Then, based on the sending time smoothing value corresponding to the k-th video frame, determining the estimated rendering time of the k-th video frame includes: The estimated rendering time is determined based on the sending time smoothing value corresponding to the k-th video frame and the network delay estimation value.

3. The method according to claim 2, characterized in that Before determining the estimated rendering time based on the sending time smoothing value corresponding to the k-th video frame and the network delay estimation value, the method further includes: When the rendering of the k-1th video frame is finished, updating the network jitter estimation parameter based on the sending time measurement value and the receiving time measurement value of the k-1th video frame; Determine a network jitter estimation value corresponding to the k-1th video frame based on the updated network jitter estimation parameter; Then, based on the sending time smoothing value corresponding to the k-th video frame and the network delay estimation value, determining the estimated rendering time includes: The estimated rendering time is determined based on the sending time smoothing value corresponding to the k-th video frame, the network delay estimation value, and the network jitter estimation value corresponding to the k-1-th video frame.

4. The method according to claim 1, characterized in that Determining a first a priori state corresponding to the kth video frame based on the most recently updated state of the sending time estimator includes: Based on the value of the transmission time estimator state after the most recent update, obtaining the value of the first priori state corresponding to the k-th video frame; and, Based on the probability distribution of the value of the sending time estimator state after the most recent update, the probability distribution of the value of the first priori state corresponding to the k-th video frame is determined.

5. The method according to claim 4, characterized in that Based on the first weight value and the transmission time measurement value of the k-th video frame, updating the first priori state corresponding to the k-th video frame to obtain the transmission time estimator state corresponding to the k-th video frame includes: Determine a first priori state corresponding to the k-th video frame to obtain an estimated value of a sending time corresponding to the k-th video frame, and determine a difference between the estimated value of the sending time corresponding to the k-th video frame and a measured value of the sending time corresponding to the k-th video frame; Obtaining a value of a transmission time estimator state corresponding to the kth video frame based on a value of a first priori state corresponding to the kth video frame, and performing a weighted summation on the difference value and the first weight value; Based on the probability distribution of the value of the first priori state corresponding to the kth video frame and the first weight value, the probability distribution of the value of the sending time estimator state corresponding to the kth video frame is determined.

6. The method according to claim 1, characterized in that Determining a sending time smoothing value corresponding to the kth video frame based on a sending time estimator state corresponding to the kth video frame includes: Determine a target sending time corresponding to the kth video frame based on a sending frame rate represented by a sending time estimator state corresponding to the kth video frame and a frame number of the kth video frame; The target sending time is determined as a sending time smoothing value of the kth video frame.

7. The method according to any one of claims 1 to 6, characterized in that: After determining the sending time smoothing value corresponding to the kth video frame based on the sending time estimator state corresponding to the kth video frame, the method further includes: Determine whether the set of sending time errors corresponding to the video stream meets the zero mean condition; wherein each sending time error in the sending time error set is the difference between the sending time smoothing value and the sending time measurement value of a video frame; When it is determined that the sending time error set does not satisfy the zero mean condition, the sending time estimator state corresponding to the k-th video frame is reset.

8. The method according to claim 2, characterized in that Determining a network delay estimator state corresponding to the kth video frame based on a most recently updated network delay estimator state and a measured value of a sending time and a measured value of a receiving time of the kth video frame, comprising: Determine a second priori state corresponding to the k-th video frame based on the most recently updated network delay estimator state; Based on the second priori state corresponding to the kth video frame, determine a second weight value corresponding to the kth video frame; wherein the larger the second weight value, the higher the confidence of the network delay measurement value, and the smaller the second weight value, the higher the confidence of the network delay estimation value obtained based on the second priori state corresponding to the kth video frame; Based on the second weight value and the sending time measurement value and the receiving time measurement value of the kth video frame, the second priori state corresponding to the kth video frame is updated to obtain the network delay estimator state corresponding to the kth video frame.

9. The method according to claim 8, characterized in that Determining a second priori state corresponding to the kth video frame based on the most recently updated network delay estimator state includes: Based on the value of the network delay estimator state after the most recent update and random noise, obtaining the value of the second priori state corresponding to the k-th video frame; and, Based on the probability distribution of the value of the network delay estimator state after the most recent update, the probability distribution of the value of the second prior state corresponding to the k-th video frame is determined.

10. The method according to any one of claims 8-9, characterized in that: After determining the network delay estimation value corresponding to the kth video frame based on the network delay estimator state corresponding to the kth video frame, the method further includes: Determine whether the network delay error set corresponding to the video stream satisfies a zero mean condition; wherein each network delay error in the network delay error set is a difference between a network delay estimation value and a network delay measurement value of a video frame; When it is determined that the network delay error set does not satisfy the zero mean condition, the probability distribution of the value of the network delay estimator state corresponding to the k-th video frame is reset.

11. The method according to any one of claims 1 to 6 or 8 to 9, characterized in that: Before determining the sending time smoothing value corresponding to the kth video frame based on the sending time estimator state corresponding to the kth video frame, the method further includes: When the frame buffer has stored complete data of the k+nth video frame, the video receiving end determines the sending time estimator state corresponding to the k+nth video frame based on the sending time estimator state after the most recent update and the sending time measurement value of the k+nth video frame; wherein n is a positive integer not less than 1; Using the sending time estimator state corresponding to the k+n-th video frame as the sending time estimator state corresponding to the k-th video frame; Then, based on the state of the sending time estimator corresponding to the k-th video frame, determining the sending time smoothing value corresponding to the k-th video frame includes: When the decoding of the k-1th video frame is finished, a sending time smoothing value of the kth video frame is determined based on a sending time estimator state corresponding to the kth video frame.

12. A video data processing device, characterized in that: The device comprises: A receiving unit, configured to receive a video stream sent by a video sending end, and store each video frame of the video stream in a frame buffer; A sending time estimation unit, used for determining a first a priori state corresponding to the kth video frame based on a sending time estimator state after the most recent update when the frame buffer has stored complete data of the kth video frame; determining a first weight value corresponding to the kth video frame based on the first a priori state corresponding to the kth video frame; wherein the larger the first weight value, the higher the confidence of the sending time measurement value, and the smaller the first weight value, the higher the confidence of the sending time smoothing value determined based on the first a priori state; updating the first a priori state corresponding to the kth video frame based on the first weight value and the sending time measurement value of the kth video frame to obtain a sending time estimator state corresponding to the kth video frame; wherein k is a positive integer not less than 1, and the sending time estimator state is used to characterize the sending frame rate of the video stream; and determining a sending time smoothing value corresponding to the kth video frame based on the sending time estimator state corresponding to the kth video frame; A rendering time estimation unit, configured to determine an estimated rendering time of the k-th video frame based on a sending time smoothing value corresponding to the k-th video frame; A decoding and rendering unit is used to read the kth video frame from the frame buffer based on the estimated rendering time, and perform rendering processing on the kth video frame.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images