Video frame processing method and apparatus, and device, computer-readable storage medium and computer program product

Abstract

Disclosed in the present application are a video frame processing method and apparatus, and a device, a computer-readable storage medium and a computer program product. The method comprises: on the basis of a capture timestamp of a video frame in a cache frame queue, predicting a first rendering timestamp of the video frame; on the basis of the video frame in the cache frame queue, determining a frame smoothing policy, wherein the frame smoothing policy is used for predicting a delay parameter of the first rendering timestamp by means of a reception time interval between adjacent data packets, and the data packets are used for carrying the video frame; on the basis of the frame smoothing policy, determining a delay parameter, and on the basis of the delay parameter, delaying the first rendering timestamp to obtain a second rendering timestamp; on the basis of the second rendering timestamp, a reception timestamp of the video frame, and a decoding time consumption parameter of the decoded video frame, determining a decoding waiting duration of the video frame; and in response to the storage duration of the video frame in the cache frame queue reaching the decoding waiting duration, decoding the video frame.

Description

A method, apparatus, device, computer-readable storage medium, and computer program product for processing video frames.

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2024109239219, filed on July 10, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of video processing technology, and in particular to a method, apparatus, device, computer-readable storage medium, and computer program product for processing video frames. Background Technology

[0004] In real-time video scenarios such as video conferencing, screen sharing, or live streaming, video frame data needs to be transmitted over a network. In environments with unstable network transmission, various problems may occur during video playback, such as video stuttering.

[0005] The real-time transmission and processing methods for video frames in related technologies (such as Web Real-Time Communications (WebRTC)) are related to actual business scenarios. For example, parameters such as video encoding algorithm strategies, network transmission algorithm strategies, and flow control algorithm strategies need to be optimized due to differences between business scenarios. Direct use will result in the prediction deviation of the video frame decoding waiting time, which will introduce a large delay and make it difficult to stably control the video frame decoding time, thus affecting the smoothness of video frame decoding. Summary of the Invention

[0006] This application provides a method, apparatus, device, computer-readable storage medium, and computer program product for processing video frames, which can stably control the decoding time of video frames, thereby improving the smoothness of video frame decoding and rendering.

[0007] This application provides a video frame processing method, applied to an electronic device, including:

[0008] Based on the acquisition timestamps of video frames in the cached frame queue, predict the first rendering timestamp of the video frame;

[0009] A frame smoothing strategy is determined based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames;

[0010] The delay parameter is determined based on the frame smoothing strategy, and the first rendering timestamp is delayed based on the delay parameter to obtain the second rendering timestamp;

[0011] Based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame, the decoding waiting time of the video frame is determined;

[0012] In response to the video frame being stored in the cached frame queue for a period of time that reaches the decoding waiting time, the video frame is decoded.

[0013] This application provides a video frame processing method, applied to an electronic device, including:

[0014] A video frame to be encoded is obtained, and the video frame to be encoded is encoded to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.

[0015] This application provides a method for processing audio / video streams, applied to electronic devices, wherein the audio / video streams are generated according to the video frame processing method described above in this application, or decoded based on the video frame processing method described above in this application.

[0016] This application provides a video frame processing apparatus, including:

[0017] The prediction processing module is configured to predict the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue.

[0018] A frame smoothing processing module is configured to determine a frame smoothing strategy based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames;

[0019] The frame smoothing processing module is further configured to determine the delay parameter based on the frame smoothing strategy, and delay the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp;

[0020] The frame smoothing processing module is further configured to determine the decoding waiting time of the video frame based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame;

[0021] The frame output module is configured to decode the video frame in response to the video frame being stored in the cached frame queue for a period of time equal to the decoding waiting time.

[0022] This application provides a video frame processing apparatus, including:

[0023] The processing module is configured to acquire a video frame to be encoded and encode the video frame to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.

[0024] This application provides an electronic device, the electronic device comprising:

[0025] Memory, configured to store computer-executable instructions;

[0026] When a processor is configured to execute computer-executable instructions or computer programs stored in the memory, it implements the video frame processing method provided in the embodiments of this application.

[0027] This application provides a computer storage medium storing executable instructions. When these executable instructions are executed by a processor, they implement the video frame processing method provided in this application.

[0028] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the video frame processing method provided in this application.

[0029] The embodiments of this application include at least the following beneficial effects: By using the temporal stability of the acquisition timestamp as a benchmark, the first rendering timestamp of the video frame is obtained, providing a reliable time reference benchmark for the calculation of subsequent decoding waiting time. By obtaining the corresponding frame smoothing strategy based on the video frames in the cache frame queue, an adaptive decision-making process for the frame smoothing strategy is realized. By using the frame smoothing strategy and the first rendering timestamp, the second rendering timestamp of the video frame is determined, realizing the selection of the corresponding frame smoothing strategy according to the specific situation of the video frames in the cache frame queue. This allows for different degrees of delay for the first rendering timestamp of different video frames, avoiding the accumulation of decoding waiting time caused by a single frame smoothing strategy and the introduction of excessive decoding waiting time. Finally, by using the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video, the decoding waiting time of the video frame is quantified, thereby improving the matching degree between the decoding waiting time of the video frame in the cache frame queue and the actual decoding requirements. In summary, by making the decision on the smoothing strategy based on the state of the video frames in the cache frame queue, the accuracy of obtaining the decoding waiting time of the video frame is improved, achieving the beneficial effects of stably controlling the decoding time of the video frame and improving the smoothness of video frame decoding and rendering. Attached Figure Description

[0030] Figure 1 is a schematic diagram of the architecture of the video frame processing system provided in an embodiment of this application;

[0031] Figure 2A is a first structural schematic diagram of the electronic device provided in an embodiment of this application;

[0032] Figure 2B is a schematic diagram of the second structure of the electronic device provided in an embodiment of this application;

[0033] Figure 3A is a first flowchart of the video frame processing method provided in an embodiment of this application;

[0034] Figure 3B is a second flowchart illustrating the video frame processing method provided in an embodiment of this application;

[0035] Figure 3C is a schematic diagram of the third process of the video frame processing method provided in the embodiments of this application;

[0036] Figure 3D is a schematic diagram of the fourth process of the video frame processing method provided in the embodiments of this application;

[0037] Figure 3E is a fifth flowchart illustrating the video frame processing method provided in an embodiment of this application;

[0038] Figure 3F is a sixth flowchart illustrating the video frame processing method provided in the embodiments of this application;

[0039] Figure 3G is a schematic diagram of the seventh process of the video frame processing method provided in the embodiments of this application;

[0040] Figure 3H is a schematic diagram of the eighth process of the video frame processing method provided in the embodiments of this application;

[0041] Figure 3I is a ninth flowchart illustrating the video frame processing method provided in an embodiment of this application;

[0042] Figure 3J is a schematic diagram of the tenth process of the video frame processing method provided in the embodiment of this application;

[0043] Figure 3K is a schematic diagram of the eleventh step of the video frame processing method provided in the embodiments of this application;

[0044] Figure 3L is a schematic diagram of the twelfth step of the video frame processing method provided in the embodiments of this application;

[0045] Figure 3M is a schematic diagram of the thirteenth step of the video frame processing method provided in the embodiments of this application;

[0046] Figure 4 is a schematic diagram of the time sliding window provided in an embodiment of this application;

[0047] Figure 5 is a flowchart illustrating the video frame processing method in a video call scenario provided in an embodiment of this application;

[0048] Figure 6A is a schematic diagram illustrating the principle of video frame processing provided in the embodiments of this application;

[0049] Figure 6B is a schematic diagram illustrating the principle of storing video frames into a buffer frame queue according to an embodiment of this application;

[0050] Figure 6C is a flowchart illustrating the process of determining the frame smoothing strategy provided in an embodiment of this application.

[0051] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0055] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of this application belong. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of embodiments of this application.

[0057] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0058] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0059] 1) The buffered frame queue, also known as the frame smoothing queue, is a data structure used to store video frames to be decoded until they can be read from the buffered frame queue for the decoder to decode.

[0060] 2) Acquisition timestamp refers to the time stamp corresponding to the moment a video frame is acquired by the acquisition device. The acquisition timestamp is used to indicate the generation time of the video frame, so as to sort the video frames in chronological order and perform related temporal analysis.

[0061] 3) The first rendering timestamp, also known as the estimated rendering timestamp, refers to the predicted time point in the video rendering process when the rendering task of a video frame is completed.

[0062] 4) The second rendering timestamp, also known as the expected rendering timestamp, refers to the predicted time point for completing the rendering task of a video frame, obtained by delaying the first rendering timestamp during the video rendering process. It is used to plan the progress of the rendering task and ensure that the video frame can be rendered within the predetermined time.

[0063] 5) Frame smoothing strategy: This strategy uses time delay to smooth out video frames, thereby reducing or eliminating problems such as uneven video frame rendering and stuttering caused by sudden changes in video speed at the video receiving device.

[0064] 6) Smooth frame rendering is a technique that controls the decoding wait time of received video frames to make the rendering time of video frames as stable as possible, thereby making the video look smoother.

[0065] 7) Receive timestamp refers to the time stamp recorded by the receiving device when the video frame data arrives at the receiving device.

[0066] 8) The current timestamp refers to the number of seconds or milliseconds that have elapsed since a specific moment (e.g., the Unix epoch, which is midnight on January 1, 1970). The current timestamp can be obtained using a time library in a programming language, such as Python's datetime library or JavaScript's Date object.

[0067] 9) Decoding wait time refers to the time interval between when a video frame enters the buffer frame queue and when it is removed to call the decoder for decoding. In other words, the storage time of the video frame in the buffer frame queue is the time when the storage time of the video frame in the buffer frame queue reaches the decoding wait time. The video frame is then removed from the buffer frame queue and transferred to the decoder for decoding.

[0068] 10) Network jitter refers to the irregularity and instability of the arrival time of data packets of video frames during the transmission process from the sending end device to the receiving end device in a computer network due to network conditions, traffic congestion, etc., which manifests as changes in the time interval between adjacent data packets arriving at the receiving end device.

[0069] 11) Packet loss retransmission is a network transmission mechanism used to recover data packets of video frames when they are lost in the network. When the sending device sends a data packet, the receiving device sends an acknowledgment packet (ACK) to indicate receipt. If the receiving device does not receive a data packet within the expected time (e.g., to determine whether a data packet has been received based on its sequence number), it sends a negative acknowledgment packet (NACK) to inform the sending device.

[0070] 12) Video frames waiting to enter the buffer frame queue refer to video frames in the buffer (JitterBuffer). When a video frame satisfies the reference relationship, it will be transferred from the buffer to the buffer frame queue.

[0071] The reference relationship refers to the fact that when a video frame (such as a P-frame or B-frame) is decoded, it needs to rely on a reference frame (such as an I-frame or a decoded P-frame). If the reference frame of the current frame is not received, not decoded, or the decoding fails, the current frame does not meet the reference relationship and needs to be buffered to wait for entry into the buffer frame queue. If the reference frame of the current frame is ready (such as being received and decoded), the current frame meets the reference relationship and can be transferred from the buffer to the buffer frame queue.

[0072] For example, under packet loss retransmission and network jitter, when the received video frame does not satisfy the reference relationship (that is, when the reference frame of the currently received video frame has not yet been received, for example, the currently received video frame is P2, and the previous reference frame corresponding to P2 can be represented as I0, and different video frames can correspond to the same reference frame), it will be buffered in the buffer of the receiving device.

[0073] 13) Reference frames are frames in a video frame sequence that are used to help decode subsequent video frames; that is, the decoding of a video frame depends on the preceding reference frames. These reference frames are used by the encoder to generate prediction information, and the decoder uses this prediction information to reconstruct the video image.

[0074] 14) Decoded frame refers to a video frame that has been successfully decoded. For example, after a video frame is removed from the buffer frame queue, the decoder is called to decode the removed video frame. After successful decoding, the removed video frame is recorded as a decoded frame.

[0075] 15) The decoding frame interval parameter is the interval between decoding frames. For example, it is a statistical measure of the decoding interval between two adjacent decoding frames (i.e., two video frames decoded sequentially) within a historical time period of a preset duration from the current time, such as the average value or quantile.

[0076] 16) The first time period, also known as the jitter period, refers to the time range during which, due to changes in network latency or other reasons, the data packets of a video frame arrive at the receiving device in a situation where the time interval between adjacent data packets is greater than a preset time interval threshold (e.g., 100ms). For example, when the time window length is 2s, if there is a situation where the time interval between adjacent data packets received within 2s arrives at the receiving device is greater than the preset time interval threshold, then the 2s range corresponding to the time window is the first time period.

[0077] 17) The second time period, also known as the packet loss retransmission period, refers to the time range from the start of retransmission of data packets to the arrival of the retransmitted data packets at the receiving device when the time interval between the arrival of data packets at the receiving device is less than or equal to a preset time interval threshold (e.g., 100ms) and data packets are lost during transmission. For example, in a video conferencing scenario, the sending device is sending a series of video frames, each frame being divided into multiple data packets for transmission. The arrival time interval of the data packets during transmission is less than or equal to the preset time interval threshold (e.g., 100ms), but the data packets of one video frame are lost during transmission. After receiving a retransmission request, the sending device will immediately begin retransmitting the lost data packets. The time from when the sending device sends the retransmission data packets until the receiving device successfully receives these data packets and puts them into the video stream is the second time period.

[0078] 18) The third time period refers to the time range during which data packets arrive at the receiving device within a preset time interval threshold (e.g., 100ms) during the transmission of video frame data packets, and no data packets are lost during transmission. For example, suppose the sending device is sending a series of video frames, each frame is divided into multiple data packets for transmission, and within a certain period (e.g., 5s), the data packets are successfully transmitted to the receiving device. During this time period, no data packets are lost, and the arrival time interval of data packets during transmission is less than or equal to the preset time interval threshold (e.g., 100ms). This time period is the third time period.

[0079] 19) Receive frame rate, which represents the number of video frames entering the buffer frame queue per second.

[0080] 20) Outgoing frame rate, which represents the number of video frames removed from the buffer frame queue per second to be decoded by the decoder.

[0081] Real-time transmission and processing methods for video frames in related technologies (such as WebRTC) are related to actual business scenarios. For example, parameters such as video encoding algorithm strategies, network transmission algorithm strategies, and flow control algorithm strategies need to be optimized due to differences between business scenarios. Direct use will result in jitter prediction deviations, which will introduce large delays and make it difficult to stably control the decoding time of video frames, thus affecting the smoothness of video frame decoding.

[0082] Related technologies employ simple statistical methods for processing video frames. These methods calculate the reception time intervals between two or more adjacent video frames, using statistical variables such as average or variance. If the statistical variable of the currently received video frame falls outside a corresponding threshold, a certain waiting delay is added before the current video frame is sent to the decoder for decoding and rendering. For example, by calculating the average variance of the reception intervals of the most recent 1 second (15-30 frames), in the event of network jitter, when video frames do not meet the reference relationship (e.g., the reference frame has not yet been received), the video frames will quickly accumulate in the buffer. Once the reference relationship is met (the corresponding reference frame is received), the frame will be quickly output. At this point, the variance rapidly increases to a certain threshold. After half the average interval of the video frames, the frame is then sent to the decoder for decoding and rendering. However, because the reception time of video frames varies with factors such as network jitter, it is impossible to smoothly output frames using a single time base, affecting the smoothness of video frame decoding.

[0083] Based on this, embodiments of this application provide a video frame processing method, apparatus, device, computer-readable storage medium, and computer program product, which can stably control the decoding time of video frames, thereby improving the smoothness of video frame decoding and rendering.

[0084] The electronic devices provided in this application can be implemented as various types of terminal devices, such as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smartphones, smart speakers, smart voice interaction devices, smart home appliances, smartwatches, smart TVs, in-vehicle terminals, smart headphones, and aircraft. Exemplary applications of the electronic devices implemented as terminal devices will be described below.

[0085] Referring to Figure 1, which is a schematic diagram of the video frame processing system architecture provided in an embodiment of this application, Figure 1 involves server 100, terminal device 200-1, terminal device 200-2, and network 300. Terminal device 200-1 and terminal device 200-2 are connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0086] In some embodiments, the terminal device or server can implement the video frame processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as instant messaging clients; or they can be applets embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0087] In some embodiments, server 100 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminals.

[0088] In some embodiments, the video frame processing system provided in this application can be implemented collaboratively by a server and a terminal device. For example, during a video call, referring to Figure 1, terminal device 200-1 communicates with terminal device 200-2 via an instant messaging client. The sensor device of terminal device 200-1 (e.g., the built-in camera of terminal device 200-1) captures video frames, divides the video frames into multiple data packets, and transmits them to terminal device 200-2 via the server 100 in the background of the instant messaging client. In response to receiving the data packets of the video frames sent by terminal device 200-1, terminal device 200-2 combines the data packets into a video frame and stores it in a buffer frame queue. Next, terminal device 200-2 performs frame smoothing processing on the video frame using the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frame. When the storage time of the video frame in the buffer frame queue reaches the waiting time, the video frame is removed from the buffer frame queue and sent to the decoder of terminal device 200-2. The decoder is called to decode the video frame, and finally, the decoded video frame is displayed on the interface of terminal device 200-2 through video rendering.

[0089] Taking a video call scenario as an example, the terminal device is used to capture video frames, encapsulate the video frames into data packets and send them to the server, and receive the peer video data packets forwarded by the server, combine them into video frames and store them in a buffered frame sequence. The video frames are smoothed using the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames. When the storage time of the video frame in the buffered frame queue reaches the waiting time, the video frame is removed from the buffered frame queue and the decoder is called to decode the video frame. The server is used to relay video data packets between terminal devices to realize the establishment and maintenance of end-to-end real-time data transmission links.

[0090] Taking a multi-person video conferencing scenario as an example, the terminal device is used to collect local video streams and encode them into data packets, receive video data packets from other participants distributed by the server, combine them into video frames, and store them in a buffered frame sequence. The video frames are then processed by the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames. When the storage time of the video frame in the buffered frame queue reaches the waiting time, the video frame is removed from the buffered frame queue, and the decoder is called to decode the video frame. The server is used to aggregate multiple video streams, perform signaling interaction management (such as participant join / leave notifications), and distribute data (such as broadcasting a video stream from one end to other terminals).

[0091] Taking a live streaming scenario as an example, the terminal device (broadcaster end) is used to capture the live stream image and split it into data packets to upload to the server; the terminal device (viewer end) is used to receive the live video data packets pushed by the server, combine them into video frames and store them in the cache frame sequence. The video frames are smoothed using the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames. When the storage time of the video frame in the cache frame queue reaches the waiting time, the video frame is dequeued from the cache frame queue and the decoder is called to decode the video frame; the server is used to receive the video stream from the broadcaster end, perform transcoding (such as multi-resolution adaptation), and distribute it to the viewer end through the Content Delivery Network (CDN).

[0092] In some embodiments, the video frame processing system provided in this application can be implemented by a server alone. For example, server 100 acquires data packets, combines the data packets into video frames, and stores them in a buffer frame queue. Next, server 100 performs frame smoothing processing on the video frames using the video frame processing method provided in this application to obtain the decoding waiting time of the video frames. When the storage time of the video frames in the buffer frame queue reaches the waiting time, the video frames are dequeued from the buffer frame queue and sent to the decoder of server 100. The decoder is then called to decode the video frames to obtain the decoded video frames.

[0093] In some embodiments, the video frame processing system provided in this application can be implemented by a terminal device. For example, terminal device 200-1 captures video frames, divides the video frames into multiple data packets, and transmits them to terminal device 200-2. In response to terminal device 200-2 receiving the data packets of the video frames sent by terminal device 200-1, terminal device 200-2 combines the data packets into video frames and stores them in a buffer frame queue. Next, terminal device 200-2 performs frame smoothing processing on the video frames using the video frame processing method provided in this application to obtain the decoding waiting time of the video frames. When the storage time of the video frames in the buffer frame queue reaches the waiting time, the video frames are dequeued from the buffer frame queue and sent to the decoder of terminal device 200-2. The decoder is called to decode the video frames, and finally, the decoded video frames are displayed on the interface of terminal device 200-2 through video rendering.

[0094] Taking an electronic device as a receiving device to process encoded video frames as an example, refer to Figure 2A. Figure 2A is a first structural schematic diagram of the electronic device provided in this application embodiment. The electronic device can be implemented as terminal device 200-1, terminal device 200-2, or server 100 as described above. The electronic device 400-1 shown in Figure 2A includes: at least one processor 410-1, a memory 450-1, at least one network interface 420-1, and a user interface 430-1. The various components in the electronic device 400-1 are coupled together through a bus system 440-1. It can be understood that the bus system 440-1 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440-1 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440-1 in Figure 2.

[0095] The processor 410-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0096] User interface 430-1 includes one or more output devices 431-1 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430-1 also includes one or more input devices 432-1, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0097] The memory 450-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450-1 may optionally include one or more storage devices physically located away from the processor 401-1.

[0098] The memory 450-1 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450-1 described in this application embodiment is intended to include any suitable type of memory.

[0099] In some embodiments, memory 450-1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0100] Operating system 451-1 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0101] The network communication module 452-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420-1, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0102] Presentation module 453-1 is used to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431-1 (e.g., a display screen, a speaker, etc.) associated with user interface 430-1;

[0103] The input processing module 454-1 is used to detect and translate one or more user inputs or interactions from one or more input devices 432-1.

[0104] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2A shows a video frame processing apparatus 455 stored in memory 450-1, which can be software in the form of programs and plug-ins, including the following software modules: prediction processing module 4551, frame smoothing processing module 4552, and frame output module 4553. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0105] Taking an electronic device as the transmitting end device and processing the video frame to be encoded as an example, refer to Figure 2B. Figure 2B is a schematic diagram of the second structure of the electronic device provided in the embodiment of this application. The electronic device can be implemented as the terminal device 200-1, terminal device 200-2 or server 100 mentioned above. The electronic device 400-2 shown in Figure 2B includes: at least one processor 410-2, a memory 430-2 and at least one network interface 420-2. The various components in the electronic device 400-2 are coupled together through a bus system 440-2. It can be understood that the bus system 440-2 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440-2 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, all buses are labeled as bus system 440-2 in Figure 2B. For a detailed description of the processor 410-2 and the memory 430-2, please refer to the above, and it will not be repeated here.

[0106] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2B shows a processing apparatus 456 for video frames stored in memory 430-2, which can be software in the form of programs and plug-ins, including the following software module: processing module 4561. The function of the module will be described below.

[0107] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video frame processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0108] The following will describe the video frame processing method provided in this application embodiment, with the electronic device (such as a terminal device) as the execution subject, in conjunction with the exemplary application and implementation of the electronic device provided in this application embodiment. The terminal device described above can be used as the sending device to execute the video frame processing method provided in this application embodiment.

[0109] In some embodiments, a video frame to be encoded is obtained and encoded to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method provided in the embodiments of this application.

[0110] For example, the sending device acquires a video frame to be encoded (or the original video frame), performs video preprocessing, video encoding, and packetization on the video frame to be encoded, and obtains a data packet of the encoded video frame. The acquisition timestamp and transmission time are sent to the receiving device along with the data packet. The receiving device decodes the encoded video frame using the video frame processing method provided in this application embodiment.

[0111] Video preprocessing refers to a series of processing steps performed on video frames before they are encoded. These processes can include color correction, brightness / contrast adjustment, noise suppression, filtering, and image resizing. The goal is to improve the quality of the video frames, making them more suitable for subsequent encoding or to meet specific display requirements. The preprocessing stage may also include advanced processing such as frame rate conversion and scene detection to optimize video content and reduce the complexity of the encoding process.

[0112] Video coding is the process of converting video frames into a compressed format for storage and transmission. For example, the image of a video frame to be encoded is decomposed into smaller units (such as macroblocks), and then a series of mathematical transformations (such as discrete cosine transform) are applied to these units to eliminate redundant information in the data. Subsequently, the transform coefficients are adjusted through a quantization process to reduce the amount of data that needs to be transmitted. Video coding also includes dividing the image data of the video frame to be encoded into different layers, such as spatial layers, temporal layers, and redundancy layers, which can provide different levels of compression while maintaining video frame quality. Commonly used video coding standards include High Efficiency Video Coding (HEVC) and the Open Media Consortium (AOMedia Video 1, AV1) coding standard.

[0113] Packet segmentation refers to the process of dividing encoded video frames into smaller data packets. These data packets, including portions of the video frames, are used for transmission over the network to the end user's device (the receiving device). The purpose of packet segmentation is to adapt to different network conditions, such as bandwidth limitations and latency requirements. Packet segmentation enables more efficient transmission control, such as adjusting the frequency, size, and order of data packet transmission to optimize network resource usage and ensure smooth video streaming in unstable network environments.

[0114] The following will describe the video frame processing method provided in this application embodiment, with the electronic device (implemented as a terminal device) as the execution subject, in conjunction with exemplary applications and implementations of the electronic device provided in this application embodiment. Referring to FIG3A, FIG3A is a first flowchart of the video frame processing method provided in this application embodiment, which can be executed by the aforementioned terminal device as the receiving end device, and will be described in conjunction with the steps shown in FIG3A.

[0115] In some embodiments, referring to Figure 1, during a video call, terminal device 200-1 can act as a sending device, and terminal device 200-2 can act as a receiving device. Here, sending and receiving devices are relative concepts. During video communication, one terminal device simultaneously acts as both a sender and a receiver. For example, when user 1 of terminal device 200-1 is having a video call with user 2 of terminal device 200-2, the camera of terminal device 200-1 captures an image of user 1, encodes it into video frames, and sends it to terminal device 200-2. At the same time, terminal device 200-1 also receives video frames sent by terminal device 200-2, which are obtained by capturing images of user 2. This means that the hardware (such as the camera and screen) and software (such as the codec and user interface) on the same device need to work together to achieve data transmission and reception. The following description is from the perspective of the receiving device.

[0116] In step 101, the first rendering timestamp of the video frame is predicted based on the acquisition timestamp of the video frame in the cached frame queue.

[0117] In some embodiments, the transmitting device of a video frame stores information such as the acquisition timestamp of each video frame and the transmission time of the transmitting device in the frame header information. After the transmitting device performs video preprocessing, video encoding and packetization on the video frame to be encoded, it obtains a data packet and sends the acquisition timestamp and transmission time information along with the data packet to the receiving device.

[0118] In some embodiments, the receiving device receives data packets and combines them into video frames (here, video frames are encoded video frames). When a video frame satisfies the reference relationship, it is stored in the buffer frame queue. Here, the reference relationship means that the decoding of the video frame depends on the previous reference frame. If the reference frame of the current video frame has not yet been received by the receiving device, the video frame will wait in the buffer until the reference frame of the video frame is successfully received, that is, after the reference relationship is satisfied, the video frame will be sent to the buffer frame queue for subsequent processing such as determining the frame smoothing strategy.

[0119] In some embodiments, referring to FIG3B, step 101 shown in FIG3A can be implemented by steps 1011 to 1013, which are described in detail below.

[0120] In step 1011, the received frame rate level of the video frame is obtained.

[0121] In some embodiments, the video frame receiving frame rate within the time period corresponding to the video frame is obtained, the video frame receiving frame rate is mapped to multiple preset frame rate levels, and the mapped frame rate level is used as the receiving frame rate level.

[0122] For example, we can count the number of video frames received per second (corresponding to the time period of the video frames mentioned above) (i.e., the video frame receiving frame rate), and map the video frame receiving frame rate to the closest frame rate on the flow control policy level. There can be multiple frame rate levels on the flow control level. For example, the frame rate level can be the video frame acquisition frame rate or encoding frame rate of the sending device, such as 12FPS, 15FPS, and 20FPS. Assuming that the number of video frames received per second is 18, and the closest frame rate on the flow control policy level is 20FPS, then the corresponding video frame receiving frame rate level is 20FPS.

[0123] This application embodiment maps the received frame rate of video frames to a preset frame rate level (such as 12FPS, 15FPS, etc.), which can filter out small fluctuations in the real-time frame rate and establish a stable frame rate benchmark, thereby improving the adaptability of the video frame processing method provided in this application embodiment to dynamic network environments.

[0124] In step 1012, the receiving time difference of the video frames is obtained.

[0125] In some embodiments, referring to FIG3C, step 1012 shown in FIG3B can be implemented by steps 10121 to 10122, which are described in detail below.

[0126] In step 10121, the difference between the received timestamp of multiple video frames and the sum of the video frame's acquisition timestamp, round-trip delay, and transmission time is obtained.

[0127] In some embodiments, the difference between the received timestamps of multiple video frames and the sum of the video frame's acquisition timestamp, round-trip time, and transmission time can be calculated. The transmitting device stores information such as the acquisition timestamp of each video frame and the transmission time of the transmitting device in the frame header information. After receiving the data packet and framing it, the receiving device obtains the acquisition timestamp and transmission time of the video frame from the frame header information.

[0128] As an example, in a video call scenario, the statistical range can be from the start of the video call to the current moment, or it can be one or more sampling periods (e.g., 2 seconds) during the video call process. The difference within one or more sampling periods can be counted. One or more sampling periods can be traced back from the current moment, so as to accurately reflect the current network status.

[0129] For example, the difference for each video frame can be expressed as: Difference = Video frame reception timestamp - Video frame acquisition timestamp - Transmission time - Round trip delay.

[0130] In step 10122, the average value of the differences corresponding to multiple video frames is taken as the reception time difference.

[0131] Following the example in step 10121, the average of the differences corresponding to multiple video frames within the above statistical range is taken as the reception time difference.

[0132] This application embodiment calculates the difference between the sum of the received timestamp and the acquisition timestamp, round-trip delay, and transmission time of multiple video frames, and takes the average value as the received time difference. This can dynamically capture the overall trend of network transmission delay, filter out the interference of abnormal fluctuations in single frames, improve the accuracy of the received time difference in representing the real-time network status, and provide a more reliable delay compensation benchmark for the subsequent prediction of the first rendering timestamp.

[0133] Referring again to Figure 3B, in step 1013, the first rendering timestamp of the video frame is determined based on the received frame rate level, the received time difference, and the acquisition timestamp.

[0134] In some embodiments, referring to FIG3D, step 1013 shown in FIG3B can be implemented by steps 10131 to 10133, which are described in detail below.

[0135] In step 10131, the reciprocal of the preset multiple of the received frame rate level is obtained.

[0136] In some embodiments, the reciprocal of a preset multiple (e.g., 1000) of the receive frame rate level is obtained. The receive frame rate level can be, for example, 12 FPS, 15 FPS, and 20 FPS. The receive frame rate level can be flexibly set according to the requirements of the flow control strategy.

[0137] Taking a preset multiplier of 1000 as an example, the reciprocal of the preset multiplier receiving frame rate level can be expressed as: 1000 × (1 / receiving frame rate level).

[0138] In step 10132, the sum between the collection timestamp and the reciprocal is obtained.

[0139] In some embodiments, the collection timestamp and the reciprocal are summed to obtain a sum.

[0140] Following the example in step 10131, the acquisition timestamp and the reciprocal are summed to obtain the sum, which can be expressed as: acquisition timestamp of video frame + 1000 × (1 / receiving frame rate level) = acquisition timestamp of video frame + 1000 / receiving frame rate level.

[0141] In step 10133, the summation of the summation and the receiving time difference is performed to obtain the first rendering timestamp.

[0142] Following the example in step 10132, the first rendering timestamp corresponding to the video frame can be expressed as: First rendering timestamp = video frame acquisition timestamp + 1000 / receiving frame rate level + receiving time difference, where the summation refers to the sum between the acquisition timestamp and the reciprocal, i.e., "video frame acquisition timestamp + 1000 / receiving frame rate level".

[0143] Referring again to Figure 3A, in step 102, a frame smoothing strategy is determined based on the video frames in the cached frame queue.

[0144] In some embodiments, video frames are obtained by combining received data packets. For multiple data packets received within the current time window (configured according to a preset duration, such as 2s), if the time interval between all or more of the adjacent data packets is greater than the time interval threshold (such as 100ms), it is determined that the current moment is in the first time period (also known as the jitter time period). It is then determined that before the time window slides again according to a predetermined step size (such as 2s), a first frame smoothing strategy is applied to the buffer frame queue. The starting point of the current time window is a historical moment that is a preset time interval away from the current moment, and the ending point of the current time window is the current moment. The first frame smoothing strategy is used to delay the video frames in the buffer frame queue within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the buffer frame queue.

[0145] It's important to note that "synchronous change" refers to the dynamic relationship between the frame smoothing factor and the number of video frames in the cached frame queue. Specifically, when the number of cached frames is large, the frame smoothing factor is increased, thereby reducing the output frame rate and decreasing decoding pressure. Conversely, when the number of cached frames is small, the frame smoothing factor is decreased, increasing the output frame rate. This utilizes existing frame resources to fill gaps caused by network fluctuations, preventing the decoder from idling. Thus, the decoding strategy is dynamically adjusted based on network conditions and cache status, improving the smoothness and stability of video playback. For example, when network jitter causes longer packet reception intervals, the number of cached frames increases, the frame smoothing factor increases accordingly, and the output frame rate decreases, thereby mitigating the impact of network fluctuations on video playback.

[0146] For example, once it is determined that the current time is within the jitter period, the first frame smoothing strategy can be started for the video frames in the buffer frame queue until the jitter value (the time interval between the reception of adjacent data packets) in the 2-second time window is lower than the preset time interval threshold (e.g., 100ms), at which point the first frame smoothing strategy for the video frames in the buffer frame queue can be stopped.

[0147] For example, referring to Figure 4, which is a schematic diagram of a time window provided in an embodiment of this application, jitter values ​​can be measured using a sliding time window algorithm. First, a time window with a length of 2 seconds is set. Within the current time window, whenever a data packet arrives, its arrival time is recorded; these arrival times are within the current time window. As time progresses, the time window slides along the time axis in steps of a certain size. As shown in Figure 4, time window 1 with a length of 2 seconds slides in steps of 1.5 seconds to obtain the next time window 2. Time window 1 and time window 2 overlap to ensure that the arrival times of data packets within the most recent 2 seconds are always included. Jitter can be measured by calculating the arrival time difference of all data packets within the time window. For example, for each data packet in the time window, the arrival time difference between it and the previous data packet in the window (the reception time interval of adjacent data packets) is calculated. These time differences are combined into a jitter time series. For example, the arrival time difference between data packet 2 and data packet 1, the arrival time difference between data packet 3 and data packet 2, and the arrival time difference between data packet 4 and data packet 3 corresponding to time window 1 shown in Figure 4 are obtained. The absolute value, mean, median, or preset quantile (e.g., 95th quantile) of these time differences can also be obtained as the jitter value corresponding to the current time window. For example, the 95th quantile of the arrival time difference between data packets corresponding to time window 1 in Figure 4 can be obtained as the jitter value corresponding to time window 1. As time goes by, the time window will continue to slide, record the arrival time of new data packets, and update the jitter value.

[0148] In some embodiments, for multiple data packets received within the current time window (configured according to a preset duration, such as 2 seconds), if the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not within the first time period), then it is determined that the current time is within the second time period (also known as the packet loss retransmission time period), and it is determined that a second frame smoothing strategy will be applied to the buffer frame queue before the next slide of the current time window. Here, the starting point of the current time window is a historical time that is a preset duration (i.e., the length of the time window) away from the current time, and the ending point of the time window is the current time (the step size of the window can also be preset, and the ending point of the time window is the current time after each slide). The second frame smoothing strategy is used to delay the video frames in the buffer frame queue based on the decoding frame interval parameter within the second time period. The decoding frame interval parameter is the interval between the decoding times of the already decoded video frames, which can be a statistical number of the decoding intervals between the already decoded video frames within a historical time period that is a preset duration away from the current time, such as the average value, a preset quantile, etc.

[0149] If the receiving time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time interval), then the current time is determined to be in the second time interval. This can be achieved in the following way: If the receiving time interval of adjacent data packets is less than or equal to the time interval threshold, and at least one data packet is lost during transmission within the current time window, then the current time is determined to be in the second time interval.

[0150] By determining that the current time is in the second time period (packet loss retransmission period) when the time interval between adjacent data packets is less than or equal to the time interval threshold and at least one data packet is lost within the current sliding window, this can accurately locate the packet loss retransmission scenario to avoid misjudging normal network fluctuations as packet loss retransmission, and can also be more targeted when the second frame smoothing strategy is triggered, reducing performance loss when there is no packet loss.

[0151] In some embodiments, for multiple data packets received within the current time window (configured according to a preset duration, such as 2 seconds), if the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period), then it is determined that the current time is in the third time period, and it is determined that a third frame smoothing strategy is applied to the buffer frame queue before the next slide of the time window. Here, the starting point of the time window is a historical time that is a preset duration away from the current time (i.e., the length of the time window), and the ending point of the time window is the current time. The third frame smoothing strategy is used to delay a specific video frame in the buffer frame queue based on a fixed frame smoothing factor within the third time period. The specific video frame refers to a video frame that meets the following condition: the time difference between the reception timestamp in the buffer frame queue and the decoding time of the previous decoded frame is less than a preset proportion of the decoding frame interval parameter.

[0152] Here, the decoding time of a decoded frame can be the time when the decoding of the decoded frame is completed, or the time when the decoding of the decoded frame begins.

[0153] If the receiving time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time interval), then the current time is determined to be in the third time interval. This can be achieved in the following way: if the receiving time interval of adjacent data packets is less than or equal to the time interval threshold, and the data packets are not lost during transmission within the current time window, then the current time is determined to be in the third time interval.

[0154] By determining that the current time is in the third time period when the time interval between adjacent data packets is less than or equal to the time interval threshold and no data packets are lost within the current sliding window, the strategy based on the fixed frame smoothing factor is avoided in case of packet loss and retransmission, reducing unnecessary delay processing of video frames. At the same time, for specific video frames in the buffer frame queue whose time difference between the received timestamp and the decoding time of the previous decoding frame is less than the preset proportion of the decoding frame interval parameter, precise delay optimization is achieved. While maintaining smooth video playback, the performance loss and playback delay caused by the mis-triggered strategy are effectively reduced.

[0155] As an example of the case-by-case application of the aforementioned second and third frame smoothing strategies, for multiple data packets received within the current time window (configured according to a preset duration, such as 2 seconds), if the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period), and a data packet is lost during transmission within the current time window, then the current time is determined to be in the second time period (also known as the packet loss retransmission period), and the second frame smoothing strategy is applied to the buffer frame queue before the next slide of the current time window; if the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and no data packet is lost during transmission within the current time window, then the current time is determined to be in the third time period, and the third frame smoothing strategy is applied to the buffer frame queue before the next slide of the current time window.

[0156] Referring again to Figure 3A, in step 103, the first rendering timestamp is delayed based on the frame smoothing strategy to obtain the second rendering timestamp.

[0157] In some embodiments, when the frame smoothing strategy is a first frame smoothing strategy that delays video frames during a first time period (jitter time period) based on a frame smoothing factor that changes synchronously with the number of video frames in the cached frame queue, referring to FIG3E, step 103 shown in FIG3A can be implemented by steps 1031A to 1033A, which are described in detail below.

[0158] In step 1031A, a frame smoothing factor is determined based on the number of video frames in the buffer frame queue, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffer frame queue.

[0159] In some embodiments, referring to FIG3F and FIG3E, step 1031A can be implemented by steps 10311 to 10313, which are described in detail below.

[0160] In step 10311, the initialized frame smoothing factor is determined based on the decoding frame interval parameter.

[0161] In some embodiments, the initial frame smoothing factor can be determined based on the decoding frame interval parameter corresponding to the current video frame.

[0162] For example, the decoding frame interval parameter is the interval between the decoding times of already decoded video frames. It can be a statistical count of the decoding intervals between already decoded video frames within a historical time period of a preset duration from the current time, such as the average value or a preset quantile (e.g., the 95th quantile). Here, the decoding frame interval (the interval between the timestamps of the two successful decoding times when two adjacent video frames are successfully decoded) is less than or equal to the minimum value between the jitter average and a preset threshold (e.g., 500). The jitter average refers to the average of the reception time intervals of adjacent data packets in the current time window.

[0163] In some embodiments, when the decoding frame interval parameter is greater than the first decoding frame interval parameter threshold, a maximum value constraint is applied to the decoding frame interval parameter, such that the decoded frame interval parameter after the maximum value constraint is equal to the first decoding frame interval parameter threshold; when the decoding frame interval parameter is less than the first decoding frame interval parameter threshold, a minimum value constraint is applied to the decoding frame interval parameter, such that the decoded frame interval parameter after the maximum value constraint is equal to the second decoding frame interval parameter threshold, wherein the first decoding frame interval parameter threshold is greater than the second decoding frame interval parameter threshold.

[0164] For example, assuming the first decoding frame interval parameter threshold is 125ms and the second decoding frame interval parameter threshold is 33ms, when the value of the decoding frame interval parameter is greater than 125ms, the value of the decoding frame interval parameter is taken as 125ms; when the value of the decoding frame interval parameter is less than 33ms, the value of the decoding frame interval parameter is taken as 33ms.

[0165] In some embodiments, the initial frame smoothing factor corresponding to the decoding frame parameter is determined according to the preset parameter range corresponding to the decoding frame interval parameter. The larger the value of the preset parameter range corresponding to the decoding frame interval parameter, the smaller the value of the initial frame smoothing factor (here, the value of the frame smoothing factor refers to the ratio of the received frame rate to the output frame rate). That is, the value of the decoding frame interval parameter is inversely proportional to the value of the frame smoothing factor, and the value of the decoding frame interval parameter is directly proportional to the output frame rate.

[0166] For example, under the same receive frame rate, the larger the decoding frame interval parameter, the larger the corresponding output frame rate. When the decoding frame interval parameter is greater than 90ms, the initial frame smoothing factor can be set to 5 / 4, which means that when the receive frame rate is around 30FPS, the minimum output frame rate is 24FPS to smooth the output frame (removing it from the buffer frame queue for decoding).

[0167] When the decoding frame interval parameter is between 60 and 90 ms, the initial frame smoothing factor can be set to 4 / 3, which means that when the receiving frame rate is around 30 FPS, the minimum output frame rate is 22.5 FPS.

[0168] When the decoding frame interval parameter is less than 60ms, the initial frame smoothing factor can be set to 3 / 2, which means that when the receiving frame rate is around 30FPS, the minimum output frame rate is 20FPS.

[0169] A larger decoding frame interval parameter indicates that the decoder needs to wait longer to receive and process the next video frame, resulting in higher network latency. By setting the corresponding frame smoothing factor according to the decoding frame interval parameter, the frame rate output is faster when the decoding frame interval parameter is larger. This improves the smoothness of the video by increasing the frame rate output and reduces the impact of video stuttering caused by network latency, thereby improving the user experience.

[0170] In step 10312, the time interval between the entry of a video frame into the buffer frame queue and the previous video frame is obtained.

[0171] Here, as mentioned above, the receiving device receives data packets, combines the data packets into video frames, and stores the video frames in a buffer. Only when the video frames satisfy the dependency relationship, that is, when the reference frame corresponding to the video frame has been received, will the video frames waiting to enter the buffer frame queue be transferred from the buffer to the frame buffer queue. In other words, there is a time interval between the video frames entering the buffer frame queue.

[0172] In step 10313, in response to a time interval less than a time interval threshold or a number of video frames greater than a first buffer frame number threshold, the initialized frame smoothing factor is adjusted according to the number of video frames, and the frame smoothing factor is determined.

[0173] In some embodiments, when the time interval is less than a time interval threshold (e.g., 10ms), or the number of video frames in the frame buffer queue is greater than a first buffer frame number threshold (e.g., 1), the initialized frame smoothing factor is adjusted according to the number of video frames to determine the frame smoothing factor. Referring to Figure 3G, the adjustment of the initialized frame smoothing factor according to the number of video frames in step 10313 shown in Figure 3F can be implemented through steps 10314 to 10315, which are described in detail below.

[0174] In step 10314, in response to the number of video frames being greater than the second buffer frame number threshold, the initialized frame smoothing factor is configured according to a fixed value to obtain the frame smoothing factor, wherein the second buffer frame number threshold is greater than the first buffer frame number threshold.

[0175] In some embodiments, when the number of video frames in the cached frame queue is greater than the second cached frame number threshold (e.g., 4), the initialized frame smoothing factor is configured with a fixed value to obtain the frame smoothing factor, for example, the frame smoothing factor is set to a fixed value of 1 / 2.

[0176] In step 10315, in response to the number of video frames being less than or equal to the second buffer frame number threshold, the initialized frame smoothing factor is inversely divided to obtain the frame smoothing factor.

[0177] In some embodiments, when the number of video frames in the cached frame queue is less than or equal to the second cached frame number threshold (e.g., 4), the initial frame smoothing factor is inversely divided to obtain the frame smoothing factor, for example, the initial frame smoothing factor 5 / 4 is inversely divided to become 4 / 5.

[0178] Referring again to Figure 3E, in step 1032A, the delay parameter is obtained through the frame smoothing factor.

[0179] In some embodiments, referring to FIG3H and FIG3E, step 1032A can be implemented by steps 1032A1 to 1032A2, which will be described in detail below.

[0180] In step 1032A1, the product of the frame smoothing factor and the decoding frame interval parameter is obtained.

[0181] In some embodiments, the statistical value of the decoding frame interval within the time period corresponding to the current video frame (such as the 95th percentile, average value, etc.) can be obtained as the decoding frame interval parameter. Here, the time period corresponding to the current video frame can be a historical time period with a preset duration from the receiving timestamp of the current video frame, or a historical time period with a preset duration from the current moment. For example, the 95th percentile of the decoding frame interval within a time period with a duration of 2 seconds from the receiving timestamp of the current video frame (the time period corresponding to the video frame) can be obtained as the decoding interval parameter. The frame smoothing factor can be obtained by referring to the frame smoothing factor determined based on the number of video frames in the buffer frame queue in step 1031A above.

[0182] In step 1032A2, the minimum value of the product and the delay control parameter is determined and the sum of the current timestamp is calculated. The difference between the sum and the first rendering timestamp is then calculated to obtain the delay parameter.

[0183] In some embodiments, the minimum value of the product and the delay control parameter (e.g., 125ms) is determined and summed with the current timestamp. The difference between the sum and the first rendering timestamp is then calculated to obtain the delay parameter.

[0184] For example, the determination of the delay parameter can be expressed as: Delay parameter = current timestamp + min(decoding frame interval parameter × frame smoothing factor, 125) - first rendering timestamp, where "min" means to obtain the minimum value between "decoding frame interval parameter × frame smoothing factor" (product) and "125" (delay control parameter).

[0185] In some embodiments, the delay parameter is less than or equal to the minimum value between the maximum jitter value and a preset threshold (e.g., 500ms), where the maximum jitter value refers to the maximum value of the reception time interval between adjacent data packets in the current first time period.

[0186] Referring again to Figure 3E, in step 1033A, the second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

[0187] In some embodiments, the first rendering timestamp and the delay parameter are summed, and the sum is used as the second rendering timestamp.

[0188] For example, the second rendering timestamp can be determined by the first rendering timestamp and the delay parameter, which can be expressed as: Second rendering timestamp = Delay parameter + First rendering timestamp.

[0189] In this embodiment, during the first time period (jitter time period), a bidirectional constraint is imposed on the decoding interval parameter by setting a first decoding frame interval parameter threshold and a second decoding frame interval parameter threshold. This not only avoids the impact of extreme interval values ​​caused by network fluctuations on frame rate stability, but also provides standardized input for the subsequent determination of the frame smoothing factor, ensuring the rationality of parameter calculation. By establishing a mapping relationship between the decoding frame interval parameter and the initialized frame smoothing factor, and dynamically adjusting the initialized frame smoothing factor according to the number of video frames in the buffer frame queue, when the number of buffered frames is greater than or equal to the second number of buffered frames (e.g., 4 frames), the frame smoothing factor is fixed to stabilize the frame output rhythm and prevent decoding overload. When the number of buffered frames is less than the second number of buffered frames, decoding is accelerated by taking the reciprocal of the frame smoothing factor, utilizing existing frame resources to fill the gaps in network fluctuations, avoiding decoder idleness, and improving the adaptability of the video frame decoding process to dynamic network jitter.

[0190] In some embodiments, referring to FIG3I, when the frame smoothing strategy is a second frame smoothing strategy used to delay video frames in the cache frame queue based on the decoding frame interval parameter during the second time period, step 103 shown in FIG3A can be implemented by steps 1031B to 1033B, which are described in detail below.

[0191] In step 1031B, in response to the slow smoothing frame number being greater than the slow smoothing frame number threshold, the slow smoothing frame number is decremented, and a frame smoothing factor is determined based on the decoding frame interval parameter. Here, the slow smoothing frame number is used to characterize the number of video frames to be processed after the video frame in the buffer frame queue. The video frames to be processed are those that need to be delayed by the second frame smoothing strategy. The slow smoothing frame number is decremented from the initial value, and the amount of decrement of the initial value is consistent with the number of video frames that are smoothed by the second frame smoothing strategy.

[0192] In some embodiments, when the number of slow-smoothing frames is greater than a slow-smoothing frame number threshold (e.g., 0), the number of slow-smoothing frames is decremented, and a frame smoothing factor is determined based on a decoding frame interval parameter. The slow-smoothing frame number characterizes the number of video frames to be processed after the current video frame in the buffer frame queue. These video frames are those that need to be delayed using the second frame smoothing strategy. The slow-smoothing frame number is decremented from an initial value, and the decrement amount is consistent with the number of video frames smoothed using the second frame smoothing strategy. In the second time period, after delaying the video frames after the current video frame using the second frame smoothing strategy, the slow-smoothing frame number is decremented once for each subsequent video frame until it equals the slow-smoothing frame number threshold. The decoding frame interval parameter is the interval between the decoding times of the decoded frames. For example, it can be a statistical representation of the decoding interval between two adjacent decoded frames (i.e., two video frames decoded sequentially) within a historical time period preset from the current time, such as an average or quantile.

[0193] For example, when the number of slow smoothing frames is greater than 0, let's assume that the number of slow smoothing frames is 3 (the number of slow smoothing frames 3 here is obtained by subtracting from the initial value. For example, if the initial value is 5, it means that 2 video frames have been smoothed before processing the current video frame, so the number of slow smoothing frames corresponding to the currently processed video frame is obtained by subtracting the initial value of 5 to 3). This means that in addition to the video frame currently being processed by the second frame smoothing strategy, the next 3 video frames after the currently processed video frame also need to be processed by the second frame smoothing strategy. When the next 3 frames are delayed by the second frame smoothing strategy, the number of slow smoothing frames is reduced by 1 after each frame is processed, until the number of slow smoothing frames returns to 0.

[0194] In other embodiments, in response to the slow smoothing frame count decreasing to a slow smoothing frame count threshold (e.g., 0), a new initial value is obtained based on the frame sorting buffer duration, wherein the new initial value is the ratio of the frame sorting buffer duration to the decoding frame interval parameter; based on the new initial value, a new slow smoothing frame count is determined; based on the new slow smoothing frame count, the steps of decreasing the slow smoothing frame count in response to the slow smoothing frame count being greater than the slow smoothing frame count threshold and determining the frame smoothing factor according to the decoding frame interval parameter are returned.

[0195] For example, when the number of slowly smoothed frames decreases to 0, a new initial value needs to be obtained based on the frame sorting buffer duration of the currently processed video frame. Assume the currently processed video frame is P3. Before P3, video frames P1 and P2 have already been smoothed. The number of slowly smoothed frames corresponding to P3 is obtained by decreasing the initial value, which is determined by the ratio of the frame sorting buffer duration corresponding to P1 to the decoding frame interval parameter. For example, the initial value for P1 is 1. Since frame P2 was produced after P1, the initial value is decremented once. When processing P3, the number of slowly smoothed frames decreases from the initial value of 1 to 0. At this point, a new initial value is obtained based on the frame sorting buffer duration corresponding to P3, thus knowing the number of frames smoothed in the current processing frame. The number of video frames that need to be processed by the second frame smoothing strategy after P3 is calculated as follows: Assuming the frame sorting wait time for video frame P3 is 200ms and the decoding frame interval parameter is 50ms, the new initial value is 4. This means that in addition to the current P3, which needs to be processed by the second frame smoothing strategy, the next 4 frames (such as P4 to P7) also need to be processed by the second frame smoothing strategy. For each frame that is smoothed out, the number of slowly smoothed frames (initial value) is decremented once. After the next 4 frames are smoothed out, the number of slowly smoothed frames returns to 0. At this time, the new initial value is obtained again based on the frame sorting buffer time and decoding frame interval parameter of the video frame being processed.

[0196] In other embodiments, in response to the slow smoothing frame count decreasing to a slow smoothing frame count threshold (e.g., 0), a new initial value is obtained based on the frame sorting buffer duration; in response to the new initial value being equal to the slow smoothing frame count threshold (i.e., the new initial value is still equal to the slow smoothing frame count threshold); the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained (see the description of step 1031C below); in response to the time difference being less than a preset proportion of the decoding frame interval parameter, a delay parameter is obtained through a fixed-value frame smoothing factor (see the description of step 1032C below); and the delay parameter is obtained through the first rendering timestamp and the delay parameter. The second rendering timestamp is determined (see step 1033C below); based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame, the decoding waiting time of the video frame is determined (see step 104 below); in response to the storage time of the video frame in the buffer frame queue reaching the decoding waiting time, the video frame is decoded (see step 105 below); in response to the time difference being greater than or equal to a preset ratio of the decoding frame interval parameter, the video frame is removed from the buffer frame queue and the video frame is decoded (see step 107 below).

[0197] In some embodiments, the frame smoothing factor can be determined based on the decoding frame interval parameter. Here, the determination of the frame smoothing factor can be referred to the description in step 10311 above, wherein the frame smoothing factor initialized in step 10311 can be used as the frame smoothing factor determined here.

[0198] Referring again to Figure 3I, in step 1032B, the delay parameter is obtained through the frame smoothing factor, where the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue.

[0199] In some embodiments, referring to FIG3J, step 1032B shown in FIG3I can be implemented by steps 1032B1 to 1032B2, which are described in detail below.

[0200] In step 1032B1, the product of the frame smoothing factor and the decoding frame interval parameter is obtained, and the product is summed with the current timestamp to obtain the summation result.

[0201] In some embodiments, the frame smoothing factor is multiplied by the decoding frame interval parameter and then added to the current timestamp to obtain the summation result.

[0202] For example, the product of the frame smoothing factor and the decoding frame interval parameter is obtained, and the product is summed with the current timestamp to obtain the summation result, which can be expressed as: summation result = frame smoothing factor × decoding frame interval parameter + current timestamp.

[0203] In step 1032B2, the summation result is subtracted from the first rendering timestamp to obtain the delay parameter.

[0204] Following the example in step 1032B1, the delay parameter can be obtained as: delay parameter = frame smoothing factor × decoding frame interval parameter + current timestamp - first rendering timestamp.

[0205] Referring again to Figure 3I, in step 1033B, the second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

[0206] Here, the implementation method for determining the second rendering timestamp can be found in the description of step 1033A above.

[0207] In some other embodiments, steps 1031B to 1032B can be implemented as processing the frame smoothing strategy of the video frames buffered during the second time period, i.e. during packet loss retransmission. For retransmission recovery frames that have introduced a large delay in the second time period (i.e., video frames obtained by reassembling frames after packet loss and retransmission), the time difference between the retransmission recovery frame and the decoding time of the previous decoding frame can be obtained. When the time difference is less than half of the decoding frame interval parameter, the process of delaying the first rendering timestamp through the third frame smoothing strategy to obtain the second rendering timestamp is initiated (see the description of steps 1032C to 1033C below). Otherwise, the frame is output in real time, i.e., the retransmission recovery frame is removed from the buffer frame queue and the retransmission recovery frame is decoded.

[0208] In the second time period (packet loss and retransmission period), this application embodiment uses a second frame smoothing strategy. Based on the decoding frame interval parameter, a frame smoothing factor is determined and combined with a slow smoothing frame count reduction mechanism. This allows video frames to be released gradually according to the number of video frames to be processed in the buffer frame queue, avoiding a sudden increase in hardware load caused by concentrated decoding of video frames. At the same time, when the number of slow smoothing frames decreases to a threshold, a new initial value is dynamically obtained based on the frame sorting buffer duration to continuously control the frame output rhythm of subsequent frames. The time difference between the retransmission recovery frame and the previous decoded frame is used to determine whether to switch strategies or output frames in real time. This effectively reduces excessive delay or timing disorder caused by packet loss and retransmission, and improves the dynamic adaptability of the video frame decoding process to buffer fluctuations and retransmission data in weak network environments.

[0209] In some embodiments, referring to FIG3K, when the frame smoothing strategy is a third frame smoothing strategy for delaying a specific video frame in the cache frame queue based on a fixed value of the frame smoothing factor during the third time period, step 103 shown in FIG3A can be implemented by steps 1031C to 1033C, which are described in detail below.

[0210] In step 1031C, the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained.

[0211] In some embodiments, the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained. Here, the decoding time of the previous decoded frame can be the time when the decoding of the decoded frame is completed or the time when the decoding of the decoded frame begins.

[0212] In other embodiments, the time interval between the current timestamp and the decoding time of the previous decoded frame can also be obtained as the time difference.

[0213] In step 1032C, in response to a decoding frame interval parameter where the time difference is less than a preset ratio, a delay parameter is obtained through a fixed-value frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue.

[0214] In some embodiments, when the time difference is less than a preset ratio (e.g., one-half) of the decoding frame interval parameter, the delay parameter is obtained by a fixed value frame smoothing factor, for example, the fixed value frame smoothing factor can be set to 1 / 2.

[0215] For example, when the time difference is greater than or equal to the preset ratio of the decoding frame interval parameter, the video frame is output in real time.

[0216] In step 1033C, the second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

[0217] Here, the implementation method for determining the second rendering timestamp can be found in the description of step 1033A above.

[0218] This application embodiment dynamically switches frame smoothing strategies for different network scenarios (the network scenarios corresponding to the first time period, the second time period, and the third time period, respectively), avoiding the limitations of a single frame smoothing strategy in complex network environments. This improves the global adaptability of the decoding process to dynamic network states, achieving the beneficial effect of stably controlling the decoding time of video frames and improving the smoothness of video frame decoding and rendering.

[0219] Referring again to Figure 3A, in step 104, the decoding waiting time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame.

[0220] In some embodiments, referring to FIG3L, step 104 shown in FIG3A can be implemented by steps 1041 to 1042, which are described in detail below.

[0221] In step 1041, the sum of the received timestamp of the video frame and the decoding time parameter is obtained.

[0222] In some embodiments, the sum of the receiving timestamp and decoding time parameter of the video frame is obtained. The decoding time parameter can be determined by statistically analyzing the decoding time of decoded video frames within a preset time period (e.g., the first 2 seconds of the video frame), such as the 95th percentile or average of the decoding time of decoded video frames within the preset time period. Here, the 95th percentile of the decoding time of a video frame is a statistical indicator used to describe the distribution of the time required to decode a video frame over a period of time. The 95th percentile of the decoding time of a video frame means that within the preset time period, 95% of the video frames have a decoding time less than or equal to this value.

[0223] In step 1042, the difference between the second rendering timestamp and the sum is used as the decoding wait time for the video frame.

[0224] For example, the determination of the decoding wait time for a video frame can be expressed as: Decoding wait time = Second rendering timestamp - Receive timestamp - Decoding time parameter.

[0225] Referring again to Figure 3A, in step 105, the video frame is decoded in response to the video frame being stored in the buffer frame queue for a certain period of time and the decoding waiting time being reached.

[0226] In some embodiments, when the storage time of a video frame in the buffer frame queue reaches the decoding waiting time, the video frame is taken out of the buffer frame queue and sent to the decoder, and the decoder is called to decode the video frame.

[0227] In some other embodiments, referring to FIG3M, after step 105 shown in FIG3A, steps 106 to 107 may also be performed, which are described in detail below.

[0228] In step 106, the time difference between the current timestamp and the decoding time of the previous decoding frame is obtained.

[0229] Here, the decoding time of the previous decoding frame can be either the time when the decoding of the previous frame is completed or the time when the decoding of the previous frame begins.

[0230] In other embodiments, the time interval between the received timestamp of a video frame and the decoding time of the previous decoded frame can also be obtained as a time difference.

[0231] In step 107, in response to a time difference greater than or equal to a preset ratio of the decoding frame interval parameter, a video frame is removed from the buffer frame queue and the video frame is decoded.

[0232] In some embodiments, when the time difference is greater than or equal to a preset ratio (e.g., one-half) of the decoding frame interval parameter, a video frame is removed from the buffer frame queue (real-time frame output) and the video frame is decoded.

[0233] Steps 101 to 107 achieve the selection of a corresponding frame smoothing strategy based on the specific situation of video frames in the cached frame queue (such as the number of video frames, the time period of the current video frame, etc.). This results in different degrees of delay for the first rendering timestamp of different video frames, avoiding the introduction of excessive decoding waiting time in the future. At the same time, the frame smoothing strategy is decided based on the state of the video frame and is not affected by business scenario-related parameters (such as video encoding algorithm strategy, network transmission algorithm strategy, flow control algorithm strategy, and other business scenario-related parameters). This improves the accuracy of obtaining the decoding waiting time of video frames, achieving the beneficial effect of stably controlling the decoding time of video frames and improving the smoothness of video frame decoding and rendering.

[0234] The following will describe an exemplary application of the embodiments of this application in a video call application scenario.

[0235] When making video calls using mobile devices, the video call process is easily affected by the network transmission environment. For example, in an environment with unstable network transmission, various problems may occur in video playback, such as video stuttering, which seriously affects the video call quality.

[0236] Referring to Figure 5, which is a flowchart illustrating the video frame processing method in a video call scenario provided in this application embodiment, the following description focuses on the receiving device (such as terminal device 200-2) in the video call.

[0237] Referring to Figure 6A, which is a schematic diagram of the principle of video frame processing provided in the embodiment of this application, while performing prediction processing based on the acquisition timestamp of the video frame to obtain the first rendering timestamp of the video frame, the frame smoothing strategy and frame smoothing factor of the video frame are determined by judging the current time period (corresponding to the first time period, second time period and third time period mentioned above), as well as the number of video frames in the current buffer frame queue and the decoding frame interval parameter, thereby determining the delay parameter to delay the first rendering timestamp to obtain the second rendering timestamp, and the decoding waiting time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame and the decoding time parameter.

[0238] In step 201, data packets are received, data packets are combined into video frames, and in response to the video frames satisfying the reference relationship, the video frames are stored in the buffer frame queue.

[0239] In some embodiments, referring to FIG6B, FIG6B is a schematic diagram of the principle of storing video frames into a buffer frame queue provided in the embodiments of this application. After receiving data packets, the receiving device performs frame assembly, puts the video frames obtained by frame assembly into a buffer, and sorts them through a frame sorting module. When the video frames meet the reference relationship (as shown in FIG6A, the I0 frame reference frame is successfully received), the video frames are stored into the buffer frame queue for decoding processing of each subsequent video frame.

[0240] For example, in a video call scenario, audio data and video frame data can be transmitted independently via protocols such as Real-time Transport Protocol (RTP) or Hypertext Transfer Protocol (HTTP). Video frame data can be processed using the video frame processing method shown in Figure 5, providing a visual experience for the user on the receiving device. Audio data can be captured by the microphone of the sending device, removing noise and unnecessary high- or low-frequency components, detecting silent or low-energy parts, and reducing or eliminating these parts. After preprocessing, the sending device's codec converts the analog or digital audio signal into a digital format and compresses it to reduce data size. Next, the encoded audio data is sent to the receiving end using a real-time transport protocol (such as RTP). The receiving end uses the same codec to decode the received encoded audio data back into the original audio signal, which is then played through a speaker, providing an auditory experience for the user on the receiving device.

[0241] In online video playback scenarios, audio data and video frame data can be encapsulated in the same container format for transmission, such as MP4, FLV, and MKV. In this case, audio data and video frame data are encapsulated together at the receiving end and during transmission. However, at the receiving end, the audio data and video frame data can be decapsulated and processed separately, such as decoding and rendering video frames, decoding audio data, and performing post-processing such as echo cancellation and noise suppression.

[0242] This application does not limit the transmission method of audio data and video frame data. Regardless of the transmission method used, audio data and video frame data can be encoded at the sending device and decoded at the receiving device to ensure reliable and efficient data transmission. Furthermore, to handle network jitter and latency, techniques such as buffering, packet loss recovery, and forward error correction can be employed.

[0243] In step 202, prediction processing is performed based on the acquisition timestamp of the video frame to obtain the first rendering timestamp of the video frame.

[0244] In some embodiments, the implementation of obtaining the first rendering timestamp (i.e., the estimated rendering timestamp) of the video frame can be found in the description of step 101 above, and will not be repeated here.

[0245] In step 203, a frame smoothing strategy is determined based on the video frames in the cached frame queue.

[0246] In some embodiments, referring to Figure 6C, which is a flowchart of the frame smoothing strategy provided in this application embodiment, it is first determined whether the current time period is the first time period (i.e., the jitter time period). If the current time period is the first time period, the process proceeds to delay the first rendering timestamp using the first frame smoothing strategy to obtain the second rendering timestamp. Otherwise, it is determined whether the current time period is the second time period (i.e., the packet loss retransmission time period). If the current time period is the second time period, the process proceeds to delay the first rendering timestamp using the second frame smoothing strategy to obtain the second rendering timestamp. If the current time period is neither the first nor the second time period (corresponding to the third time period mentioned above), the time difference between the current timestamp and the decoding time of the previous decoded frame is obtained. When the time difference is less than half of the decoding frame interval parameter, the process proceeds to delay the first rendering timestamp using the third frame smoothing strategy to obtain the second rendering timestamp. Otherwise, the frame is output in real time, that is, the video frame is removed from the buffer frame queue and the video frame is decoded.

[0247] Among them, for multiple data packets received within the current time window according to a preset duration (e.g., 2s), if the time interval between the reception of all or more of the adjacent data packets exceeds the time interval threshold (e.g., 100ms), then the current time is determined to be in the first time period.

[0248] For multiple data packets received within a preset duration in the current time window, if the receiving time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period), it is determined whether there is packet loss (e.g., whether packet loss occurred within the preset duration). If there is packet loss, the current time is in the second time period.

[0249] In step 204, the first rendering timestamp is delayed based on the frame smoothing strategy to obtain the second rendering timestamp.

[0250] Here, the implementation method for delaying the first rendering timestamp using the first frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031A to 1033A above; the implementation method for delaying the first rendering timestamp using the second frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031B to 1033B above; the implementation method for delaying the first rendering timestamp using the third frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031C to 1033C above, and will not be repeated here.

[0251] In step 205, the decoding wait time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter.

[0252] In some embodiments, the implementation of determining the decoding wait time of a video frame can be found in the description of step 104 above, and will not be repeated here.

[0253] In step 206, the video frame is decoded in response to the video frame being stored in the buffer frame queue for a certain period of time and the decoding waiting time being reached.

[0254] In some embodiments, when the storage time of a video frame in the buffer frame queue reaches the decoding waiting time, the video frame is taken out of the buffer frame queue and sent to the decoder. The decoder is then called to decode and render the video frame, so that the video frame is displayed on the receiving device.

[0255] Steps 201 to 206 achieve the selection of a corresponding frame smoothing strategy based on the specific situation of video frames in the cached frame queue (such as the number of video frames, the time period of the current video frame, etc.). This results in different degrees of delay for the first rendering timestamp of different video frames, minimizing the introduction of decoding waiting time and stably controlling the decoding time of video frames. This achieves the beneficial effects of reducing the stuttering rate of user video calls, improving the smoothness of video calls, and enhancing the user experience.

[0256] The video frame processing method provided in this application reduces the stuttering rate of online video calls from 5.2% to 3.2%, a reduction of 38%. This improves the smoothness of real-time video calls for users in weak network scenarios.

[0257] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0258] The following continues to describe an exemplary structure of the video frame processing device 455 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2A, the video frame processing device 455 stored in the memory 450-1 may include:

[0259] The prediction processing module 4551 is configured to predict the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue.

[0260] The frame smoothing processing module 4552 is configured to determine a frame smoothing strategy based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames.

[0261] In some embodiments, the frame smoothing processing module 4552 is further configured to determine the delay parameter based on the frame smoothing strategy, and delay the first rendering timestamp based on the delay parameter to obtain a second rendering timestamp.

[0262] In some embodiments, the frame smoothing processing module 4552 is further configured to determine the decoding waiting time of the video frame based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame.

[0263] The frame output module 4553 is configured to decode the video frame in response to the video frame being stored in the cached frame queue for a duration equal to the decoding waiting time.

[0264] In some embodiments, the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval between adjacent data packets is greater than a time interval threshold, determine that the current moment is within a first time period, and determine to apply a first frame smoothing strategy to the buffered frame queue before the next slide of the time window, wherein the starting point of the time window is a historical moment at a preset time distance from the current moment, the ending point of the time window is the current moment, and the first frame smoothing strategy is used to delay the video frames within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the buffered frame queue.

[0265] In some embodiments, the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, determine that the current time is in a second time period, and determine to apply a second frame smoothing strategy to the buffered frame queue before the next slide of the time window, wherein the starting point of the time window is a historical time that is a preset time interval away from the current time, the ending point of the time window is the current time, and the second frame smoothing strategy is used to delay the video frames in the buffered frame queue based on the decoding frame interval parameter within the second time period.

[0266] In some embodiments, the frame smoothing processing module 4552 is further configured to determine that the current time is in a second time period if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, and at least one data packet is lost during transmission within the current time window.

[0267] In some embodiments, the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, determine that the current time is in a third time period, and determine to apply a third frame smoothing strategy to the buffer frame queue before the next slide of the time window, wherein the starting point of the time window is a historical time that is a preset time interval away from the current time, the ending point of the time window is the current time, and the third frame smoothing strategy is used to delay a specific video frame in the buffer frame queue based on a fixed frame smoothing factor within the third time period, wherein the time difference between the reception timestamp of the specific video frame and the decoding time of the previous decoded frame is less than a preset proportion of the decoding frame interval parameter.

[0268] In some embodiments, the frame smoothing processing module 4552 is further configured to determine that the current time is in the third time period if the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and the data packets are not lost during transmission within the current time window.

[0269] In some embodiments, when the frame smoothing strategy is a first frame smoothing strategy used to delay the video frames within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the cached frame queue, the frame smoothing processing module 4552 is further configured to determine the frame smoothing factor based on the number of video frames in the cached frame queue, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; obtain the delay parameter through the frame smoothing factor; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter.

[0270] In some embodiments, the frame smoothing processing module 4552 is further configured to determine an initialized frame smoothing factor based on a decoded frame interval parameter; obtain the time interval between the video frame and the previous video frame entering the buffer frame queue; and adjust the initialized frame smoothing factor based on the number of video frames in response to the time interval being less than a time interval threshold or the number of video frames being greater than a first buffer frame number threshold, thereby determining the frame smoothing factor.

[0271] In some embodiments, the frame smoothing processing module 4552 is further configured to, in response to the number of video frames being greater than a second cached frame number threshold, configure the initialized frame smoothing factor according to a fixed value to obtain the frame smoothing factor, wherein the second cached frame number threshold is greater than the first cached frame number threshold; and in response to the number of video frames being less than or equal to the second cached frame number threshold, perform reciprocal processing on the initialized frame smoothing factor to obtain the frame smoothing factor.

[0272] In some embodiments, the frame smoothing processing module 4552 is further configured to obtain the product of the frame smoothing factor and the decoding frame interval parameter; determine the sum of the minimum value of the product and the delay control parameter with the current timestamp; and calculate the difference between the sum and the first rendering timestamp to obtain the delay parameter.

[0273] In some embodiments, when the frame smoothing strategy is a second frame smoothing strategy used to delay video frames in the cached frame queue based on the decoding frame interval parameter during the second time period, the frame smoothing processing module 4552 is further configured to, in response to the slow smoothing frame number being greater than the slow smoothing frame number threshold, decrease the slow smoothing frame number and determine a frame smoothing factor according to the decoding frame interval parameter, wherein the slow smoothing frame number is used to characterize the number of video frames to be processed after the video frame in the cached frame queue, the video frames to be processed are video frames that need to be delayed by the second frame smoothing strategy, the slow smoothing frame number is decreased from an initial value, and the decrease in the initial value is consistent with the number of video frames smoothed by the second frame smoothing strategy; obtain the delay parameter through the frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter.

[0274] In some embodiments, the frame smoothing processing module 4552 is further configured to: in response to the slow smoothing frame number decreasing to the slow smoothing frame number threshold, obtain a new initial value based on the frame sorting buffer duration, wherein the new initial value is the ratio of the frame sorting buffer duration to the decoding frame interval parameter; determine a new slow smoothing frame number based on the new initial value; return the steps of decreasing the slow smoothing frame number in response to the slow smoothing frame number being greater than the slow smoothing frame number threshold, and determining a frame smoothing factor according to the decoding frame interval parameter.

[0275] In some embodiments, the frame smoothing processing module 4552 is further configured to obtain the product of the frame smoothing factor and the decoding frame interval parameter, sum the product with the current timestamp to obtain a summation result, and subtract the summation result from the first rendering timestamp to obtain the delay parameter.

[0276] In some embodiments, when the frame smoothing strategy is a third frame smoothing strategy used to delay specific video frames in the cached frame queue based on a fixed value frame smoothing factor during the third time period, the frame smoothing processing module 4552 is further configured to: obtain the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame; in response to the time difference being less than a preset proportion of the decoding frame interval parameter, obtain the delay parameter through the fixed value frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter.

[0277] In some embodiments, the frame output module 4553 is further configured to obtain the time difference between the current timestamp and the decoding time of the previous decoded frame; in response to the time difference being greater than or equal to a preset proportion of the decoding frame interval parameter, remove the video frame from the buffer frame queue and decode the video frame.

[0278] In some embodiments, the prediction processing module 4551 is further configured to obtain the received frame rate level of the video frame; obtain the received time difference of the video frame; and determine the first rendering timestamp of the video frame based on the received frame rate level, the received time difference, and the acquisition timestamp.

[0279] In some embodiments, the prediction processing module 4551 is further configured to obtain the video frame receiving frame rate within a period corresponding to the video frame, map the video frame receiving frame rate to a plurality of preset frame rate levels, and use the mapped frame rate level as the receiving frame rate level.

[0280] In some embodiments, the prediction processing module 4551 is further configured to obtain the difference between the reception timestamp of a plurality of video frames and the sum of the acquisition timestamp, round-trip delay and transmission time of the video frames; and to use the average value of the differences corresponding to the plurality of video frames as the reception time difference.

[0281] In some embodiments, the prediction processing module 4551 is further configured to obtain the reciprocal of the received frame rate level at a preset multiple; obtain the sum between the acquisition timestamp and the reciprocal; and perform summation processing on the sum and the received time difference to obtain the first rendering timestamp.

[0282] In some embodiments, the frame smoothing processing module 4552 is further configured to obtain the sum of the receiving timestamp of the video frame and the decoding time parameter; and to use the difference between the second rendering timestamp and the sum as the decoding waiting time of the video frame.

[0283] The following continues to describe an exemplary structure of the video frame processing device 456 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2B, the video frame processing device 456 stored in the memory 450-2 may include:

[0284] The processing module 4561 is configured to acquire a video frame to be encoded and encode the video frame to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.

[0285] This application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video frame processing method described in this application.

[0286] This application provides a computer storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the video frame processing method provided in this application, for example, the video frame processing method shown in any of the figures 3A to 3M.

[0287] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0288] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0289] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0290] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0291] In summary, through the embodiments of this application, by using the temporal stability of the acquisition timestamp as a benchmark, the first rendering timestamp of the video frame is obtained, providing a reliable time reference benchmark for the subsequent calculation of decoding waiting time. By obtaining the corresponding frame smoothing strategy based on the video frames in the cache frame queue, an adaptive decision-making process for the frame smoothing strategy is realized. Through the frame smoothing strategy and the first rendering timestamp, the second rendering timestamp of the video frame is determined, realizing the selection of the corresponding frame smoothing strategy according to the specific situation of the video frames in the cache frame queue. This allows for different degrees of delay for the first rendering timestamp of different video frames, avoiding the accumulation of decoding waiting time caused by a single frame smoothing strategy and the introduction of excessive decoding waiting time. Finally, by using the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame, the decoding waiting time of the video frame is quantified, thereby improving the matching degree between the decoding waiting time of the video frame in the cache frame queue and the actual decoding requirements. In conclusion, by making smoothing strategy decisions based on the state of the video frames in the cache frame queue, the accuracy of obtaining the decoding waiting time of the video frame is improved, achieving the beneficial effect of stably controlling the decoding time of the video frame and improving the smoothness of video frame decoding and rendering.

[0292] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for processing video frames, applied to an electronic device, comprising: Based on the acquisition timestamps of video frames in the cached frame queue, predict the first rendering timestamp of the video frame; A frame smoothing strategy is determined based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames; The delay parameter is determined based on the frame smoothing strategy, and the first rendering timestamp is delayed based on the delay parameter to obtain the second rendering timestamp; Based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame, the decoding waiting time of the video frame is determined; In response to the video frame being stored in the cached frame queue for a period of time that reaches the decoding waiting time, the video frame is decoded.

2. The method according to claim 1, wherein, The video frames are obtained by combining received data packets. Determining a frame smoothing strategy based on the video frames in the buffered frame queue includes: For multiple data packets received within the current time window, if the time interval between adjacent data packets is greater than a time interval threshold, then the current time is determined to be within a first time period. Before the next slide of the time window, a first frame smoothing strategy is applied to the buffered frame queue. The starting point of the time window is a historical time at a preset time interval from the current time, and the ending point of the time window is the current time. The first frame smoothing strategy is used to delay the video frames within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the buffered frame queue.

3. The method according to claim 1 or 2, wherein, The step of determining a frame smoothing strategy based on the video frames in the cached frame queue includes: For multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, then the current time is determined to be within a second time period, and a second frame smoothing strategy is applied to the buffer frame queue before the next slide of the time window. The starting point of the time window is a historical time that is a preset time interval away from the current time, and the ending point of the time window is the current time. The second frame smoothing strategy is used to delay the video frames in the buffer frame queue based on the decoding frame interval parameter within the second time period.

4. The method according to claim 3, wherein, If the reception time interval of adjacent data packets is less than or equal to the time interval threshold, then determining that the current time is in the second time period includes: If the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and at least one data packet is lost during transmission within the current time window, then the current time is determined to be in the second time period.

5. The method according to claim 1, 2 or 4, wherein, The step of determining a frame smoothing strategy based on the video frames in the cached frame queue includes: For multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, then the current time is determined to be within a third time period. Furthermore, a third frame smoothing strategy is applied to the buffered frame queue before the next slide of the time window. The starting point of the time window is a historical time point a preset time interval from the current time, and the ending point of the time window is the current time. The third frame smoothing strategy is used to delay specific video frames in the buffered frame queue within the third time period based on a fixed frame smoothing factor. The time difference between the reception timestamp of the specific video frame and the decoding time of the previous decoded frame is less than a preset proportion of the decoding frame interval parameter.

6. The method according to claim 5, wherein, If the reception time interval of adjacent data packets is less than or equal to the time interval threshold, then the current time is determined to be in the third time period, including: If the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and the data packets are not lost during transmission within the current time window, then the current time is determined to be in the third time period.

7. The method according to any one of claims 1 to 6, wherein, When the frame smoothing strategy is a first frame smoothing strategy used to delay the video frames within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the cached frame queue, the step of determining the delay parameter based on the frame smoothing strategy and delaying the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp includes: The frame smoothing factor is determined based on the number of video frames in the buffer frame queue, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffer frame queue; The delay parameter is obtained through the frame smoothing factor; The second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

8. The method according to claim 7, wherein, Determining the frame smoothing factor based on the number of video frames in the cached frame queue includes: The initial frame smoothing factor is determined based on the decoding frame interval parameter; Obtain the time interval between the entry of the video frame and the previous video frame into the buffer frame queue; In response to the time interval being less than a time interval threshold, or the number of video frames being greater than a first buffer frame number threshold, the initialized frame smoothing factor is adjusted according to the number of video frames, and the frame smoothing factor is determined.

9. The method according to claim 8, wherein, The step of adjusting the initialized frame smoothing factor according to the number of video frames, and determining the frame smoothing factor, includes: In response to the number of video frames being greater than the second buffer frame number threshold, the initialized frame smoothing factor is configured with a fixed value to obtain the frame smoothing factor, wherein the second buffer frame number threshold is greater than the first buffer frame number threshold; In response to the number of video frames being less than or equal to the second buffer frame number threshold, the inverse of the initialized frame smoothing factor is taken to obtain the frame smoothing factor.

10. The method according to any one of claims 7 to 9, wherein, The step of obtaining the delay parameter through the frame smoothing factor includes: Obtain the product of the frame smoothing factor and the decoding frame interval parameter; The minimum value of the product and the delay control parameter is summed with the current timestamp, and the difference between the sum and the first rendering timestamp is calculated to obtain the delay parameter.

11. The method according to any one of claims 1 to 10, wherein, When the frame smoothing strategy is a second frame smoothing strategy used to delay video frames in the cached frame queue based on the decoding frame interval parameter during the second time period, the step of determining the delay parameter based on the frame smoothing strategy and delaying the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp includes: In response to the number of slow smoothing frames exceeding a slow smoothing frame threshold, the number of slow smoothing frames is decreased, and a frame smoothing factor is determined based on the decoding frame interval parameter. The number of slow smoothing frames is used to characterize the number of video frames to be processed after the video frame in the buffer frame queue. The video frames to be processed are video frames that need to be delayed by the second frame smoothing strategy. The number of slow smoothing frames is decreased from an initial value, and the decrease in the initial value is consistent with the number of video frames smoothed by the second frame smoothing strategy. The delay parameter is obtained through the frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue; The second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

12. The method according to claim 11, wherein, The method further includes: In response to the slow smoothing frame count decreasing to the slow smoothing frame count threshold, a new initial value is obtained based on the frame sorting buffer duration, wherein the new initial value is the ratio of the frame sorting buffer duration to the decoding frame interval parameter; Based on the new initial value, a new number of slow smoothing frames is determined, and based on the new number of slow smoothing frames, the response that the number of slow smoothing frames is greater than the number of slow smoothing frames threshold is returned, the number of slow smoothing frames is decremented, and a frame smoothing factor is determined according to the decoding frame interval parameter.

13. The method according to claim 11 or 12, wherein, The step of obtaining the delay parameter through the frame smoothing factor includes: Obtain the product of the frame smoothing factor and the decoding frame interval parameter, and sum the product with the current timestamp to obtain the summation result; The difference between the summation result and the first rendering timestamp is calculated to obtain the delay parameter.

14. The method according to any one of claims 1 to 13, wherein, When the frame smoothing strategy is a third frame smoothing strategy used to delay specific video frames in the cached frame queue based on a fixed value of the frame smoothing factor during the third time period, the step of determining the delay parameter based on the frame smoothing strategy and delaying the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp includes: Obtain the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame; In response to the decoding frame interval parameter where the time difference is less than a preset ratio, the delay parameter is obtained through the fixed value of the frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue. The second rendering timestamp is determined using the first rendering timestamp and the delay parameter.

15. The method according to any one of claims 1 to 14, wherein, The method further includes: Get the time difference between the current timestamp and the decoding time of the previous decoded frame; In response to a time difference greater than or equal to a preset ratio of the decoding frame interval parameter, the video frame is removed from the buffer frame queue and the video frame is decoded.

16. The method according to any one of claims 1 to 15, wherein, The prediction of the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue includes: Obtain the received frame rate level of the video frame; Obtain the time difference of the received video frames; The first rendering timestamp of the video frame is determined based on the received frame rate level, the received time difference, and the acquisition timestamp.

17. The method according to claim 16, wherein, Determining the first rendering timestamp of the video frame based on the received frame rate level, the received time difference, and the acquisition timestamp includes: Obtain the reciprocal of the received frame rate level, which is a preset multiple; Obtain the sum between the collection timestamp and the reciprocal; The summation of the summation and the receiving time difference is performed to obtain the first rendering timestamp.

18. The method according to claim 16 or 17, wherein, The step of obtaining the received frame rate level of the video frame includes: Obtain the video frame receiving frame rate within the corresponding period of the video frame, map the video frame receiving frame rate to multiple preset frame rate levels, and use the mapped frame rate level as the receiving frame rate level.

19. The method according to claim 16 or 17, wherein, The step of obtaining the reception time difference of the video frames includes: Obtain the difference between the received timestamps of multiple video frames and the sum of the acquired timestamps, round-trip delays, and transmission times of the video frames; The average of the differences corresponding to multiple video frames is taken as the reception time difference.

20. The method according to any one of claims 1 to 19, wherein, The step of determining the decoding wait time of the video frame based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame includes: Obtain the sum of the received timestamp of the video frame and the decoding time parameter; The difference between the second rendering timestamp and the sum is used as the decoding wait time for the video frame.

21. A method for processing video frames, applied to an electronic device, comprising: A video frame to be encoded is acquired, and the video frame to be encoded is encoded to obtain an encoded video frame, wherein the encoded video frame is decoded by the method described in any one of claims 1 to 20.

22. A method for processing audio / video bitstreams, applied to electronic devices, wherein, The audio / video stream The video frame is generated by the video frame processing method according to claim 21, or decoded based on the video frame processing method according to any one of claims 1 to 20.

23. A video frame processing apparatus, comprising: The prediction processing module is configured to predict the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue. A frame smoothing processing module is configured to determine a frame smoothing strategy based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames; The frame smoothing processing module is further configured to determine the delay parameter based on the frame smoothing strategy, and delay the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp; The frame smoothing processing module is further configured to determine the decoding waiting time of the video frame based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame; The frame output module is configured to decode the video frame in response to the video frame being stored in the cached frame queue for a period of time equal to the decoding waiting time.

24. An electronic device, comprising: Memory, configured to store computer-executable instructions; When a processor is configured to execute computer-executable instructions or computer programs stored in the memory, it implements the video frame processing method according to any one of claims 1 to 20, or the video frame processing method according to claim 21.

25. A computer storage medium storing executable instructions, wherein when executed by a processor, the computer executable instructions implement the video frame processing method according to any one of claims 1 to 20, or implement the video frame processing method according to claim 21.

26. A computer program product comprising a computer program or computer-executable instructions, wherein when the computer program or computer-executable instructions are executed by a processor, the video frame processing method of any one of claims 1 to 20, or the video frame processing method of claim 21, is implemented.

Images