Frame rate determination method, device and system, equipment, storage medium and program product

Abstract

The invention provides a frame rate determination method, device, system and equipment, a storage medium and a program product, the method is applied to a coding end, and the coding end is used for coding a collected video frame according to a current frame rate and then sending the coded video frame to a decoding end; the method comprises the following steps: if a collected current video frame is coded, recording a coding completion moment of the current video frame; determining the current feedback time delay of the decoding end according to the coding completion moment and the receiving moment of the feedback signaling returned to the decoding end; the feedback signaling represents that the decoding end successfully decodes the current video frame; determining a safe frame rate according to the current feedback time delay and a preset maximum video frame number which can be stored in a reference frame buffer area at the coding end; the security frame rate is used for limiting the number of video frames sent to the decoding end by the encoding end within the current feedback time delay to be not greater than the preset maximum number of video frames; and setting the current frame rate so that the current frame rate is not greater than the safe frame rate.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of terminal technology, and in particular to a frame rate determination method, apparatus, system, device, storage medium, and program product. Background Technology

[0002] In scenarios such as real-time audio and video communication, Long-Term Reference (LTR) frame technology uses acknowledged decoded past frames as references to achieve efficient error recovery in unstable network environments, avoiding the transmission of critical frames with large amounts of data. This mechanism relies on acknowledgment feedback sent from the decoder to the encoder. However, in network environments with high round-trip latency, this feedback signaling will introduce significant delays.

[0003] The LTR reference frame buffer at the encoder has a limited capacity. During the feedback delay, if the encoder continuously sends new frames at a high frame rate, the number of new frames can easily exceed the buffer capacity, causing LTR candidate frames to be "squeezed out" of the buffer before feedback arrives, resulting in an "LTR chain break." Once the LTR mechanism fails, when packet loss occurs and error recovery is needed, the encoder will be forced to generate keyframes. The burst of keyframe data can cause network congestion and bitrate fluctuations, ultimately leading to noticeable stuttering in the video at the decoder. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a frame rate determination method is provided, the method being applied at an encoding end, the encoding end being configured to encode acquired video frames according to the current frame rate and then send them to a decoding end; the method includes: If the current video frame being captured has been encoded, record the time when the encoding of the current video frame has been completed; The current feedback delay of the decoding end is determined based on the encoding completion time and the reception time of the feedback signal returned by the decoding end; wherein the feedback signal indicates that the decoding end has successfully decoded the current video frame; A safe frame rate is determined based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store; wherein, the safe frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to not exceed the preset maximum number of video frames; Set the current frame rate so that it is not greater than the safe frame rate.

[0005] According to a second aspect of one or more embodiments of this specification, a frame rate determination apparatus is provided, the apparatus being applied at an encoding end, the encoding end being configured to encode acquired video frames according to the current frame rate and then send them to a decoding end; the apparatus includes: The recording module is used to: record the time when the encoding of the current video frame is completed if the current video frame is encoded. The latency statistics module is used to: determine the current feedback latency of the decoding end based on the encoding completion time and the reception time of the feedback signal returned by the decoding end; wherein the feedback signal indicates that the decoding end has successfully decoded the current video frame; A frame rate decision module is used to: determine a safe frame rate based on the current feedback delay and the preset maximum number of video frames that can be stored in the reference frame buffer at the encoding end; wherein the safe frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to not exceed the preset maximum number of video frames; A frame rate control module is used to: set the current frame rate so that it is not greater than the safe frame rate.

[0006] According to a third aspect of one or more embodiments of this specification, a video processing system is provided, the system including an encoding end and a decoding end, the encoding end being configured to perform the steps of the method described in the first aspect.

[0007] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first aspect by executing the executable instructions.

[0008] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] According to a sixth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] As can be seen from the above embodiments, the embodiments of this specification can accurately perceive the changes in network round-trip time (RTT) by "recording the encoding completion time and the reception time to determine the current feedback delay". Then, by "determining the safe frame rate based on the current feedback delay and the preset maximum number of video frames" and "setting the current frame rate not to be greater than the safe frame rate", a forward-looking protection mechanism is established. This ensures that during the current feedback delay from the encoding of any video frame to its confirmation feedback, the total number of newly sent video frames at the encoding end will not exceed the capacity of the reference frame buffer (i.e., the preset maximum number of video frames). This allows the earliest sent candidate frame to be marked as LTR to remain in the buffer and not be squeezed out by subsequent new frames when its feedback signaling arrives, thereby ensuring the successful execution of the LTR marking operation and maintaining the continuity of the LTR chain.

[0011] Thus, by ensuring the continued effectiveness of the LTR mechanism, when packet loss occurs in the network, the decoding end can reliably use LTR frames for error recovery, which greatly eliminates the need to generate key frames (I-frames) due to LTR failure, avoiding network congestion and drastic bit rate fluctuations caused by sending key frames. Attached Figure Description

[0012] Figure 1A This is a schematic diagram of a video processing system provided in an exemplary embodiment.

[0013] Figure 1B This is an exemplary embodiment of a frame rate determination method and its application scenario diagram.

[0014] Figure 2 This is a flowchart illustrating a frame rate determination method in an exemplary embodiment.

[0015] Figure 3A This is a schematic diagram of a video processing system provided in an exemplary embodiment.

[0016] Figure 3B This is a schematic diagram illustrating an application scenario of another frame rate determination method provided by an exemplary embodiment.

[0017] Figure 4 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0018] Figure 5 This is a block diagram of a frame rate determination apparatus provided in an exemplary embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0020] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0021] First, the terminology used in the embodiments of this specification will be explained: RTC: Real-Time Communication.

[0022] LTR: Long-Term Reference. A video coding technique that allows the use of earlier, high-quality frames as long-term references for efficient error recovery.

[0023] RTT: Round-Trip Time. The time it takes for a data packet to travel from the sender to the receiver and back.

[0024] I-frame / keyframe: Intra-coded frame. A complete image that does not depend on any other frame; it contains a large amount of data.

[0025] QP: Quantization Parameter. A key parameter in video coding; a higher value indicates greater compression but lower image quality.

[0026] DPB: Decoded Picture Buffer, reference frame buffer.

[0027] ltr_dpb_size: refers to the effective size of the reference frame buffer that the encoder can use for the LTR.

[0028] like Figure 1AThe diagram shown is a schematic representation of a video processing system according to an exemplary embodiment of this specification. The video processing system of this embodiment includes an encoding end 11 and a decoding end 12. As an example, the encoding end can be the initiator and sender of the video stream, capable of compressing and encoding the acquired raw video data (containing individual video frames) and sending it to the decoding end via network 13. As an example, the decoding end can be the receiver and restorer of the video stream, capable of real-time decoding the received encoded video stream to present the video image.

[0029] As an example, Figure 1A The “network” in this context is a logical communication channel, and its specific physical and logical topology can be implemented in various ways. For example, the server is an optional, non-essential network node.

[0030] For example, in one implementation, it can operate without server intervention. The encoding and decoding ends communicate directly via a point-to-point connection established over an IP network. For instance, in a video call application within a local area network, or between two devices on the public network with public IP addresses, video streams can be directly transmitted after establishing a session via a direct connection protocol.

[0031] Alternatively, in other implementations, the video processing system may incorporate one or more servers. As an example, the server may assist the encoder and decoder in discovery, authentication, and session establishment, while the encoder and decoder may still choose to connect directly after the media stream channel is established. Alternatively, in cases where the encoder and decoder cannot communicate directly due to factors such as network address translation or firewall restrictions, the server may act as an intermediary, receiving the video stream from the encoder and forwarding it to one or more decoders.

[0032] In real-time audio and video communication (RTC) applications, to ensure video continuity and clarity in unstable network environments, modern video coding standards (such as H.264 / AVC, H.265 / HEVC, AV1, etc.) have generally introduced Long-Term Reference (LTR) frame technology. An LTR frame is a video frame that both communicating parties have confirmed has been successfully received and decoded. The encoding end can store it in a Decoded Picture Buffer (DPB) as a long-term reference for subsequent frames. When packet loss occurs and video errors occur, the decoding end can request the encoding end to use the most recent LTR frame as a reference to encode subsequent frames, thereby quickly restoring the video stream without sending a large, congestion-prone critical frame (I-frame).

[0033] The effective operation of the LTR mechanism relies on a closed-loop feedback process. For example, after successfully decoding a frame, the decoding end can send an acknowledgment message to the encoding end through a feedback channel such as the Real-time Transmission Control Protocol (RTCP). Only after receiving this message can the encoding end securely mark the corresponding frame as an LTR frame for subsequent use.

[0034] However, the reliability of this feedback process is closely related to the network's round-trip time (RTT). The reference frame buffer size for the encoder and decoder is limited; for example, the buffer size available for LTR (hereinafter referred to as ltr_dpb_size) may be only 8 or 16 frames, depending on the specifications in the coding standard. This means that from the time a frame (LTR candidate frame) is encoded and sent until it is officially marked as an LTR frame and used, the number of newly encoded frames cannot exceed ltr_dpb_size.

[0035] In an ideal network environment with a low RTT, LTR feedback can reach the encoder quickly, and the above mechanism works well. However, in real-world RTC scenarios, especially in high RTT environments such as cross-border communications, satellite networks, or mobile networks with poor signal, LTR feedback signaling can experience delays of hundreds of milliseconds or even longer. During this delay, the encoder continuously encodes and sends new video frames to maintain the preset frame rate. When the number of new frames encoded and sent exceeds ltr_dpb_size, even if the delayed LTR feedback eventually arrives, its corresponding LTR candidate frame has already been "squeezed out" of the reference frame buffer by subsequent new frames, causing the LTR marking operation to fail, which can be called an "LTR chain break".

[0036] When the LTR chain breaks, the encoder loses an effective means of error recovery. If the decoder needs to perform error recovery, the only option is to forcibly send a completely new keyframe to reconstruct the decoder's reference state. Keyframes do not depend on any historical frames, have extremely large data volumes, and their transmission instantly consumes a significant amount of bandwidth, easily leading to increased network congestion and drastic fluctuations in the bitrate curve. In video call scenarios, this manifests as noticeable video stuttering on the user's end.

[0037] Therefore, existing technologies cannot guarantee the continued effectiveness of the LTR mechanism in high RTT environments, such as video calls, where the smoothness and stability of video calls may decrease.

[0038] As an example, as shown in 1B, this specification illustrates an application scenario of a frame rate determination method according to an exemplary embodiment, which shows the failure of the LTR mechanism in a high RTT environment.

[0039] Assume `ltr_dpb_size` is 8 frames. The encoder finishes encoding frame F1 at time T1 and expects to use it as an LTR frame. Due to the high RTT, the feedback arrives at time T1'. During the time interval from T1 to T1', the encoder continuously sends frames F2 to F9 at a high frame rate, a total of 8 frames. When the feedback arrives at time T1', the newly sent 8 frames have exceeded the buffer size of 8, causing frame F1 to be squeezed out of the reference frame buffer, resulting in LTR marking failure and a broken LTR chain.

[0040] Based on this, the embodiments of this specification provide a frame rate determination method that can maintain the continuity of the LTR chain at the encoding end, so that even if network packet loss occurs, the decoding end can use LTR frames for error recovery, preventing network congestion and drastic bit rate fluctuations caused by the encoding end sending key frames. The embodiments of this specification will now be described in detail.

[0041] like Figure 2 The diagram shown is a flowchart illustrating a frame rate determination method according to an exemplary embodiment of this specification. The method is applied at an encoding end, which encodes the acquired video frames according to the current frame rate and then sends them to a decoding end. The method may include the following steps: In step 202, if the current video frame has been encoded, the encoding completion time of the current video frame is recorded.

[0042] In step 204, the current feedback delay of the decoding end is determined based on the encoding completion time and the reception time of the feedback signaling returned by the decoding end.

[0043] The feedback signaling indicates that the decoding end has successfully decoded the current video frame.

[0044] In step 206, the safe frame rate is determined based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store.

[0045] The secure frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to no more than the preset maximum number of video frames.

[0046] In step 208, the current frame rate is set so that it is not greater than the safe frame rate.

[0047] As an example, the method of this embodiment can be applied to, for example, Figure 1A The image shows the encoding end in a video processing system. The encoding end can be contained within electronic devices capable of video capture, encoding, and network communication. As an example, these electronic devices are the runtime environments for applications (such as clients) with video processing capabilities.

[0048] As an example, the encoding end can be implemented in the following (but not limited to) types of electronic devices: Mobile smart terminals: such as smartphones, tablets, etc. These devices have built-in cameras (as video capture modules), hardware video encoders, and can run real-time communication applications. The steps of the method can be executed by the application by calling the device's processing resources.

[0049] Personal computers: such as desktops and laptops, capture video through built-in or external cameras and use CPUs or GPUs for software or hardware encoding.

[0050] Dedicated video conferencing equipment: such as embedded devices like video conferencing terminals, smart cameras, and drone image transmission systems, which integrate a complete processing module from acquisition to encoding.

[0051] In these embodiments, the "encoding end" can physically refer to the terminal devices themselves, or logically to the software modules or hardware processing units (such as encoder chips and their drivers / firmware) within them used to perform video encoding and the methods of this embodiment. In this context, the "client" can refer to the application running on these electronic devices, responsible for implementing complete real-time communication functions.

[0052] As an example, the implementation scenario could be as follows: After the client starts, it calls the device's camera to capture video. The captured raw video frames are sent to the encoding end of the client in this embodiment for compression encoding. Furthermore, a background service or dedicated thread running within the client (corresponding to the execution body of the frame rate determination method in this embodiment) can be started to execute steps 202 to 208 of this embodiment.

[0053] In other embodiments, the encoding end included in this embodiment can also be a computer device such as a server. The server can be a physical server containing an independent host, or the server can be a virtual server hosted by a host cluster. During operation, the server can run a server-side program of an application to implement the relevant functions of the application. For example, when the server runs a video processing service program, the program can include the encoding end of this embodiment, and the encoding end can execute the frame rate determination method embodiment of this embodiment.

[0054] In the above scenario, the device where the decoding end is located can be any, such as the aforementioned electronic devices or computer devices such as servers. This embodiment does not limit this.

[0055] As an example, this embodiment can be applied to various video communication scenarios, such as various video communication scenarios that introduce Long Term Reference (LTR) frame technology. In such video communication scenarios, the encoding end and the decoding end can establish point-to-point or one-to-many real-time video communication with low latency and closed-loop feedback channels. In order to ensure the smoothness of video and the quality of the picture under network fluctuations, Long Term Reference (LTR) frame technology can be introduced for efficient error recovery.

[0056] Video communication scenarios include, but are not limited to: Real-time audio and video calls and video conferencing: In video interactions between two or more terminals, each party acts as both an encoder and decoder. Network round-trip time (RTT) directly impacts the interactive experience. This embodiment effectively addresses the issue of LTR mechanism failure under high RTT networks, reducing video stuttering and improving call quality.

[0057] Cloud gaming and cloud desktops: In this scenario, the cloud server can act as an encoding end, encoding the game screen or desktop interface into a video stream in real time and sending it to the user terminal (decoding end). User operation commands can be uploaded as feedback signaling, and LTR technology is used to cope with network packet loss. This embodiment, through adaptive frame rate control, can maintain the continuity and stability of the game or desktop screen to the maximum extent while ensuring operation responsiveness.

[0058] Remote video surveillance and industrial IoT vision control: In scenarios involving remote real-time monitoring or machine vision control via wireless networks (such as 4G / 5G, satellite links), the monitoring end (encoding end) and the control center (decoding end) constitute a real-time video processing system. This embodiment ensures that even with high link latency, the monitoring video stream will not be frequently interrupted due to LTR failure, guaranteeing the real-time performance of monitoring and the reliability of remote control.

[0059] As an example, the encoding end in this embodiment can be responsible for executing adaptive frame rate control logic: by measuring the time between the completion of video frame encoding and the receipt of the corresponding decoding confirmation feedback (i.e., feedback delay), and combining it with the capacity of its internal reference frame buffer (i.e., the preset maximum number of video frames), a safe frame rate is dynamically calculated.

[0060] As an example, in step 202 of this embodiment, when the encoding of the current video frame is completed, the encoding completion time of the current video frame can be recorded.

[0061] As an example, for any encoded video frame n, the precise time point Tn at which its encoding is completed can be recorded, that is, each frame is used as the "current video frame".

[0062] As another example, the current video frame may not be every video frame. For example, it may be a video frame that "needs to be paid attention to" selected from the captured video frames. That is, for a series of continuously captured video frames, some video frames may be used as the "current video frame" in this embodiment, while some video frames may not be used as the "current video frame" in this embodiment. In actual applications, it can be set as needed, and this embodiment does not limit it.

[0063] In practice, there are various ways to record the encoding completion time of video frames, and this embodiment does not limit this. For example, the encoding end can maintain data in memory to record the encoding completion time of one or more current video frames. The specific data structure can be set according to actual needs, and this embodiment does not limit this.

[0064] As an example, this embodiment can also synchronously monitor feedback signals received from the network indicating decoding completion. For instance, in this embodiment, the decoding end assumes the feedback responsibility: whenever a video frame is successfully decoded, it can generate a feedback signal indicating successful decoding and send it back to the encoding end via the network, thereby providing information input for the adaptive control of the encoding end.

[0065] In scenarios where Long Term Reference Frame (LTR) technology is introduced, after successfully decoding a frame (especially a frame marked as an LTR candidate frame by the encoder), the decoder can send an "acknowledgment message" or "decoding success feedback" to the encoder via a feedback channel such as RTCP. In this embodiment, this is called feedback signaling. The core purpose of this feedback signaling is to notify the encoder: "I have successfully decoded a certain frame, and you can now safely mark it as an LTR frame." Without this feedback, the encoder cannot confirm whether the frame has been correctly received by the decoder.

[0066] It can be seen that in scenarios where long-term reference frames (LTRs) are introduced, the feedback signaling serves to assist the encoder in marking LTR frames. This can be understood as the encoder using this feedback signaling as a switch: upon receiving feedback, it performs the "marking LTR frame" action. It does not continuously and quantitatively measure the latency of this feedback, nor does it consider linking this latency to global coding frame rate control.

[0067] In the embodiments of this specification, the feedback signaling is utilized and given a new purpose: to determine the feedback delay between the completion of video frame encoding and the receipt of feedback from the encoding end, so as to use the feedback delay as a control variable to dynamically determine the safe frame rate.

[0068] As an example, the feedback signaling returned by the decoding end may contain the identifier of the video frame. Therefore, the encoding end can determine which video frame the feedback signaling belongs to by parsing the feedback signaling, thereby determining the encoding completion time of each video frame and the reception time of the corresponding feedback signaling.

[0069] As an example, in step 204, the current feedback delay of the decoding end is determined based on the encoding completion time and the reception time of the feedback signal returned by the decoding end. In practice, this can be achieved in several ways. For instance, the current feedback delay can be the single-frame feedback delay, that is, the single-frame feedback delay is determined based on the encoding completion time of each video frame and the reception time of the corresponding feedback signal. This method allows for dynamic frame rate control frame by frame with high precision.

[0070] Alternatively, in other examples, the current feedback latency of the decoding end can be determined by comprehensively considering the single-frame feedback latency (the time of encoding completion and the time of receiving the corresponding feedback signal) of multiple video frames. Specifically, the number of single-frame feedback latencies used to comprehensively determine the current feedback latency of the decoding end can be set according to actual needs; this embodiment does not limit this. Furthermore, there are multiple ways to "comprehensively determine" this latency, such as using statistical methods. The specific statistical information used can also be set according to actual needs, such as average, mean square error, maximum value, median, etc.; this embodiment does not limit this. This method can achieve dynamic frame rate control based on multiple frames, reducing the consumption of hardware resources through certain time intervals.

[0071] After determining the current feedback delay, the encoding end can perform the safe frame rate decision in step 206. The purpose of this decision is to derive a safe frame rate that can ensure the continued effectiveness of the Long Term Reference (LTR) mechanism by establishing a constraint relationship between the two key parameters mentioned above: the current feedback delay and the preset maximum number of video frames in the reference frame buffer (i.e., ltr_dpb_size).

[0072] The maximum number of preset video frames that the reference frame buffer at the encoding end can store can be customized in practical applications, or it can be determined based on the specifications for long-term reference frames in the encoding standard. This embodiment does not limit this.

[0073] As an example, there are several ways to determine the safe frame rate. For instance, it can be achieved through a mathematical or logical mapping relationship, taking the current feedback delay and the preset maximum video frame rate as input, and outputting a specific frame rate value, i.e., the safe frame rate. The design goal of this mapping relationship can be to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to no more than the preset maximum video frame rate. This can be understood as ensuring that, if the encoding end is running at this safe frame rate, the total number of video frames generated and sent within any time period equivalent to the "current feedback delay" is limited to no more than the preset maximum video frame rate.

[0074] Thus, even in scenarios with high network latency, during the entire latency period from the transmission of a frame to its confirmation feedback, newly generated frames at the encoding end will not fill or overflow the reference frame buffer, thereby ensuring that the LTR candidate frame can be safely kept in the buffer, waiting to be officially marked and referenced.

[0075] In practice, there are various ways to implement the mapping relationship between "current feedback latency + preset maximum video frame rate" and "safe frame rate," but they all share the common goal of adhering to the design objectives mentioned above. For example, frame rate is a measure of the number of frames changing over time; it can refer to how many frames are played per second, such as using "frames per second" (FPS). Therefore, the safe frame rate can be determined by dividing the preset maximum video frame rate by the current feedback latency. Alternatively, a pre-set lookup list or table can be used to retrieve the corresponding safe frame rate from the pre-set list or table based on "current feedback latency + preset maximum video frame rate."

[0076] As an example, determining the safe frame rate based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store may include: Determine the quotient between the preset maximum number of video frames that the reference frame buffer at the encoding end can store and the current feedback delay, and determine the safe frame rate based on the quotient.

[0077] In this embodiment, the quotient between the preset maximum number of video frames that the reference frame buffer can store and the current feedback delay can be calculated. This quotient can represent the theoretical maximum frame rate that "makes the number of video frames sent from the encoding end to the decoding end within the current feedback delay not greater than the preset maximum number of video frames".

[0078] Furthermore, there are multiple ways to determine the safe frame rate based on the quotient (theoretical maximum frame rate). For example, the quotient can be directly used as the safe frame rate, or the quotient can be adjusted (e.g., reduced) to obtain the safe frame rate according to actual needs. This embodiment does not limit this.

[0079] As can be seen from the above embodiments, this embodiment can quickly determine the safe frame rate by calculating the quotient value as described above.

[0080] As an example, determining the safe frame rate based on the quotient value may include: The safe frame rate is obtained by reducing the quotient value according to a preset safety factor; wherein the preset safety factor is a value less than 1.

[0081] In this embodiment, considering factors such as calculation errors and instantaneous network fluctuations, a preset safety system is introduced to reserve margin for network fluctuations, further ensuring that the actual frame rate is less than the preset maximum video frame rate of the buffer. Specifically, the safety factor in this embodiment is less than 1, making the determined safe frame rate lower than the quotient (theoretical maximum frame rate). In practical applications, the specific value of this preset safety factor can be flexibly set according to actual needs; this embodiment does not limit it. In this embodiment, reserving a margin based on theoretical calculations when determining the safe frame rate improves system robustness.

[0082] After determining the safe frame rate in step 206, step 208 can be executed to set the current frame rate so that it is not greater than the safe frame rate. In practical applications, the method of setting the current frame rate according to the safe frame rate can be set according to actual needs. For example, the safe frame rate can be used as the current frame rate, or the current frame rate can be obtained by reducing the safe frame rate. The reduction strategy can also be set based on various factors, such as the actual video processing scenario or the network jitter situation, etc. This embodiment does not limit this.

[0083] As can be seen, the embodiments in this specification, by introducing an adaptive frame rate control mechanism based on real-time network conditions, bring about the following significant technical effects: 1. Ensure the effectiveness of the Long-Term Reference (LTR) mechanism to fundamentally prevent reference frame loss. This specification's embodiments establish a forward-looking protection mechanism by "determining a safe frame rate based on the current feedback delay and a preset maximum number of video frames" and "setting the current frame rate to be no greater than the safe frame rate." Its core effect is that it ensures that within the current feedback delay from the encoding of any video frame to the return of its acknowledgment feedback, the total number of newly transmitted video frames at the encoding end will not exceed the capacity of the reference frame buffer (i.e., the preset maximum number of video frames). This allows the earliest transmitted candidate frame to be marked as LTR to remain in the buffer and not be squeezed out by subsequent new frames when its feedback signal arrives, thereby ensuring the successful execution of the LTR marking operation and maintaining the continuity of the LTR chain.

[0084] 2. Significantly reduces unnecessary keyframes, improving video bitrate stability and network bandwidth utilization efficiency. Because the aforementioned effects ensure the continued effectiveness of the LTR mechanism, the decoding end can reliably use LTR frames for error recovery when packet loss occurs in the network. This directly leads to a significant reduction in the need to generate keyframes (I-frames) due to LTR failure. Keyframes contain far more data than regular frames, and their periodic and bursty transmission is a major cause of drastic fluctuations in video bitrate and network congestion. The embodiments in this specification reduce these unnecessary keyframes, resulting in a more stable video stream bitrate, mitigating the impact on the network, and thus improving overall bandwidth utilization efficiency.

[0085] 3. Achieve dynamic adaptation to high-latency networks, improving call fluency and stability. This embodiment of the specification, by "recording the encoding completion time and reception time to determine the current feedback latency," can accurately and in real time perceive changes in network round-trip time (RTT). Furthermore, through the step of "determining a safe frame rate based on the current feedback latency," the key parameter of network latency is directly converted into frame rate control commands at the encoding end. When the network RTT increases and the feedback slows down, the system automatically calculates a lower safe frame rate and limits the current frame rate, prioritizing the LTR mechanism and the continuity of the video stream in a "slow and stable" mode. This dynamic adaptive capability enables the system to effectively cope with high-latency and fluctuating network environments, ultimately resulting in reduced video stuttering and improved overall call smoothness and stability for the user.

[0086] In some examples, determining the current feedback delay of the decoding end based on the encoding completion time and the reception time of the feedback signaling returned by the decoding end may include: Based on the reception time of the feedback signaling, record the time difference between the encoding completion time and the reception time; The current feedback delay of the decoding end is determined based on the time difference of the video frames.

[0087] In this embodiment, when the encoding end receives feedback signaling from the decoding end for a specific video frame (denoted as "current video frame"), the following operations can be performed: Obtain the pre-recorded encoding completion time of the current video frame. Record the reception time of the received feedback signaling. Calculate and record the time difference between the two times mentioned above.

[0088] The time difference in this embodiment can be understood as the single-frame feedback delay experienced by a single video frame from the completion of encoding to the confirmation of decoding. It is the most original and direct data sample, objectively reflecting the network round-trip delay and processing delay experienced by the frame.

[0089] As an example, the encoding end can maintain data in memory to record the "time difference (single frame feedback delay)" of one or more current video frames. The specific data structure can be set according to actual needs, and this embodiment does not limit it.

[0090] Therefore, after obtaining the single-frame feedback delay, the current feedback delay at the decoding end can be further determined. As in the aforementioned embodiment, the single-frame feedback delay can be used as the "current feedback delay," or the "current feedback delay" can be determined based on multiple single-frame feedback delays.

[0091] As can be seen from the above embodiments, this embodiment provides an accurate data basis for determining the current feedback delay of the decoding end by recording the time difference between the encoding completion time of the video frame and the receiving time of the corresponding reverse signaling.

[0092] As mentioned in the previous embodiments, in practical applications, the current feedback delay can be determined frame by frame or by combining multiple frames. This embodiment provides a determination method based on historical datasets, rather than relying solely on the instantaneous measurement value of a single frame. This can effectively filter out interference caused by network jitter and reduce computational overhead.

[0093] As an example, recording the time difference between the encoding completion time and the reception time includes: The time difference of the current video frame is recorded in a preset sequence; Determining the current feedback delay of the decoding end based on the time difference of the current video frame includes: Periodic execution: Based on the time difference of multiple video frames recorded in the preset sequence, determine the current feedback delay of the decoding end in the current period.

[0094] As an example, after calculating the single time difference (i.e., single-frame feedback delay) of the current video frame, the encoding end can record and maintain this time difference as a data sample within a preset sequence. Optionally, this "preset sequence" can be a logical container used to store multiple historical time difference (single-frame feedback delay) samples within a time period, either sequentially or unordered. Its specific implementation can vary; for example, it can be a fixed-length sliding window whose content is dynamically updated as new samples are added, always maintaining the most recent N samples; or it can be a time-based collection that retains all samples collected within the most recent T time units (e.g., within the most recent 5 seconds).

[0095] As an example, the step of "periodically executing: determining the current feedback latency of the decoding end in the current period based on the time difference of multiple video frames recorded in the preset sequence" constitutes the latency evaluation method of this embodiment. The encoding end does not immediately decide the frame rate every time feedback is received; instead, the decision-making process can be triggered at a set period (such as every 100 milliseconds, every 10 feedbacks received, or every 1 second, etc., a custom period). At the arrival of each decision period, a representative latency statistic can be determined based on all or part of the valid samples included in the current preset sequence through a predetermined decision-making method, and this value is determined as the "current feedback latency" used for frame rate decision in this period.

[0096] As can be seen from the above embodiments, the advantages of this periodic statistical mechanism are: Stability: The current feedback delay of the current period can be determined by comprehensively analyzing multiple time difference samples recorded in the preset sequence. This can smooth out the random fluctuations of a single measurement, and the resulting "current feedback delay" can better represent the general latency of the network.

[0097] Controllable computational overhead: Performing relatively complex statistical operations periodically, rather than frame-by-frame, helps control the computational load and avoids impacting high-priority coding tasks.

[0098] At periodic decision points, the encoder can calculate a representative current feedback latency that can guide frame rate decisions based on multiple time differences in a preset sequence. This embodiment also provides several optional statistical methods, one of which can be flexibly selected and implemented according to its trade-off between performance and robustness.

[0099] As an example, determining the current feedback delay of the decoding end in the current period based on the time difference of multiple video frames recorded in the preset sequence may include any of the following steps: Based on the time difference of multiple video frames recorded in the preset sequence, a preset percentile is calculated, and the current feedback delay of the decoding end in the current period is determined based on the preset percentile; wherein, the preset percentile is located between the median of the time difference of the multiple video frames and the maximum value of the time difference of the multiple video frames. Calculate the average of the time differences of multiple video frames recorded in the preset sequence, and determine the current feedback delay of the decoding end in the current period based on the average value; or, The maximum value is obtained from the time difference of multiple video frames recorded in the preset sequence, and the current feedback delay of the decoding end in the current period is determined based on the maximum value.

[0100] As an example: (1) The first type of method can be to use a preset percentile.

[0101] As an example, in this embodiment, all time difference samples in the sequence can be sorted in ascending order of value, and a predetermined percentile (such as the 90th, 95th, or 98th percentile, etc.) located between the median and the maximum value can be calculated, and this percentile can be determined as the current feedback delay.

[0102] The advantage of using percentiles in this embodiment lies in their robustness: they can effectively filter out abnormally high latency caused by a few extreme network jitters (these outliers are usually reflected at or near the maximum value), avoiding overly conservative decisions; at the same time, they are more sensitive to the prevalent high latency in the network than statistical measures such as the median or mean. By setting the percentile between the median and the maximum value, an optimal balance is achieved between "ignoring occasional anomalies" and "fully sensing congestion," thus making the determined frame rate both efficient and safe.

[0103] As an example, the 95th percentile is a statistical concept that means the value located at the 95th percentile when all sample data are arranged in ascending order. It indicates that 95% of the data are less than or equal to this value.

[0104] For example: Assuming the feedback latency (in milliseconds) of the most recent 20 video frames was measured, the following preset sequence was obtained: [80, 85, 90, 92, 95, 100, 105, 110, 120, 130, 140, 150, 160, 170, 500, 180, 190, 200, 1500, 2500] Step 1: Sort Sort these 20 data points from smallest to largest: [80, 85, 90, 92, 95, 100, 105, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 500, 1500, 2500] Step 2: Locate the 95th percentile. Position = Total number of data in the sequence × 95% = 20 × 0.95 = 19th (where, if the result is not an integer, interpolation or other methods can be used, or an approximate position can be taken).

[0105] Step 3: Obtain the value In the sorted list, the 19th data point is 1500 ms.

[0106] Therefore, the value of cost_95th = 1500ms in this embodiment indicates that within the statistical period, the feedback latency of 95% of the frames is within 1500 milliseconds. Although extreme latency of up to 2500 milliseconds has occurred in the network (possibly due to instantaneous retransmission or routing jitter), the decision in this embodiment can be based on 1500 milliseconds, which is more representative of "common but poor" network conditions.

[0107] If we take the maximum value (100%): the safe frame rate can be calculated based on 2500ms, but the result is lower and more conservative.

[0108] If we take the median (50%): the safe frame rate can be calculated based on 140ms, which is high, but may not be able to cope with the frequent network conditions that are much slower than 140ms.

[0109] Taking the 95th percentile: strikes a balance between "not overreacting" and "being adequately preventative." It ensures stability in the vast majority of cases (95%).

[0110] (2) The second type of method can be to calculate the average value.

[0111] This embodiment calculates the arithmetic mean of all samples in a preset sequence and uses it as the current feedback latency. This method is simple to implement, has low computational overhead, and is suitable for stable environments with relatively concentrated network latency distribution and low jitter, reflecting the overall average latency level.

[0112] (3) The third type of method is to directly obtain the maximum value.

[0113] In this embodiment, the maximum value among all time difference samples in the sequence is directly used as the current feedback latency. This is a relatively conservative strategy, ensuring that the frame rate decision is sufficient to handle the worst network conditions that occurred within the observed time period, thus providing the highest safety margin.

[0114] In summary, this embodiment provides a variety of statistical strategies, ranging from aggressive (average value) to conservative (maximum value), as well as robust and efficient (preset percentile), enabling this implementation scheme to flexibly adapt to different network environments and different levels of quality requirements.

[0115] As an example, the preset sequence can be used to record the time difference of a preset number of video frames; The step of recording the time difference of the current video frame in a preset sequence may include: Check whether the number of time differences recorded in the preset sequence is equal to the preset quantity; If so, then after deleting the time difference of the earliest video frame in the preset sequence, the time difference of the current video frame is recorded in the preset sequence; If not, the time difference of the current video frame is recorded in a preset sequence.

[0116] In one implementation of this embodiment, the preset sequence can be constructed as a fixed-length first-in-first-out queue, the length of which is defined by a preset quantity, the specific value of which can be set according to actual needs. The preset sequence in this embodiment can be understood as a sliding window, which ensures that the maintained time difference dataset is always up-to-date and reflects recent network conditions through the following mechanism: For example, when a new time difference of the current video frame needs to be recorded, the system first checks whether the number of time difference samples already stored in the preset sequence has reached the preset number.

[0117] If the window is full, the oldest recorded time difference sample in the sequence can be deleted first, and then the new time difference sample can be added to the end of the sequence. This operation is equivalent to "sliding" the statistical window forward one step on the time axis, removing the oldest data point and incorporating the latest data point.

[0118] If the window is not full, new time difference samples can be added directly to the end of the sequence.

[0119] As can be seen, this embodiment can maintain data timeliness: it can automatically limit the time span of historical data involved in the statistics, ensuring that the calculated current feedback latency can quickly respond to changes in network status and avoid being misled by outdated historical data.

[0120] It can also maintain computational stability: by fixing the number of samples, it ensures that the size of the sample set used for statistical calculations is constant, thereby avoiding fluctuations in statistical results that may be caused by the continuous growth or sudden changes in the number of samples, and providing a stable computational basis for frame rate decisions.

[0121] As an example, setting the current frame rate to be no greater than the safe frame rate may include: Compare the safe frame rate with the current frame rate; If the safe frame rate is equal to the current frame rate, then the current frame rate is maintained; If the safe frame rate is less than the current frame rate, then the current frame rate is reduced to the safe frame rate.

[0122] This embodiment provides how to apply the determined safe frame rate to the frame rate control logic at the encoding end, especially in scenarios where it is necessary to reduce the frame rate to ensure stability; for example: Once the safe frame rate is obtained, it can be compared with the encoder's current output frame rate (i.e., the current frame rate).

[0123] If the safe frame rate equals the current frame rate, it indicates that the current frame rate setting already meets network security requirements and does not need adjustment. Therefore, the current frame rate can be maintained, avoiding unnecessary control operations and reducing control overhead at the encoding end.

[0124] If the safe frame rate is lower than the current frame rate, it indicates that the current network feedback latency is high. Maintaining the current frame rate may cause the LTR mechanism to fail. This embodiment can reduce the current frame rate to the safe frame rate. Therefore, this embodiment adopts this preventive control strategy. By actively and timely reducing the frame rate, it prioritizes the continuity of the video stream and the effectiveness of the LTR mechanism, thereby fundamentally avoiding keyframe storms and video stuttering caused by buffer overflow.

[0125] As an example, the method may also include: If the safe frame rate is greater than the current frame rate, compare the safe frame rate with the preset maximum frame rate; If the safe frame rate is less than the preset maximum frame rate, then the current frame rate is increased to the safe frame rate; If the safe frame rate is greater than the preset maximum frame rate, then the current frame rate is increased to the preset maximum frame rate.

[0126] To ensure that bandwidth resources are fully utilized to improve video smoothness when network conditions improve, this embodiment further provides frame rate up-adjustment control logic: When the safe frame rate is greater than the current frame rate, it indicates that the network feedback latency has been reduced, and there is potential to increase the frame rate. However, to avoid the frame rate from increasing indefinitely beyond the device's processing capacity or the needs of the actual application scenario, this embodiment introduces a preset maximum frame rate as an upward adjustment boundary; wherein, the preset maximum frame rate can be set according to actual needs.

[0127] In this embodiment, the "preset maximum frame rate" is a custom upper limit value determined by business logic or system configuration, independent of the adaptive calculation of the safe frame rate. Its introduction ensures that the adaptive frame rate control in this embodiment does not deviate from the actual application scenario and limitations.

[0128] The safe frame rate is a theoretically safe value calculated based on network feedback latency and buffer size. Under excellent network conditions (e.g., extremely low RTT), the calculated safe frame rate can be very high (e.g., exceeding 100 FPS). However, in real-world business scenarios, excessively high frame rates may not provide significant benefits, for example: For applications like video calls, the human eye has a saturation point in its perception of frame rate. Typically, increasing the frame rate from 15 FPS to 30 FPS significantly improves smoothness; however, increasing it from 30 FPS to 60 FPS has a negligible improvement in call experience, far less important than increasing resolution or image quality. Therefore, limiting the frame rate to a "good enough" level is a better resource allocation strategy.

[0129] Terminal performance and power consumption constraints: Higher frame rates mean that encoders, processors, and network modules need to work under higher loads continuously, which will significantly increase device power consumption and heat generation. For mobile devices (such as phones and tablets), prioritizing battery life and device temperature is a crucial business requirement in call scenarios. Therefore, setting a reasonable frame rate cap is a key means of balancing experience and power consumption.

[0130] Therefore, the "preset maximum frame rate" in this embodiment can be a configurable parameter. It can be statically preset or dynamically adjusted based on various service factors such as device type, network type, user plan, application scenario, or power consumption strategy.

[0131] Based on this, when the safe frame rate is greater than the current frame rate, this embodiment can further compare the safe frame rate with the preset maximum frame rate: If the safe frame rate is less than the preset maximum frame rate, it means that the frame rate allowed by the network conditions is within the device's limits. The current frame rate can be increased to the safe frame rate to improve the visual experience as much as possible while ensuring safety.

[0132] If the safe frame rate is greater than or equal to the preset maximum frame rate, it indicates that the network conditions are good, but it should not exceed the preset performance limit. Therefore, in this embodiment, the current frame rate is increased to the preset maximum frame rate.

[0133] As can be seen from the above embodiments, the frame rate adjustment method in this embodiment embodies an adaptive principle that balances efficiency and constraints. It enables intelligent restoration of video quality when network conditions improve, while ensuring that encoding load and resource consumption remain within a controllable range through the hard constraint of a preset maximum frame rate, thus achieving a dynamic balance between stability and smoothness.

[0134] The three processing steps involved in the above embodiments are: "recording the time difference between the completion time of video frame encoding and the reception time of the corresponding feedback signal", "determining the current feedback delay" and "determining the safe frame rate"; the processing cycle of these three steps can be set according to actual needs. For example, the update cycle for "recording time difference" can be performed frame by frame; that is, whenever the encoding end receives a decoding completion feedback signal, it can calculate and record the time difference of that frame (single-frame feedback delay). Therefore, this update cycle is event-driven, consistent with the arrival frequency of the feedback signal, and optionally, it can be triggered by each feedback.

[0135] The statistical period for "determining the current feedback delay" refers to the frequency at which the decoder calculates the current feedback delay in the current period. This statistical period can be determined based on a preset sequence that is maintained. For example, it can be triggered every time the time difference is recorded into the preset sequence (i.e., synchronized with the update period), or it can be performed periodically at fixed time intervals (such as a custom value such as every 100 milliseconds).

[0136] The decision cycle for the "safe frame rate" can be the frequency at which the safe frame rate is recalculated. As an example, the safe frame rate decision can be performed after the current feedback latency of the current cycle is determined; that is, the decision cycle can be synchronized with the statistical cycle, meaning the safe frame rate is recalculated each time the current feedback latency is updated. Of course, other methods, such as asynchronous decision-making, are also optional.

[0137] In other words, the above processing cycles do not necessarily need to be exactly the same. In practical applications, they can be set as needed. This embodiment does not limit this.

[0138] The following example illustrates this further. Figure 3A The diagram shown is a schematic of a video processing system according to an exemplary embodiment of this specification. The system includes an encoding end 31 and a decoding end 32, which communicate with each other through a network 33.

[0139] The encoding end 31 may include a video acquisition module (not shown), a video encoder 311, and the frame rate determination device 312 provided in this embodiment.

[0140] The frame rate determination device 312 may further include a latency measurement module 3121, a latency statistics module 3122, and a frame rate decision module 3123.

[0141] The decoding end 32 may include a video decoder 321 and a decoding feedback module 322.

[0142] During operation, the video encoder 311 encodes the acquired video, generates video frames, and sends them to the decoder 32 via the network 33.

[0143] The video decoder 311 receives and decodes video frames. After the video frame is successfully decoded, the decoding feedback module 322 generates a decoding completion feedback signal (e.g., an RTCP feedback message) and sends it back to the encoding end 31 via the network 33.

[0144] This embodiment can be applied to a frame rate determination device 312, and the workflow of this device may include the following steps: (1) Start delay measurement.

[0145] The delay measurement module 3121 begins operation. For any video frame n encoded by the video encoder 311, the delay measurement module 3121 can record the precise time point Tn (encoding completion moment) when the encoding is completed. Simultaneously, this module can listen for decoding completion feedback signals received from the network 300.

[0146] (2) Receive feedback and calculate the time difference (i.e., single-frame feedback delay).

[0147] When the encoding end 31 receives the decoding completion feedback signal from the decoding end 32 for video frame n, the delay measurement module 3121 records the time point Tn' of receiving the signal. Subsequently, the single feedback delay cost_n = Tn' - Tn of video frame n is calculated.

[0148] (3) Update the delay sample and perform statistics.

[0149] The delay measurement module 3121 pushes the calculated cost_n to the delay statistics module 3122. The delay statistics module 3122 maintains a sliding window or set containing historical feedback delays and recalculates the statistical feedback delay based on the updated set.

[0150] In a preferred embodiment, the module can calculate the 95th percentile of all latency samples as a robust statistical feedback latency cost_95th. Using the 95th percentile can effectively filter out extreme latency values ​​caused by occasional network jitter, while also accurately reflecting the current network congestion situation.

[0151] (4) Make a decision and determine the safe frame rate.

[0152] The latency statistics module 3122 passes the calculated cost_95th to the frame rate decision module 3123.

[0153] The frame rate decision module 3123 also obtains its LTR reference frame buffer size ltr_dpb_size from the video encoder 311 or the system configuration. Subsequently, the frame rate decision module 3123 calculates the safe frame rate target_fps according to the following method: target_fps = K × (ltr_dpb_size × 1000) / cost_95th Here, K is a preset safety factor, such as 0.97. Setting the safety factor K is to retain a certain safety margin based on theoretical calculations to cope with instantaneous network fluctuations and ensure that, in most cases, the number of frames sent within a feedback cycle is less than ltr_dpb_size.

[0154] (5) Set the current frame rate.

[0155] The frame rate decision module 3123 sets the current frame rate based on the calculated target_fps, and can configure the set current frame rate to the video encoder 311. The video encoder 311 will adjust its video encoding rate so that its output frame rate does not exceed target_fps.

[0156] The above process can be used as a closed-loop control system. For example, in a video call scenario, it can run continuously and periodically during an RTC call to achieve real-time and dynamic adjustment of the frame rate.

[0157] like Figure 3B The diagram shown is an application scenario illustration of a frame rate determination method according to an exemplary embodiment of this specification, which is consistent with the aforementioned Figure 1B Compared to the embodiments described in this specification, the current feedback delay is measured to be approximately T1' - T1, which can be used to reduce the target frame rate. Within the same feedback delay, the encoder only sends frames F2 to F7, a total of 6 frames. Since the newly sent 6 frames are less than the buffer size 8, when the feedback arrives at time T1', frame F1 is still in the reference frame buffer and can be successfully marked as an LTR frame, thus avoiding unnecessary keyframes.

[0158] Let's take a specific numerical value as an example: Assuming the encoder's ltr_dpb_size is 8, the calculated cost_95th is 500ms, and the safety factor K is set to 0.97, then the safe frame rate target_fps determined during frame rate decision is 0.97 × (8 × 1000) / 500 = 15.52 FPS.

[0159] The encoder will run at a frame rate of approximately 15 FPS, ensuring that only about 15 × 0.5 = 7.5 frames are sent within a 500ms feedback period, which is less than 8, thus guaranteeing the effectiveness of the LTR mechanism.

[0160] As can be seen from the above embodiments, this specification provides a target frame rate determination scheme for real-time communication (RTC) based on long-term reference (LTR) feedback delay, which can be applied to the field of real-time communication technology. To address the problem of LTR mechanism failure, keyframe issues, and video stuttering caused by excessively high LTR feedback delay in high round-trip time (RTT) networks, this specification proposes the following: For multiple video frames, measure the feedback time difference from the completion of encoding to the receipt of decoding completion feedback; calculate the current feedback delay (e.g., the 95th percentile) based on multiple feedback time differences; determine a safe frame rate based on this statistical current feedback delay and a preset LTR reference frame buffer size; and finally apply this safe frame rate to the video encoder. By establishing a closed-loop adaptive control between feedback delay and frame rate, this specification ensures the LTR mechanism remains effective under high RTT, significantly reducing unnecessary keyframe generation. For example, in video call scenarios, it can make the video bitrate more stable and the call experience smoother.

[0161] Compared with the prior art, the embodiments in this specification have the following beneficial effects: Ensuring the effectiveness of the LTR mechanism: The embodiments in this specification establish a closed-loop control relationship between feedback latency and safe frame rate, actively adapt to changes in network RTT, and ensure that the number of frames sent within a feedback cycle can always be accommodated by the LTR buffer, thereby ensuring the continuous effectiveness of the LTR mechanism under high RTT networks.

[0162] Improved video smoothness and bitrate stability: Due to the stable operation of the LTR mechanism, the embodiments in this specification significantly reduce the number of keyframes that are forced to be generated due to error recovery, effectively avoiding periodic bitrate spikes and network impacts caused by keyframes, resulting in smoother video bitrate and a smoother viewing experience for users.

[0163] Enhanced network adaptability: The embodiments in this specification can sense changes in network feedback link latency in real time and automatically adjust the frame rate strategy. When RTT increases, the frame rate is automatically reduced to ensure stability; when RTT decreases, the frame rate is automatically increased to make full use of network resources, realizing intelligent and dynamic adaptation to complex network environments.

[0164] Figure 4 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 4 As shown, device 400 mainly consists of a communication interface 402, a user interface 404, a processor 406, and a data storage 408. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 410. The communication interface 402 enables device 400 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 402 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 402 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 402 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0165] User interface 404 includes receiving user input and providing output to the user. Therefore, user interface 404 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 404 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 404 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 400 may support remote access from other devices via communication interface 402 or another physical interface (not shown). User interface 404 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 404 may also be configured as a display device for rendering or displaying text fragments.

[0166] Processor 406 may contain one or more general-purpose processors and / or special-purpose processors.

[0167] Data storage 408 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 406. Data storage 408 may include removable and non-removable components.

[0168] Processor 406 is capable of executing program instructions 418 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 408 to perform the various functions described herein. Data storage 408 may comprise a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 400, enable device 400 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 406 executing program instructions 418 may result in processor 406 using data 412.

[0169] For example, program instructions 418 may include an operating system 422 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 400 and one or more applications 420 (e.g., a browser, social application, or game application). Similarly, data 412 may include operating system data 416 and application data 414. Operating system data 416 is primarily accessible to the operating system 422, while application data 414 is primarily accessible to one or more applications 420. Application data 414 may reside in a file system visible or hidden from the user of device 400.

[0170] Application 420 can communicate with operating system 422 through one or more application programming interfaces (APIs). These APIs help application 420 read and / or write application data 414, transmit or receive information via communication interface 402, receive or display information on user interface 404, etc.

[0171] In some terminology, application 420 may be simply referred to as "app". Furthermore, application 420 can be downloaded to device 400 through one or more online app stores or app markets. However, applications can also be installed on device 400 in other ways, such as through a web browser or a physical interface on device 400 (e.g., a USB port).

[0172] Please refer to Figure 5 The frame rate determination device can be applied to, for example, Figure 4 The device shown implements the technical solution of this specification. The device is applied at the encoding end, which encodes the acquired video frames according to the current frame rate and then sends them to the decoding end; the frame rate determination device may include: The recording module 51 is used to: record the encoding completion time of the current video frame if the current video frame has been encoded; The delay statistics module 52 is used to: determine the current feedback delay of the decoding end based on the encoding completion time and the reception time of the feedback signal returned by the decoding end; wherein the feedback signal indicates that the decoding end has successfully decoded the current video frame; The frame rate decision module 53 is used to: determine a safe frame rate based on the current feedback delay and the preset maximum number of video frames that can be stored in the reference frame buffer at the encoding end; wherein the safe frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to not be greater than the preset maximum number of video frames; The frame rate control module 54 is used to: set the current frame rate so that it is not greater than the safe frame rate.

[0173] In some examples, the latency statistics module 52 determines the current feedback latency of the decoding end based on the encoding completion time and the reception time of the feedback signaling returned by the decoding end, including: Based on the reception time of the feedback signaling, record the time difference between the encoding completion time and the reception time; The current feedback delay of the decoding end is determined based on the time difference of the video frames.

[0174] In some examples, the recording module 51 records the time difference between the encoding completion time and the reception time, including: The time difference of the current video frame is recorded in a preset sequence; Determining the current feedback delay of the decoding end based on the time difference of the current video frame includes: Periodic execution: Based on the time difference of multiple video frames recorded in the preset sequence, determine the current feedback delay of the decoding end in the current period.

[0175] In some examples, the latency statistics module 52 determines the current feedback latency of the decoding end in the current period based on the time difference of multiple video frames recorded in the preset sequence, including any of the following steps: Based on the time difference of multiple video frames recorded in the preset sequence, a preset percentile is calculated, and the current feedback delay of the decoding end in the current period is determined based on the preset percentile; wherein, the preset percentile is located between the median of the time difference of the multiple video frames and the maximum value of the time difference of the multiple video frames. Calculate the average of the time differences of multiple video frames recorded in the preset sequence, and determine the current feedback delay of the decoding end in the current period based on the average value; or, The maximum value is obtained from the time difference of multiple video frames recorded in the preset sequence, and the current feedback delay of the decoding end in the current period is determined based on the maximum value.

[0176] In some examples, the preset sequence is used to record the time difference of a preset number of video frames; The recording module 51 records the time difference of the current video frame in a preset sequence, including: Check whether the number of time differences recorded in the preset sequence is equal to the preset quantity; If so, then after deleting the time difference of the earliest video frame in the preset sequence, the time difference of the current video frame is recorded in the preset sequence; If not, the time difference of the current video frame is recorded in a preset sequence.

[0177] In some examples, the frame rate decision module 53 determines a safe frame rate based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store, including: Determine the quotient between the preset maximum number of video frames that the reference frame buffer at the encoding end can store and the current feedback delay, and determine the safe frame rate based on the quotient.

[0178] In some examples, the frame rate decision module 53 determines a safe frame rate based on the quotient, including: The safe frame rate is obtained by reducing the quotient value according to a preset safety factor; wherein the preset safety factor is a value less than 1.

[0179] In some examples, the frame rate control module 54 sets the current frame rate to be no greater than the safe frame rate, including: Compare the safe frame rate with the current frame rate; If the safe frame rate is equal to the current frame rate, then the current frame rate is maintained; If the safe frame rate is less than the current frame rate, then the current frame rate is reduced to the safe frame rate.

[0180] In some examples, the frame rate control module 54 is also used for: If the safe frame rate is greater than the current frame rate, compare the safe frame rate with the preset maximum frame rate; If the safe frame rate is less than the preset maximum frame rate, then the current frame rate is increased to the safe frame rate; If the safe frame rate is greater than the preset maximum frame rate, then the current frame rate is increased to the preset maximum frame rate.

[0181] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0182] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0183] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0184] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0185] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0186] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0187] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0188] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0189] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0190] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0191] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A frame rate determination method, characterized in that, The method is applied at the encoding end, which encodes the acquired video frames according to the current frame rate and then sends them to the decoding end; the method includes: If the current video frame being captured has been encoded, record the time when the encoding of the current video frame has been completed; The current feedback delay of the decoding end is determined based on the encoding completion time and the reception time of the feedback signal returned by the decoding end; wherein the feedback signal indicates that the decoding end has successfully decoded the current video frame; A safe frame rate is determined based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store; wherein, the safe frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to not exceed the preset maximum number of video frames; Set the current frame rate so that it is not greater than the safe frame rate.

2. The method according to claim 1, wherein determining the current feedback delay of the decoding end based on the encoding completion time and the reception time of the feedback signaling returned by the decoding end includes: Based on the reception time of the feedback signaling, record the time difference between the encoding completion time and the reception time; The current feedback delay of the decoding end is determined based on the time difference of the video frames.

3. The method according to claim 2, characterized in that, The recording of the time difference between the encoding completion time and the reception time includes: The time difference of the current video frame is recorded in a preset sequence; Determining the current feedback delay of the decoding end based on the time difference of the current video frame includes: Periodic execution: Based on the time difference of multiple video frames recorded in the preset sequence, determine the current feedback delay of the decoding end in the current period.

4. The method according to claim 3, characterized in that, Determining the current feedback delay of the decoding end in the current period based on the time difference of multiple video frames recorded in the preset sequence includes any of the following steps: Based on the time difference of multiple video frames recorded in the preset sequence, a preset percentile is calculated, and the current feedback delay of the decoding end in the current period is determined based on the preset percentile; wherein, the preset percentile is located between the median of the time difference of the multiple video frames and the maximum value of the time difference of the multiple video frames. Calculate the average of the time differences of multiple video frames recorded in the preset sequence, and determine the current feedback delay of the decoding end in the current period based on the average value; or, The maximum value is obtained from the time difference of multiple video frames recorded in the preset sequence, and the current feedback delay of the decoding end in the current period is determined based on the maximum value.

5. The method according to claim 3, characterized in that, The preset sequence is used to record the time difference of a preset number of video frames; The step of recording the time difference of the current video frame in a preset sequence includes: Check whether the number of time differences recorded in the preset sequence is equal to the preset quantity; If so, then after deleting the time difference of the earliest video frame in the preset sequence, the time difference of the current video frame is recorded in the preset sequence; If not, the time difference of the current video frame is recorded in a preset sequence.

6. The method according to claim 1, characterized in that, The step of determining the safe frame rate based on the current feedback delay and the preset maximum number of video frames that the reference frame buffer at the encoding end can store includes: Determine the quotient between the preset maximum number of video frames that the reference frame buffer at the encoding end can store and the current feedback delay, and determine the safe frame rate based on the quotient.

7. The method according to claim 6, characterized in that, Determining the safe frame rate based on the quotient includes: The safe frame rate is obtained by reducing the quotient value according to a preset safety factor; wherein the preset safety factor is a value less than 1.

8. The method according to claim 1, characterized in that, Setting the current frame rate to be no greater than the safe frame rate includes: Compare the safe frame rate with the current frame rate; If the safe frame rate is equal to the current frame rate, then the current frame rate is maintained; If the safe frame rate is less than the current frame rate, then the current frame rate is reduced to the safe frame rate.

9. The method according to claim 8, characterized in that, The method further includes: If the safe frame rate is greater than the current frame rate, compare the safe frame rate with the preset maximum frame rate; If the safe frame rate is less than the preset maximum frame rate, then the current frame rate is increased to the safe frame rate; If the safe frame rate is greater than the preset maximum frame rate, then the current frame rate is increased to the preset maximum frame rate.

10. A frame rate determination device, characterized in that, The device is applied at the encoding end, which encodes the acquired video frames according to the current frame rate and then sends them to the decoding end; the device includes: The recording module is used to: record the time when the encoding of the current video frame is completed if the current video frame is encoded. The latency statistics module is used to: determine the current feedback latency of the decoding end based on the encoding completion time and the reception time of the feedback signal returned by the decoding end; wherein the feedback signal indicates that the decoding end has successfully decoded the current video frame; A frame rate decision module is used to: determine a safe frame rate based on the current feedback delay and the preset maximum number of video frames that can be stored in the reference frame buffer at the encoding end; wherein the safe frame rate is used to limit the number of video frames sent by the encoding end to the decoding end within the current feedback delay to not exceed the preset maximum number of video frames; A frame rate control module is used to: set the current frame rate so that it is not greater than the safe frame rate.

11. A video processing system, characterized in that, The system includes an encoding end and a decoding end, wherein the encoding end is used to perform the steps of the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as claimed in any one of claims 1 to 9 by executing the executable instructions.

13. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 9.

Images