Methods and related rendering devices for reducing display latency based on variable refresh rate technology
By dynamically adjusting the refresh rate of the display device using variable refresh rate technology, the problem of video display latency in existing technologies has been solved, achieving more efficient video display and lower latency.
Patent Information
- Application Number
- CN202311515984.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-11-14
AI Technical Summary
Existing methods for reducing video display latency are prone to frame drops or increase the computational load on cloud/streaming game servers, and cannot effectively shorten video latency in cloud/streaming games.
By statistically analyzing the latency of video frames and dynamically adjusting the refresh rate of the display device based on its variable refresh rate capability, variable refresh rate (VRR) technology can be used to shorten video display latency.
It effectively shortens video display latency, improves the gaming experience, avoids frame drops, and reduces the server's computational burden.
Smart Images

Figure CN120014994B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to display technology, and more particularly to a method and rendering apparatus for reducing video display latency through variable refresh rate technology. Background Technology
[0002] Today, with the widespread adoption of electronic products and cloud gaming, the demand for low-latency cloud gaming experiences is increasing. During cloud gaming, if the game's input deviates from the synchronization of audio / video, the gaming experience will be diminished. Therefore, reducing video latency in cloud gaming is crucial. Existing methods for reducing video display latency mainly fall into two categories: one is to adjust the amount of video buffering to achieve synchronization, and the other is to increase the video refresh rate to speed up video updates. However, both methods have drawbacks. Adjusting the buffering amount can easily lead to dropped frames, while increasing the video refresh rate increases the computational load on the cloud gaming server and the video decoding computation on the playback end. Therefore, an innovative method is needed to improve video display latency. Summary of the Invention
[0003] In view of this, the present invention proposes a mechanism for shortening video display latency using variable refresh rate technology. In this invention, the video frame latency corresponding to each video frame is statistically analyzed, and the refresh rate of the display device is dynamically adjusted based on the display device's support for variable refresh rates, thereby shortening the display latency of the video stream on the display device.
[0004] One embodiment of the present invention provides a method for displaying a video stream on a display device having a variable refresh rate function. The method includes: receiving the video stream and decoding the video stream to generate a plurality of decoded video frames; determining a video frame readiness time corresponding to each of the plurality of decoded video frames; determining a video frame display time corresponding to each of the plurality of decoded video frames; determining a target refresh rate of the display device based on a video frame delay between the video frame readiness time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and controlling the display device to display the video stream at the target refresh rate.
[0005] Another embodiment of the present invention provides a rendering apparatus for controlling a display device having a variable refresh rate function. The rendering apparatus includes a decoding unit and a refresh rate control unit. The decoding unit receives a video stream and decodes the video stream to generate a plurality of decoded video frames. The refresh rate control unit is coupled to the decoding unit and is configured to determine a video frame readiness time and a video frame display time corresponding to each of the plurality of decoded video frames, and to determine a target refresh rate of the display device based on a video frame delay between the video frame readiness time and the video frame display time corresponding to at least one of the plurality of decoded video frames. The rendering apparatus controls the display device to display the video stream at the target refresh rate. Attached Figure Description
[0006] Figure 1 This is a schematic diagram of the rendering apparatus and its application architecture according to an embodiment of the present invention.
[0007] Figure 2 This illustration depicts how an embodiment of the present invention reduces video display latency.
[0008] Figure 3 This illustration depicts how another embodiment of the invention reduces video display latency.
[0009] Figure 4 This illustration depicts how another embodiment of the invention reduces video display latency.
[0010] Figure 5 This is a flowchart of a method for controlling a display device according to an embodiment of the present invention. Detailed Implementation
[0011] Numerous specific details are described below to provide the reader with a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate how the invention can be implemented in the absence of one or more specific details, or by utilizing other methods, elements, or materials. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the core concepts of the invention.
[0012] The phrase "in one embodiment" in this specification means that a particular feature, structure, or characteristic described in that embodiment may be included in at least one embodiment of the invention. Therefore, the phrase "in one embodiment" appearing throughout this specification does not necessarily mean the same embodiment. Furthermore, the aforementioned particular features, structures, or characteristics may be combined in any suitable form in one or more embodiments.
[0013] Figure 1 A schematic diagram illustrating the rendering apparatus and its application architecture according to an embodiment of the present invention is shown. For example... Figure 1 As shown, the rendering device 200 receives a video stream from a video source 20 via a network 10 and decodes the video stream. Furthermore, the rendering device 200 sends a video signal to a display device 100 via a transmission interface 300 to control the display device 100 to display the decoded video content of the video stream. In one embodiment, the rendering device 200 may be a streaming media player, a smart TV box, a game console, etc. Moreover, the display device 100 may be various types of monitors or televisions supporting Variable Refresh Rate (VRR) technology, such as Liquid Crystal Display (LCD) / TVs, Organic Light-Emitting Diode (OLED) monitors / TVs, or Micro-LED monitors / TVs, etc.
[0014] In one embodiment, network 10 can be a wired network, a wireless network, or a hybrid network of both. The wired network may include Ethernet, modem lines, optical fiber networks, etc. The wired network further includes hardware devices that provide wired network connectivity, including but not limited to routers, switches, bridges, etc. These devices connect network nodes and manage data exchange between them. Furthermore, the wireless network may include Wi-Fi, Bluetooth, mobile telecommunications networks (such as 4G LTE, 5G), satellite communication networks, etc. The wireless network further includes hardware devices that provide wireless network connectivity, including but not limited to wireless routers, wireless access points (APs), mobile network base stations (such as 4G base stations, 5G base stations), etc. These devices connect wireless devices and manage data flow between them. In hybrid network embodiments, the hardware devices of the wired and wireless networks may be combined to provide more flexible network connectivity. For example, a Wi-Fi access point may be connected to a wired Ethernet network so that Wi-Fi devices can access data remotely via the Ethernet network.
[0015] In one embodiment, the video source 20 can be a server that provides cloud gaming services to the rendering device 200 via network 10 in the form of video streaming. For example, the server could be a cloud gaming service platform, such as Google's Stadia or NVIDIA's GeForce Now. Games on these platforms run on the server, and the game visuals are encoded using specific video compression techniques, with the encoded data transmitted to the rendering device 200 via network 10. Alternatively, the video source 20 can also be a wireless projection (or wireless display) source device, such as a mobile device including a smartphone or tablet, or any device that supports wireless projection or wireless display. This wireless projection source device can provide video content from its location to the rendering device 200 via network 10 based on wireless projection or wireless display protocols such as Miracast, AirPlay, and Chromecast. For example, it can transmit real-time screen content from a mobile phone / tablet, or local or online video content playing on a mobile phone / tablet, to the rendering device 200 via the AirPlay protocol, or share video content from a specific application.
[0016] In one embodiment, the transmission interface 300 can transmit data based on a specific data transmission standard, such as High Definition Multimedia Interface (HDMI), DisplayPort (DP), or Thunderbolt. These interfaces can all support the transmission of data from the rendering device 200 to the display device 100, including video data, audio data, and control commands. Specifically, the transmission interface 300 allows the rendering device 200 to send specific control commands to dynamically change the refresh rate of the display device 100. To achieve dynamic refresh rate adjustment, the rendering device 200 can employ various VRR technologies. For example, NVIDIA's G-SYNC and AMD's FreeSync technologies send refresh rate adjustment commands to the display device 100, requesting it to change to a specific refresh rate, such as various refresh rates between 48 Hz and 240 Hz.
[0017] The rendering apparatus 200 includes a streaming service processing unit 210, a video decoding unit 220, a video rendering unit 230, a storage unit 240, an interface transmission unit 250, and a refresh rate control unit 260. The streaming service processing unit 210 sends a request to the video source 20 via the network 10, thereby receiving the video stream provided by the video source 20 via the network 10. The video decoding unit 220 decodes the video stream to generate decoded video frames. The decoded video frames generated by the video decoding unit 220 are written to the storage unit 240. The storage unit 240 may be a Dynamic Random Access Memory (DRAM). The video rendering unit 230 selects the next suitable decoded video frame from the storage unit 240 and passes it to the interface transmission unit 250 to convert it into a signal conforming to a specific transmission specification. In one embodiment, the interface transmission unit 250 may be an HDMI transmission unit. The interface transmission unit 250 transmits video data, synchronization signals, and control signals to the display device 100 according to the Transition Minimized Differential Signaling (TMDS) timing.
[0018] During the Vertical Blank Interval (VBI), such as the Control Period in the TMDS timing of HDMI, the video rendering unit 230 selects the next suitable decoded video frame from the storage unit 240 and hands it over to the display device 100 for display. On the other hand, the time it takes for each decoded video frame to be written to the storage unit 240 is recorded as a Video Frame Ready Time (FRT) (e.g., in the form of a timestamp). When a decoded video frame is selected by the video rendering unit 230, the expected display time of the decoded video frame can be determined based on the timing of the vertical synchronization signal of the display device 100, which can be used as the Video Frame Display Time (FDT) of the decoded video frame. The time difference between the Video Frame Ready Time (FRT) and the Video Frame Display Time (FDT) can be determined as a Video Frame Delay (FL).
[0019] Furthermore, the refresh rate adjustment unit 260 changes the refresh rate of the display device 100 based on the video display latency. Specifically, the refresh rate adjustment unit 260 enables the VRR function of the display device via the interface transmission unit 250 and commands the interface transmission unit 250 to change the timing (frequency) of the VSYNC signal, thereby changing the refresh rate of the display device 100 to shorten the video display latency. For details on how the refresh rate adjustment unit 260 changes the refresh rate of the display device 100 based on the video display latency, please refer to the following embodiment.
[0020] Please refer to Figure 2 In the example shown in the figure, assume that the frame rate of the video stream is 60 fps (i.e., 60 Hz) during a certain time period. Simultaneously, the refresh rate of the display device 100 is also 60 Hz. As can be seen from the figure, there is a desynchronization between the vertical sync signal VSYNC_S of the video source 20 and the vertical sync signal VSYNC_D of the display device 100; that is, the vertical sync signal offset VSYNC_Offset is greater than 0. The refresh rate control unit 260 calculates that VSYNC_Offset is greater than 0 (or greater than a threshold value) and adjusts the refresh rate of the display device 100 using VRR technology, reducing the refresh rate of the display device 100 from the preset refresh rate of 60 Hz to 59.94 Hz. Afterward, the refresh rate control unit 260 continues to calculate the vertical sync signal offset VSYNC_Offset. After a period of time, when the vertical sync signal offset VSYNC_Offset continues to change to below a preset time difference, the refresh rate control unit 260 will again adjust the refresh rate of the display device 100 back to the preset refresh rate of 60 Hz based on VRR technology. In some embodiments, when the refresh rate control unit 260 calculates that the vertical sync signal offset VSYNC_Offset is below the preset time difference, the refresh rate control unit 260 will not immediately adjust the refresh rate of the display device 100 back to the preset refresh rate of 60 Hz, but will gradually adjust the refresh rate of the display device 100 from 59.94 Hz back to 60 Hz over a period of time.
[0021] exist Figure 3The example shown assumes that the video stream's frame rate is 60 fps (i.e., 60 Hz) during a certain time period. Simultaneously, the refresh rate of the display device 100 is also 60 Hz. However, as can be seen from the diagram, there is a desynchronization between the vertical sync signal VSYNC_S of the video source 20 and the vertical sync signal VSYNC_D of the display device 100; that is, the vertical sync signal offset VSYNC_Offset is greater than 0. When the refresh rate control unit 260 calculates that VSYNC_Offset is greater than 0 (or greater than a threshold value), it will use VRR technology to adjust the refresh rate of the display device 100, increasing the refresh rate of the display device 100 from the preset refresh rate of 60 Hz to 65 Hz. Afterward, the refresh rate control unit 260 will continue to calculate the vertical sync signal offset VSYNC_Offset. After a period of time, when the vertical sync signal offset VSYNC_Offset continues to change to below a preset time difference, the refresh rate control unit 260 will again adjust the refresh rate of the display device 100 back to the preset refresh rate of 60 Hz based on VRR technology. In some embodiments, when the refresh rate control unit 260 calculates that the vertical sync signal offset VSYNC_Offset is below the preset time difference, the refresh rate control unit 260 will not immediately adjust the refresh rate of the display device 100 back to the preset refresh rate of 60 Hz, but will gradually adjust the refresh rate of the display device 100 from 65 Hz back to 60 Hz over a period of time.
[0022] In the above embodiments, the refresh rate control unit 260 controls the refresh rate of the display device 100 by calculating the offset (i.e., the vertical sync signal offset VSYNC_Offset) between the vertical sync signal VSYNC_S of the detected video source 20 and the vertical sync signal VSYNC_D of the display device 100. However, in the following embodiments, the refresh rate control unit 260 adjusts the refresh rate of the display device 100 based on statistical data.
[0023] exist Figure 4 In the example shown, the refresh rate control unit 260 can record the timestamps of the video frames from the time they are decoded to the time they are sent to the display device 100 for display, in order to obtain relevant statistical data. Furthermore, the refresh rate control unit 260 calculates an average video frame latency FLA (which can be in milliseconds) based on the video frame latency FL (i.e., statistical data (A)) corresponding to a specific number (or a time period) of decoded video frames. In one embodiment, this specific number may be 1 to 120; that is, the refresh rate control unit 260 may calculate the average video frame latency FLA based on the video frame latency FL corresponding to 1 to 120 decoded video frames.
[0024] Furthermore, when the average video frame latency (FLA) exceeds a preset video latency threshold (e.g., FLT), the refresh rate control unit 260 determines that the latency is too high. At this time, the refresh rate control unit 260 uses VRR technology to adjust the refresh rate of the display device 100 from 60 Hz to 65 Hz. Afterward, the refresh rate control unit 260 continuously calculates the average video frame latency (FLA) (i.e., statistical data (B)). After a period of time, when the average video frame latency (FLA) falls below the video latency threshold (FLT), the refresh rate control unit 260 adjusts the refresh rate of the display device 100 back to 60 Hz via the interface transmission unit 250 (or gradually adjusts it back over a period of time). In one embodiment, the refresh rate control unit 260 can determine whether to increase or decrease the refresh rate of the display device 100 based on the magnitude of the latency. For example, in a 60 fps video stream, the average video frame latency (FLA) may be between 0 and 16.67 ms. When the average video frame latency (FLA) falls between 0 and FLT ms, the refresh rate control unit 260 may choose to decrease the refresh rate of the display device 100 to increase the average video frame latency (FLA); conversely, when the average video frame latency (FLA) falls between FLT and 16.67 ms, the refresh rate control unit 260 may choose to increase the refresh rate of the display device 100 to shorten the average video frame latency (FLA). In one embodiment, when the frame rate of the video stream is 60 fps and the maximum supported refresh rate of the display device 100 is 60 Hz, the refresh rate control unit 260 can only adjust the average video frame latency (FLA) by decreasing the refresh rate of the display device 100.
[0025] In one embodiment, the refresh rate control unit 260 sets a target video frame delay (which may be in milliseconds). For example, the video delay threshold FLT can be set as the target video frame delay. Furthermore, the refresh rate control unit 260 also sets a VRR time period T, which represents the duration for which VRR control (increasing or decreasing the initial refresh rate) is applied to the display device 100. The target video frame delay FLT represents the expected average video frame delay after VRR control is applied within the VRR time period T. Based on the average video frame delay FLA, the target video frame delay FLT, and the VRR time period T, the refresh rate control unit 260 can determine the target refresh rate VrrF. That is, the display device 100 is subjected to VRR control, adjusting its operating refresh rate from the initial refresh rate F to the target refresh rate VrrF. The target refresh rate VrrF can be determined by the following formula: VrrF = 1000 / ((1000 / F) - (FLA – FLT) / T). Furthermore, the VRR control time T can be determined by the following formula: T = (F... VrrF (FLA–FLT)) / (1000 (VrrF–F)). In this way, the rendering device 200 can more proactively and efficiently control the display device 100 via VRR, thereby reducing video display latency.
[0026] Figure 5 A method for displaying a video stream on a display device with variable refresh rate capability is illustrated. As shown in the figure, the method of the present invention includes the following simplified process: Step S310: Receive the video stream and decode the video stream to generate multiple decoded video frames; Step S320: Determine the preparation time of a video frame corresponding to each of the multiple decoded video frames; Step S330: Determine the display time of a video frame corresponding to each of the multiple decoded video frames; Step S340: Determine a target refresh rate for the display device based on a video frame delay between the video frame preparation time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and Step S350: Control the display device to display the video stream at the target refresh rate.
[0027] Since the principles and details of the above steps have been explained in detail through the above embodiments, they will not be repeated here. It is worth noting that the above process can be improved by adding other additional steps or making appropriate modifications and adjustments to better improve the quality of video display and reduce latency.
[0028] Embodiments of the present invention can be embodied as apparatus, method, or computer program product. Accordingly, embodiments of the present invention can take the form of an entity entirely implemented in hardware, an entity entirely implemented in software (including firmware, resident software, microcode, etc.), or an entity combining software and hardware aspects, which can be collectively referred to as a "module" or "system". Furthermore, embodiments of the present invention can take the form of a computer program product embodied in any tangible medium having computer-usable program code embodied therein. In terms of hardware, the present invention can be implemented by applying any of the following techniques or related combinations: individual operating logic of logic gates capable of executing logic functions according to data signals, and application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), or field-programmable gate arrays (FPGAs) with appropriate combinational logic.
[0029] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of systems, methods, and computer program products for different possible implementations of embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of program code, including one or more executable instructions to implement a specific logical function. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in block diagrams and / or flowcharts, can be implemented by a special-purpose hardware infrastructure system, or a combination of special hardware and computer program instructions. These computer program instructions may be stored in a readable computer medium to command a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the readable computer medium produce the functions / operations specified in the blocks or combinations of blocks in the flowcharts and / or block diagrams.
[0030] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made in accordance with the claims of the present invention shall be covered by the present invention.
[0031] [Symbol Explanation]
[0032] 10 Network
[0033] 20 Video sources
[0034] 100 display devices
[0035] 200 rendering devices
[0036] 210 Streaming Service Processing Unit
[0037] 220 video decoding units
[0038] 230 video rendering units
[0039] 240 storage units
[0040] 250 Interface Transmission Unit
[0041] 260Hz refresh rate control unit
[0042] 300 Transmission Interface.
Claims
1. A method for displaying a video stream on a display device with variable refresh rate functionality, comprising: The video stream is received and decoded to generate multiple decoded video frames. Determine the preparation time of a video frame corresponding to each of the multiple decoded video frames; Determine the display time of a video frame corresponding to each of the multiple decoded video frames; A target refresh rate of the display device is determined based on a video frame delay between the video frame preparation time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and Control the display device to display the video stream at the target refresh rate. wherein The steps for determining the target refresh rate of the display device include: Based on the video frame delays between the preparation time and display time of a specific number of video frames in the multiple decoded video frames, an average video frame delay is determined; and The target refresh rate is determined based on the average video frame latency, a target video frame latency, and a variable refresh rate control period. The variable refresh rate control period represents the duration for which the display device undergoes variable refresh rate control, and the target video frame latency represents the expected average video frame latency achievable after variable refresh rate control is applied within the specified time period. The target refresh rate VrrF is determined by the following relationship: VrrF=1000 / ((1000 / F)-(FLA-FLT) / T), where F is the initial refresh rate of the display device before variable refresh rate control, FLA is the average latency of the video frame, FLT is the target latency of the video frame, and T is the variable refresh rate control time period.
2. The method of claim 1, further comprising: When the video frame delay exceeds a predetermined value, the display device is controlled to operate at the target refresh rate; and When the video frame delay is less than the predetermined value, the display device is controlled to operate at a preset refresh rate.
3. The method as described in claim 2, wherein, The steps for determining the target refresh rate of the display device include: When a preset refresh rate of the display device corresponds to the frame rate of the video stream and the video frame delay is greater than the preset value, a refresh rate lower than the preset refresh rate is used as the target refresh rate.
4. The method of claim 2, wherein, The steps for determining the target refresh rate of the display device include: When the video frame delay falls within a first numerical range, a refresh rate lower than the preset refresh rate is used as the target refresh rate. as well as When the video frame delay falls within a second numerical range, a refresh rate higher than the preset refresh rate is used as the target refresh rate.
5. The method of claim 1, wherein, The video stream is provided by a video source, which may be a cloud gaming service platform or a wireless projection source device.
6. The method of claim 1, wherein, The video frame preparation time of one of the plurality of decoded video frames corresponds to the time when the decoded video frame is written to a storage unit of a rendering device, and the video frame display time of the decoded video frame corresponds to the timing of the vertical synchronization signal of the decoded video frame on the display device.
7. A rendering apparatus for controlling a display device having a variable refresh rate function, comprising: A decoding unit is used to receive a video stream and decode the video stream to generate multiple decoded video frames; and A refresh rate control unit, coupled to the decoding unit, is used to determine the video frame preparation time corresponding to each of the plurality of decoded video frames and the video frame display time corresponding to each of the plurality of decoded video frames, and is used to determine a target refresh rate of the display device based on a video frame delay between the video frame preparation time and the video frame display time corresponding to at least one of the plurality of decoded video frames. in, The rendering device is used to control the display device to display the video stream at the target refresh rate. The step of determining the target refresh rate of the display device includes: Based on the video frame delays between the preparation time and display time of a specific number of video frames in the multiple decoded video frames, an average video frame delay is determined; and The target refresh rate is determined based on the average video frame latency, a target video frame latency, and a variable refresh rate control period. The variable refresh rate control period represents the duration for which the display device undergoes variable refresh rate control, and the target video frame latency represents the expected average video frame latency achievable after variable refresh rate control is applied within the specified time period. The target refresh rate VrrF is determined by the following relationship: VrrF=1000 / ((1000 / F)-(FLA-FLT) / T), where F is the initial refresh rate of the display device before variable refresh rate control, FLA is the average latency of the video frame, FLT is the target latency of the video frame, and T is the variable refresh rate control time period.
8. The rendering apparatus of claim 7, wherein, This rendering device is used for: When the video frame delay exceeds a predetermined value, the display device is controlled to operate at the target refresh rate; and When the video frame delay is less than the predetermined value, the display device is controlled to operate at a preset refresh rate.
9. The rendering apparatus of claim 8, wherein, This refresh rate control unit: When a preset refresh rate of the display device corresponds to the frame rate of the video stream and the video frame delay is greater than the preset value, a refresh rate lower than the preset refresh rate is used as the target refresh rate.
Citation Information
Patent Citations
Method and device for reducing video latency
US9984653B1
