Method for shortening display delay based on variable refresh rate technology and related rendering device

Abstract

The invention discloses a method for shortening display delay based on a variable refresh rate technology and a related rendering device. The method for displaying a video stream on a display device with a variable refresh rate function comprises the following steps of: receiving the video stream and decoding the video stream so as to generate a plurality of decoded video frames; determining a video frame ready time corresponding to each of the plurality of decoded video frames; determining a video frame display time corresponding to each of the plurality of decoded video frames; determining a target refresh rate of the display device according to a video frame delay between the video frame ready time corresponding to at least one of the plurality of decoded video frames and the video frame display time; and controlling the display device to display the video stream at the target refresh rate.

Description

Technical Field

[0001] The present invention relates to display technology, and more particularly to a rendering device related to a method for reducing video display delay through variable refresh rate technology. Background Art

[0002] Nowadays, with the popularity of electronic products and cloud / streaming games (Cloud Gaming), people have higher and higher requirements for the low-latency experience of cloud games. In the process of operating cloud / streaming games, if the game operation feedback deviates from the synchronization of audio / video, the gaming experience will be reduced. Therefore, it is very important to shorten the video delay of cloud / streaming games. There are two main methods to reduce video display delay: one is to achieve synchronization by adjusting the number of video caches, and the other is to increase the video refresh rate to speed up the video update. However, both methods have certain defects. The method of adjusting the number of caches is prone to frame drops, while the method of increasing the video refresh rate will increase the computing power of the cloud / streaming game server and the video decoding computing power of the playback end. Therefore, this field needs an innovative method to improve video display delay. Summary of the invention

[0003] In view of this, the present invention proposes a mechanism for shortening video display delay by using variable refresh rate technology. In the implementation of the present invention, the video frame delay corresponding to the video frame will be counted, and the refresh rate of the display device will be dynamically adjusted according to the display device's support capability for variable refresh rate, thereby shortening the display delay of the video stream on the display device.

[0004] An embodiment of the present invention provides a method for displaying a video stream on a display device with a variable refresh rate function. The method includes: receiving the video stream and decoding the video stream to generate a plurality of decoded video frames; determining a video frame ready time corresponding to each of the plurality of decoded video frames; determining a video frame display time corresponding to each of the plurality of decoded video frames; determining a target refresh rate of the display device according to a video frame delay between the video frame ready time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and controlling the display device to display the video stream at the target refresh rate.

[0005] Another embodiment of the present invention provides a rendering device for controlling a display device with a variable refresh rate function. The rendering device includes: a decoding unit and a refresh rate control unit. The decoding unit is used to receive a video stream and decode the video stream to generate a plurality of decoded video frames. The refresh rate control unit is coupled to the decoding unit, and is used to determine a video frame ready time corresponding to each of the plurality of decoded video frames and a video frame display time corresponding to each of the plurality of decoded video frames, and is used to determine a target refresh rate of the display device according to a video frame delay between the video frame ready time and the video frame display time corresponding to at least one of the plurality of decoded video frames. The rendering device is used to control the display device to display the video stream at the target refresh rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 FIG. 4 is a schematic diagram of a rendering device and its application architecture according to an embodiment of the present invention.

[0007] Figure 2 It illustrates how to shorten the video display delay according to one embodiment of the present invention.

[0008] Figure 3 Another embodiment of the present invention is shown to illustrate how to shorten the video display delay.

[0009] Figure 4 It illustrates how to shorten the video display delay according to another embodiment of the present invention.

[0010] Figure 5 The figure is a flow chart of a method for controlling a display device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In the following text, many specific details are described to provide readers with a thorough understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention in the absence of one or more specific details, or using other methods, elements, materials, etc. In other cases, well-known structures, materials, or operations will not be shown or described in detail to avoid obscuring the core concepts of the present invention.

[0012] The reference to "an embodiment" in the specification means that the specific features, structures or characteristics described in the embodiment may be included in at least one embodiment of the present invention. Therefore, the "in an embodiment" that appears in various places in this specification does not necessarily mean the same embodiment. In addition, the aforementioned specific features, structures or characteristics may be combined in one or more embodiments in any suitable form.

[0013] Figure 1 A schematic diagram of a rendering device and its application architecture according to an embodiment of the present invention is shown. Figure 1 As shown, the rendering device 200 receives a video stream from a video source 20 through a network 10 and decodes the video stream. In addition, the rendering device 200 sends a video signal to a display device 100 through a transmission interface 300 to control the display device 100 to display the video content after the video stream is decoded. In one embodiment, the rendering device 200 can be a streaming media player, a smart TV box, a game console, etc. Furthermore, the display device 100 can be various types of displays or TVs that support variable refresh rate (VRR) technology, such as liquid crystal display (LCD) / TV, organic light-emitting diode (OLED) display / TV, or Micro-LED display / TV, etc.

[0014] In one embodiment, the network 10 may be a wired network, a wireless network, or a hybrid network composed of the two. The wired network may include: Ethernet, modem lines, optical fiber networks, etc. The wired network also includes hardware devices that can provide wired network connection capabilities, including but not limited to: routers, switches, bridges, etc. These devices can connect various network nodes together and manage the exchange of data between these nodes. Furthermore, the wireless network may include: Wi-Fi, Bluetooth, mobile telecommunication networks (such as 4G LTE, 5G), satellite communication networks, etc. The wireless network also includes hardware devices that can provide wireless network connection capabilities, these hardware devices include but are not limited to: wireless routers, wireless access points (Access Point, AP), mobile network base stations (such as 4G base stations, 5G base stations), etc. These devices connect wireless devices together and manage the flow of data between these devices. In an embodiment of a hybrid network, the hardware devices of the wired network and the wireless network may be combined to provide a more flexible network connection. For example, a Wi-Fi wireless access point may be connected to a wired Ethernet network to allow Wi-Fi devices to access remote data through the Ethernet network.

[0015] In one embodiment, the video source 20 may be a server, which may be used to provide cloud gaming services to the rendering device 200 in the form of video streaming via the network 10. For example, the server may be a cloud gaming service platform, such as Google's Stadia or NVIDIA's GeForce Now, where the game programs on these platforms will run on the server, and the game screen will be encoded using a specific video compression technology, and the encoded data will be transmitted to the rendering device 200 via the network 10. On the other hand, the video source 20 may also be a wireless projection (or wireless display) source device, such as a mobile device including a smartphone or a tablet, or any device that supports wireless projection or wireless display functions. The wireless projection source device may provide the rendering device 200 with video content on the wireless projection source device via the network 10 based on wireless projection or wireless display protocols such as Miracast, AirPlay, and Chromecast. For example, the live screen content of a mobile phone / tablet, or the local or network video content being played on the mobile phone / tablet is transmitted to the rendering device 200 via the AirPlay protocol, or the video content shared in a specific application.

[0016] In one embodiment, the transmission interface 300 can transmit data based on a specific data transmission standard, such as High Definition Multimedia Interface (HDMI), DisplayPort (DP) or Thunderbolt. These interfaces can support the transmission of data from the rendering device 200 to the display device 100, including video data, audio data, control instructions, etc. Among them, the transmission interface 300 allows the rendering device 200 to transmit specific control instructions to dynamically change the refresh rate of the display device 100. In order to achieve dynamic adjustment of the refresh rate, the rendering device 200 can adopt various VRR technologies. For example, NVIDIA's G-SYNC and AMD's FreeSync technology send a refresh rate adjustment command to the display device 100, requiring the display device 100 to change to a specific refresh rate. For example, various refresh rates between 48Hz and 240Hz.

[0017] The rendering device 200 includes a stream service processing unit 210, a video decoding unit 220, a video rendering unit 230, a storage unit 240, an interface transmission unit 250 and a refresh rate control unit 260. The stream service processing unit 210 is used to send a request to the video source 20 through the network 10, so as to receive the video stream provided by the video source 20 through the network 10. The video decoding unit 220 is used to decode the video stream to generate a decoded video frame. The decoded video frame generated by the video decoding unit 220 will be written to the storage unit 240. Among them, the storage unit 240 can be a dynamic random access memory (DRAM). The video rendering unit 230 is used to select the next suitable decoded video frame from the storage unit 240, and transmit it to the interface transmission unit 250 to convert it into a signal that meets a specific transmission specification. In one embodiment, the interface transmission unit 250 can be an HDMI transmission unit. The interface transmission unit 250 transmits the video data, the synchronization signal and the control signal to the display device 100 according to the Transition Minimized Differential Signaling (TMDS) timing.

[0018] The video rendering unit 230 selects the next appropriate decoded video frame from the storage unit 240 during the vertical blanking interval (VBI), for example, the control period in the TMDS timing of HDMI, and hands it over to the display device 100 for display. On the other hand, the time when each decoded video frame is written into the storage unit 240 will be recorded as a video frame ready time FRT (e.g., in the form of a timestamp). When a decoded video frame is selected by the video rendering unit 230, the time when the decoded video frame is expected to be displayed can be determined based on the timing of the vertical synchronization signal of the display device 100, which can be used as a video frame display time FDT of the decoded video frame. The time difference between the video frame ready time FRT and the video frame ready time FDT can be determined as a video frame delay FL.

[0019] Furthermore, the refresh rate adjustment unit 260 changes the refresh rate of the display device 100 according to the video display delay. The refresh rate adjustment unit 260 turns on the VRR function of the display device through the interface transmission unit 250, and commands the interface transmission unit 250 to change the timing (frequency) of the VSYNC signal, thereby changing the refresh rate of the display device 100 to shorten the video display delay. Regarding how the refresh rate adjustment unit 260 changes the refresh rate of the display device 100 according to the video display delay, please refer to the following embodiments.

[0020] Please refer to Figure 2 In the example shown in the figure, it is assumed that in a time period, the frame rate (FrameRate) of the video stream is 60fps (i.e., 60Hz). At the same time, the refresh rate (Refresh Rate) of the display device 100 is also 60Hz. It can be seen from the figure that there is an asynchrony between the vertical synchronization signal VSYNC_S of the video source 20 and the vertical synchronization signal VSYNC_D of the display device 100, that is, the vertical synchronization signal offset VSYNC_Offset is greater than 0. The refresh rate control unit 260 calculates that VSYNC_Offset is greater than 0 (or greater than a critical value), and adjusts the refresh rate of the display device 100 using VRR technology, so that the refresh rate of the display device 100 is reduced from the preset refresh rate of 60Hz to 59.94Hz. Afterwards, the refresh rate control unit 260 will continue to calculate the vertical synchronization signal offset VSYNC_Offset. After a period of time, when the vertical synchronization signal offset VSYNC_Offset continues to change to be lower than a preset time difference, the refresh rate control unit 260 will adjust the refresh rate of the display device 100 back to the preset refresh rate of 60Hz based on the VRR technology again. In some embodiments, when the refresh rate control unit 260 calculates that the vertical synchronization signal offset VSYNC_Offset is lower than the preset time difference, the refresh rate control unit 260 will not immediately adjust the refresh rate of the display device 100 back to the preset refresh rate of 60Hz, but will gradually adjust the refresh rate of the display device 100 from 59.94Hz back to 60Hz within a period of time.

[0021] exist Figure 3In the example shown, it is assumed that in a time period, the frame rate of the video stream is 60fps (i.e., 60Hz). At the same time, the refresh rate of the display device 100 is also 60Hz. However, it can be seen from the figure that there is an asynchronization between the vertical synchronization signal VSYNC_S of the video source 20 and the vertical synchronization signal VSYNC_D of the display device 100, that is, the vertical synchronization signal offset VSYNC_Offset is greater than 0. When the refresh rate control unit 260 calculates that VSYNC_Offset is greater than 0 (or greater than a critical value), the refresh rate of the display device 100 will be adjusted using the VRR technology, so that the refresh rate of the display device 100 is increased from the preset refresh rate of 60Hz to 65Hz. After that, the refresh rate control unit 260 will continue to calculate the vertical synchronization signal offset VSYNC_Offset. After a period of time, when the vertical synchronization signal offset VSYNC_Offset continues to change to be lower than a preset time difference, the refresh rate control unit 260 will adjust the refresh rate of the display device 100 back to the preset refresh rate of 60Hz based on the VRR technology again. In some embodiments, when the refresh rate control unit 260 calculates that the vertical synchronization signal offset VSYNC_Offset is lower than the preset time difference, the refresh rate control unit 260 does not immediately adjust the refresh rate of the display device 100 back to the preset refresh rate of 60 Hz, but slowly adjusts the refresh rate of the display device 100 from 65 Hz back to 60 Hz over a period of time.

[0022] In the above embodiment, the refresh rate control unit 260 calculates and detects the offset (i.e., vertical synchronization signal offset VSYNC_Offset) between the vertical synchronization signal VSYNC_S of the video source 20 and the vertical synchronization signal VSYNC_D of the display device 100 to control the refresh rate of the display device 100. However, in the following embodiment, the refresh rate control unit 260 adjusts the refresh rate of the display device 100 based on statistical data.

[0023] exist Figure 4 In the example shown, the refresh rate control unit 260 may record the timestamps of the video frame from the time it is decoded to the time it is sent to the display device 100 for display, so as to obtain relevant statistical data. Further, the refresh rate control unit 260 may calculate an average video frame delay FLA (in ms) based on the video frame delays FL (i.e., statistical data (A)) corresponding to a specific number (or within a time period) of decoded video frames. In one embodiment, the specific number may be 1 to 120, that is, the refresh rate control unit 260 may calculate the average video frame delay FLA based on the video frame delays FL corresponding to 1 to 120 decoded video frames.

[0024] Furthermore, when the average delay FLA of the video frame is greater than a preset video delay threshold value (e.g., FLT), the refresh rate control unit 260 determines that the delay is too high. At this time, the refresh rate control unit 260 uses VRR technology to adjust the refresh rate of the display device 100 from 60Hz to 65Hz. After that, the refresh rate control unit 260 continues to calculate the average delay FLA of the video frame (i.e., statistical data (B)). After a period of time, when the average delay FLA of the video frame is lower than the video delay threshold value FLT, the refresh rate control unit 260 adjusts the refresh rate of the display device 100 back to 60Hz (or adjusts it back slowly within a period of time) through the interface transmission unit 250. In one embodiment, the refresh rate control unit 260 can decide whether to increase or decrease the refresh rate of the display device 100 based on the amount of delay. Taking a 60fps video stream as an example, the average delay FLA of the video frame may be between 0 and 16.67ms. When the average video frame delay FLA falls between 0 and FLT ms, the refresh rate control unit 260 may choose to reduce the refresh rate of the display device 100 to increase the average video frame delay FLA; and when the average video frame delay FLA falls between FLT and 16.67 ms, the refresh rate control unit 260 may choose to increase the refresh rate of the display device 100 to shorten the average video frame delay FLA. In one embodiment, when the frame rate of the video stream is 60fps and the maximum refresh rate supported by the display device 100 is 60Hz, the refresh rate control unit 260 can only adjust the average video frame delay FLA by reducing the refresh rate of the display device 100.

[0025] In one embodiment, the refresh rate control unit 260 sets a video frame target delay (the unit may be ms), for example, the video delay threshold FLT may be set as the video frame target delay. Furthermore, the refresh rate control unit 260 also sets a VRR time period T, which represents the length of time for VRR control (increasing or decreasing the initial refresh rate) of the display device 100. The video frame target delay FLT represents the average video frame delay that is expected to be achieved after VRR control is performed within the VRR time period T. Based on the video frame average delay FLA, the video frame target delay FLT and the VRR time period T, the refresh rate control unit 260 may determine the target refresh rate VrrF. That is, the display device 100 is controlled by VRR, and its operating refresh rate is adjusted from the initial refresh rate F to the target refresh rate VrrF. The target refresh rate VrrF can be determined by the following formula, namely, VrrF=1000 / ((1000 / F)-(FLA–FLT) / T), and the VRR control time T can be determined by the following formula, namely, T=(F*VrrF*(FLA–FLT)) / (1000*(VrrF–F)). In the above manner, the rendering device 200 can more actively and efficiently perform VRR control on the display device 100, thereby shortening the video display delay.

[0026] Figure 5 A method for displaying a video stream on a display device with a variable refresh rate function is shown. As shown in the figure, the method of the present invention includes the following simplified process:

[0027] Step S310: receiving the video stream and decoding the video stream to generate a plurality of decoded video frames;

[0028] Step S320: Determine a video frame ready time corresponding to each of the plurality of decoded video frames;

[0029] Step S330: Determine a video frame display time corresponding to each of the plurality of decoded video frames;

[0030] Step S340: determining a target refresh rate of the display device according to a video frame delay between the video frame ready time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and

[0031] Step S350: Control the display device to display the video stream at the target refresh rate.

[0032] Since the principles and details of the above steps have been described in detail through the above embodiments, they will not be repeated here. It is worth noting that the above process can be improved by adding other additional steps or making appropriate modifications and adjustments to better improve the quality of video display and reduce delay.

[0033] The embodiments of the present invention may be specifically implemented as an apparatus, method or computer program product. Accordingly, the embodiments of the present invention may take the form of an entity implemented entirely by hardware, an entity implemented entirely by software (including firmware, resident software, microcode, etc.), or an entity combining software and hardware aspects, which may be generally referred to as a "module" or "system". In addition, the embodiments of the present invention may take the form of a computer program product embodied in any tangible expression medium, and the medium has a computer-usable program code embodied in the medium. In terms of hardware, the present invention may be implemented by applying any of the following technologies or related combinations: separate operation logic of logic gates capable of executing logic functions according to data signals, and an application specific integrated circuit (ASIC), a programmable gate array (PGA) or a field programmable gate array (FPGA) with appropriate combinational logic.

[0034] The flow chart and block diagram illustrate the architecture, function and operation of the system, method and computer program product of different possible implementation methods of the embodiment of the present invention. In this regard, each block in the flow chart or block diagram may represent a module, section or a portion of the program code, including one or more executable instructions to implement a specific logic function. It should also be noted that each block in the block diagram and / or flow chart, and the block combination in the block diagram and / or flow chart can be implemented by a special-purpose hardware-based system, or a combination of special hardware and computer program instructions. These computer program instructions can be stored in a readable computer medium to command a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the readable computer medium generate functions / operations specified in the blocks or block combinations that include implementing the flow chart and / or block diagram.

[0035] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

[0036]

Explanation of symbols

[0037] 10 Network

[0038] 20 Video Sources

[0039] 100 Display device

[0040] 200 Rendering Device

[0041] 210 Streaming service processing unit

[0042] 220 Video decoding unit

[0043] 230 Video Rendering Unit

[0044] 240 storage units

[0045] 250 Interface Transmission Unit

[0046] 260 refresh rate control unit

[0047] 300 transmission interface.

Claims

1. A method for displaying a video stream on a display device having a variable refresh rate function, comprising: receiving the video stream and decoding the video stream to generate a plurality of decoded video frames; Determining a video frame ready time corresponding to each of the plurality of decoded video frames; Determining a video frame display time corresponding to each of the plurality of decoded video frames; Determining a target refresh rate of the display device according to a video frame delay between the video frame ready time and the video frame display time corresponding to at least one of the plurality of decoded video frames; and The display device is controlled to display the video stream at the target refresh rate.

2. The method of claim 1, further comprising: When the video frame delay is greater than a predetermined value, controlling the display device to operate at the target refresh rate; and When the video frame delay is less than the predetermined value, the display device is controlled to operate at a preset refresh rate.

3. The method of claim 2, wherein: The step of determining the target refresh rate of the display device comprises: When a preset refresh rate of the display device corresponds to the frame rate of the video stream and the video frame delay is greater than the predetermined value, a refresh rate lower than the preset refresh rate is used as the target refresh rate.

4. The method of claim 2, wherein: The step of determining the target refresh rate of the display device comprises: When the video frame delay falls within a first value interval, a refresh rate lower than the preset refresh rate is used as the target refresh rate; as well as When the video frame delay falls within a second value interval, a refresh rate higher than the preset refresh rate is used as the target refresh rate.

5. The method of claim 1, wherein: The step of determining the target refresh rate of the display device comprises: Determine an average video frame delay according to a plurality of video frame delays between the video frame ready time and the video frame display time respectively corresponding to a specific number of video frames among the plurality of decoded video frames; and The target refresh rate is determined according to the video frame average delay, a video frame target delay and a variable refresh rate control time period.

6. The method of claim 1, wherein: The video stream is provided by a video source, which can be a cloud game service platform or a wireless projection source device.

7. The method of claim 1, wherein: The video frame ready time of a decoded video frame among the multiple decoded video frames corresponds to the time when the decoded video frame is written to a storage unit of a rendering device, and the video frame display time of the decoded video frame corresponds to the timing of the vertical synchronization signal of the decoded video frame on the display device.

8. A rendering device for controlling a display device having a variable refresh rate function, comprising: a decoding unit, configured to receive a video stream and decode the video stream to generate a plurality of decoded video frames; and a refresh rate control unit, coupled to the decoding unit, for determining a video frame ready time corresponding to each of the plurality of decoded video frames and a video frame display time corresponding to each of the plurality of decoded video frames, and for determining a target refresh rate of the display device according to a video frame delay between the video frame ready time and the video frame display time corresponding to at least one of the plurality of decoded video frames; in, The rendering device is used to control the display device to display the video stream at the target refresh rate.

9. The rendering device according to claim 8, wherein: The rendering device is used to: When the video frame delay is greater than a predetermined value, controlling the display device to operate at the target refresh rate; and When the video frame delay is less than the predetermined value, the display device is controlled to operate at a preset refresh rate.

10. The rendering device according to claim 9, wherein: The refresh rate control unit: When a preset refresh rate of the display device corresponds to the frame rate of the video stream and the video frame delay is greater than the predetermined value, a refresh rate lower than the preset refresh rate is used as the target refresh rate.

Images