Display device, display control method, and storage medium

Abstract

The present disclosure relates to a display device, a display control method and a storage medium, and relates to the technical field of multimedia playing. The display device comprises a controller configured to: in response to a calibration instruction for video frame timestamps of respective video frames in a video stream, convert the respective video frame timestamps into first system timestamps respectively corresponding to the respective video frames; in the video stream, if it is determined that a vertical refresh period corresponding to an Mth video frame coincides with a vertical refresh period corresponding to any video frame after the Mth video frame, adjust first system timestamps respectively corresponding to the Mth video frame and the video frames after the Mth video frame based on a first system timestamp corresponding to a first video frame in the video stream and the vertical refresh period, to obtain second system timestamps respectively corresponding to the Mth video frame and the video frames after the Mth video frame; and render and display the Mth video frame and the video frames after the Mth video frame based on the second system timestamps.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia playback technology, and in particular to a display device, a display control method, and a storage medium. Background Technology

[0002] Currently, vertical synchronization is one way to solve screen tearing on terminal devices. That is, after the display finishes drawing a frame of video, it sends a vertical synchronization signal (V-sync signal) to the processor, and the processor begins to process the next video frame to send to the display.

[0003] However, when B-frames are present in a video frame, they need to rely on I-frames or P-frames that precede or follow them in the video stream for decoding. This can cause a delay in decoding time, disrupt the decoding time and display order of video frames, and cause the rendering time of a video frame to miss its display time, resulting in frame drops. Therefore, how to avoid frame drops when displaying video streams is an urgent problem to be solved. Summary of the Invention

[0004] To solve, or at least partially solve, the above-mentioned technical problems, this disclosure provides a display device, a display control method, and a storage medium. When it is determined that the decoding time of at least one video frame is delayed, if the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, then the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted so that the obtained second system time is later than the first system timestamp. When rendering the Mth video frame and the video frames after the Mth video frame based on the second system timestamp, it is equivalent to delaying the rendering and display time of the Mth video frame and the video frames after the Mth video frame, thus avoiding the loss of data frames due to the delay in the decoding time of the Mth video frame.

[0005] In a first aspect, this disclosure provides a display device, the display device comprising:

[0006] The controller is configured as follows:

[0007] In response to a calibration command for the video frame timestamps of each video frame in the video stream, the video frame timestamps of each video frame are converted into first system timestamps corresponding to each video frame. The video frame timestamps are used to indicate the display order of each video frame, and the first system timestamps are used to indicate the timestamps for rendering each video frame. The rendering of each video frame is completed within the vertical refresh cycle corresponding to the video frame.

[0008] In the video stream, if it is determined that the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, then the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period, respectively, to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame, respectively, wherein the second system timestamp corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame;

[0009] The system renders the Mth video frame and subsequent video frames based on the second system timestamp, and controls the display to show the Mth video frame and subsequent video frames.

[0010] In some embodiments of this disclosure, the controller is further configured to:

[0011] Before rendering the Mth video frame and subsequent video frames based on the second system timestamp, and before controlling the display to show the Mth video frame and subsequent video frames, the method further includes:

[0012] In the video frames after the Mth video frame, if it is determined that the vertical refresh period corresponding to the Nth video frame coincides with the vertical refresh period corresponding to any video frame after the Nth video frame, then the third system timestamp corresponding to the Nth video frame is determined, and the second system timestamp corresponding to the video frames after the Nth video frame is adjusted based on the third system timestamp of the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames after the Nth video frame respectively.

[0013] The system renders the Nth video frame and subsequent video frames based on the third system timestamp, and controls the display to show the Nth video frame and subsequent video frames.

[0014] In some embodiments of this disclosure, the controller is specifically configured as follows:

[0015] The step of converting the video frame timestamps of each video frame into a first system timestamp corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream includes:

[0016] For the first video frame in the video stream, the timestamp when the decoding of the first video frame is completed is used as the first system timestamp of the first video frame;

[0017] For each video frame after the first video frame in the video stream, the difference between the video frame timestamp of the video frame and the video frame timestamp of the first video frame is obtained, and the difference is added to the first system timestamp of the first video frame to obtain the first system timestamp corresponding to the video frame.

[0018] In some embodiments of this disclosure, the controller is specifically configured as follows:

[0019] The adjustment of the first system timestamps corresponding to the Mth video frame and subsequent video frames based on the first system timestamp of the first video frame in the video stream and the vertical refresh period to obtain the second system timestamps corresponding to the Mth video frame and subsequent video frames includes:

[0020] For the Mth video frame and each video frame in the subsequent video frames, the sum of the first system timestamp of the video frame and a preset number of vertical refresh cycles is obtained as the second system timestamp corresponding to the video frame.

[0021] In some embodiments of this disclosure, the controller is specifically configured as follows:

[0022] The process of determining the third system timestamp corresponding to the Nth video frame, and adjusting the second system timestamps corresponding to the video frames following the Nth video frame based on the third system timestamp of the Nth video frame and the vertical refresh period, to obtain the third system timestamps corresponding to the video frames following the Nth video frame respectively, includes:

[0023] For the Nth video frame, the decoding completion time of the Nth video frame is used as the third system timestamp of the Nth video frame;

[0024] For video frames after the Nth video frame, the sum of the third system timestamp of the Nth video frame and a preset number of vertical refresh cycles is obtained as the third system timestamp corresponding to that video frame.

[0025] In some embodiments of this disclosure, the controller is further configured to:

[0026] After converting the video frame timestamps of each video frame into the first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes:

[0027] When the video stream is in a paused playback state, in response to the user's operation to end the pause playback of the video stream, the fourth system time of each video frame after the current video frame displayed on the display device is determined.

[0028] The system renders each video frame after the current video frame based on the timestamp of the fourth system, and controls the display to show each video frame after the current video frame.

[0029] In some embodiments of this disclosure, the controller is specifically configured as follows:

[0030] The step of determining the fourth system time of each video frame after the current video frame displayed on the display device in response to a user's pause playback operation for ending the video stream includes:

[0031] For each video frame following the current video frame, the fourth system timestamp corresponding to each frequency frame is obtained according to the fourth system time of the current video frame and the vertical refresh period.

[0032] In some embodiments of this disclosure, the controller is further configured to:

[0033] After converting the video frame timestamps of each video frame into the first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes:

[0034] In response to a user's fast-forward playback operation on the video stream, the fifth system time corresponding to the current video frame displayed on the display device is determined after the fast-forward playback operation ends;

[0035] For each video frame following the current video frame, the fifth system timestamp corresponding to each video frame is obtained according to the fifth system time of the current video frame and the vertical refresh period;

[0036] The current video frame and all subsequent video frames are rendered based on the fifth system timestamp, and the display is controlled to show the current video frame and all subsequent video frames.

[0037] Secondly, embodiments of this disclosure provide a display control method, including:

[0038] In response to a calibration command for the video frame timestamps of each video frame in the video stream, the video frame timestamps of each video frame are converted into first system timestamps corresponding to each video frame. The video frame timestamps are used to indicate the display order of each video frame, and the first system timestamps are used to indicate the timestamps for rendering each video frame. The rendering of each video frame is completed within the vertical refresh cycle corresponding to the video frame.

[0039] In the video stream, if it is determined that the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, then the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period, respectively, to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame.

[0040] The system renders the Mth video frame and subsequent video frames based on the second system timestamp, and controls the display to show the Mth video frame and subsequent video frames.

[0041] In some embodiments of this disclosure, before rendering the Mth video frame and subsequent video frames based on the second system timestamp, and before controlling the display to show the Mth video frame and subsequent video frames, the method further includes:

[0042] In the video frames after the Mth video frame, if it is determined that the vertical refresh period corresponding to the Nth video frame coincides with the vertical refresh period corresponding to any video frame after the Nth video frame, then the third system timestamp corresponding to the Nth video frame is determined, and the second system timestamp corresponding to the video frames after the Nth video frame is adjusted based on the third system timestamp of the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames after the Nth video frame respectively.

[0043] The system renders the Nth video frame and subsequent video frames based on the third system timestamp, and controls the display to show the Nth video frame and subsequent video frames.

[0044] In some embodiments of this disclosure, the step of converting the video frame timestamps of each video frame into a first system timestamp corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream includes:

[0045] For the first video frame in the video stream, the timestamp when the decoding of the first video frame is completed is used as the first system timestamp of the first video frame;

[0046] For each video frame after the first video frame in the video stream, the difference between the video frame timestamp of the video frame and the video frame timestamp of the first video frame is obtained, and the difference is added to the first system timestamp of the first video frame to obtain the first system timestamp corresponding to the video frame.

[0047] In some embodiments of this disclosure, adjusting the first system timestamps corresponding to the Mth video frame and subsequent video frames based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period to obtain the second system timestamps corresponding to the Mth video frame and subsequent video frames includes:

[0048] For the Mth video frame and each video frame in the subsequent video frames, the sum of the first system timestamp of the video frame and a preset number of vertical refresh cycles is obtained as the second system timestamp corresponding to the video frame.

[0049] In some embodiments of this disclosure, determining the third system timestamp corresponding to the Nth video frame, and adjusting the second system timestamp corresponding to the video frames following the Nth video frame based on the third system timestamp of the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames following the Nth video frame respectively, includes:

[0050] For the Nth video frame, the decoding completion time of the Nth video frame is used as the third system timestamp of the Nth video frame;

[0051] For video frames after the Nth video frame, the sum of the third system timestamp of the Nth video frame and a preset number of vertical refresh cycles is obtained as the third system timestamp corresponding to that video frame.

[0052] In some embodiments of this disclosure, after converting the video frame timestamps of each video frame into first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes:

[0053] In the video frames following the Mth video frame, if it is determined that the vertical refresh period corresponding to the Nth video frame coincides with the vertical refresh period corresponding to any video frame following the Nth video frame, then the third system timestamp corresponding to the Nth video frame is determined, and the second system timestamp corresponding to the video frames following the Nth video frame is adjusted based on the third system timestamp of the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames following the Nth video frame respectively.

[0054] In some embodiments of this disclosure, determining the fourth system time of each video frame after the current video frame displayed on the display device in response to a user's end-pause playback operation on the video stream includes:

[0055] For each video frame following the current video frame, the fourth system timestamp corresponding to each frequency frame is obtained according to the fourth system time of the current video frame and the vertical refresh period.

[0056] In some embodiments of this disclosure, after converting the video frame timestamps of each video frame into first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes:

[0057] In response to a user's fast-forward playback operation on the video stream, the fifth system time corresponding to the current video frame displayed on the display device is determined after the fast-forward playback operation ends;

[0058] For each video frame following the current video frame, the fifth system timestamp corresponding to each video frame is obtained according to the fifth system time of the current video frame and the vertical refresh period;

[0059] The current video frame and all subsequent video frames are rendered based on the fifth system timestamp, and the display is controlled to show the current video frame and all subsequent video frames.

[0060] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the display control method as described in the second aspect.

[0061] The display device provided in this disclosure has the following advantages compared with the prior art:

[0062] The controller, in response to a calibration command for the video frame timestamps of each video frame in the video stream, converts the video frame timestamps of each video frame into a first system timestamp corresponding to each video frame. The video frame timestamps indicate the display order of the video frames, and the first system timestamp indicates the timestamp for rendering each video frame. Rendering of each video frame is completed within its corresponding vertical refresh cycle. In the video stream, if it is determined that the vertical refresh cycle corresponding to the Mth video frame coincides with the vertical refresh cycle corresponding to any video frame after the Mth video frame, then based on the first system timestamp in the video stream... The first system timestamp corresponding to each video frame and the vertical refresh period are used to adjust the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame, respectively, to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame, respectively. The second system timestamp corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame. The Mth video frame and the video frames after the Mth video frame are rendered based on the second system timestamps, and the display is controlled to display the Mth video frame and the video frames after the Mth video frame. Since the second system time corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame, and the second system time corresponding to the video frames after the Mth video frame is later than the first system timestamp corresponding to that video frame, rendering the Mth video frame and the video frames after the Mth video frame based on the second system timestamp is equivalent to delaying the rendering and display time of the Mth video frame and the video frames after the Mth video frame, thus avoiding the loss of data frames when the display device displays the data stream due to the delay in the decoding time of the Mth video frame. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0064] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1A Application scenario diagrams provided for one or more embodiments of this disclosure;

[0066] Figure 1B A rendering diagram of a video frame provided in one or more embodiments of this disclosure;

[0067] Figure 2AThis is a hardware configuration block diagram of the display device 200 according to one or more embodiments of the present disclosure;

[0068] Figure 2B This is a schematic diagram of the software configuration in the display device 200 according to one or more embodiments of the present disclosure;

[0069] Figure 3 A flowchart illustrating a display control method provided in one or more embodiments of this disclosure;

[0070] Figure 4 A rendering diagram of a video frame provided in one or more embodiments of this disclosure;

[0071] Figure 5 A flowchart illustrating a display control method provided in one or more embodiments of this disclosure;

[0072] Figure 6 A rendering diagram of a video frame provided in one or more embodiments of this disclosure;

[0073] Figure 7 A flowchart illustrating a display control method provided in one or more embodiments of this disclosure;

[0074] Figure 8 This is a rendering diagram of a video frame provided in one or more embodiments of this disclosure. Detailed Implementation

[0075] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0076] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0077] The terms "first," "second," "third," etc., used in this disclosure, including the specification, claims, and drawings, are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement embodiments in a sequence other than those given in the illustrations or description of this application.

[0078] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0079] First, let's illustrate the application scenarios of this disclosure with examples:

[0080] Figure 1A This diagram illustrates an application scenario provided by one embodiment of the present disclosure. A user controls a display device 200 based on a control device 100, and the display device 200 is used to display a video stream. Currently, vertical synchronization is a method to solve screen tearing on terminal devices. That is, after the display finishes rendering one frame of video, a vertical synchronization signal (V-sync signal) is sent to the processor, and the processor begins processing the next video frame to send to the display.

[0081] However, when B-frames are present in the video stream, they require decoding based on I-frames or P-frames preceding or following them. This can cause decoding delays, resulting in some V-sync cycles receiving multiple video frames but only displaying the last frame, leading to the loss of other data frames. Conversely, some V-sync cycles may not receive any video frames, causing stuttering. For example, a video stream may include video frames F1, F2, F3, and F4, displayed in the order of F1, F2, F3, and F4. Figure 1B As shown, Figure 1B This is a rendering diagram of a video frame provided in one embodiment of the present disclosure. In the first V-sync cycle, two video frames, F1 and F2, are received. No video frames are received in the third and fourth V-sync cycles, which may cause video frame F1 to be lost. After video frame F3 is displayed, stuttering may occur on the display device. In this embodiment of the disclosure, V... n This indicates the nth V-sync cycle.

[0082] Based on this, this disclosure provides a display device in which, when it is determined that the decoding time of at least one video frame is delayed, if the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted so that the obtained second system time is later than the first system timestamp. When rendering the Mth video frame and the video frames after the Mth video frame based on the second system timestamp, it is equivalent to delaying the rendering and display time of the Mth video frame and the video frames after the Mth video frame, thus avoiding the phenomenon of data frame loss due to the delay in the decoding time of the Mth video frame.

[0083] In some embodiments, the display device can be a smart TV 200, which can be controlled by a control device, such as a remote control. Communication between the remote control and the display device includes infrared or Bluetooth communication, as well as other short-range communication methods, to control the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, or control panel input.

[0084] In some embodiments, the display device 200 may receive commands without using the control device or remote control described above, but may receive user control through touch or gestures.

[0085] In some embodiments, the display device 200 can also be controlled in ways other than control devices and remote controls. For example, it can be controlled by receiving user voice commands directly through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving user voice commands through a voice control device set outside the display device 200.

[0086] In some embodiments, the display device 200 also communicates with a server. The display device 200 may be allowed to communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server provides various content and interactive features to the display device 200. The server may be a cluster or multiple clusters, and may include one or more types of servers.

[0087] Figure 2A The diagram illustrates, for example, a hardware configuration block diagram of a display device 200, such as... Figure 2AAs shown, the display device 200 includes at least one of a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface (i.e., a user input interface) 280. The controller 250 includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 may be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and may also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The conference creation terminal 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the business server 500 via the communicator 220. The detector 230 is used to collect signals from the external environment or signals interacting with the outside world. The first controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the first controller 250 is located, such as an external set-top box. The user interface 280 can be used to receive control operations from control devices (such as a keyboard).

[0088] In some embodiments, the controller 250 controls the operation of the conference creation terminal and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the conference creation terminal 200. Users can input commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives these commands via the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives these commands by recognizing the sounds or gestures through sensors.

[0089] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be interface elements such as icons, windows, and controls displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0090] Figure 2B This is a schematic diagram of the software configuration in a display device 200 according to one or more embodiments of the present disclosure, such as... Figure 2B As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0091] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the applications in the application layer include, but are not limited to, the examples above.

[0092] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0093] In some embodiments, the kernel layer is a layer between hardware and software, and includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0094] To illustrate this solution in more detail, the following will describe the process of the display device of this disclosure executing the display control method of the embodiments of this disclosure in conjunction with specific exemplary embodiments. It is understood that the steps involved in the following exemplary embodiments may include more or fewer steps in actual implementation, and the order of these steps may also be different, as long as the display control method provided in the embodiments of this disclosure can be implemented. The embodiments of this disclosure do not limit the scope of implementation.

[0095] Reference Figure 3 As shown, Figure 3 This is a flowchart illustrating a display control method provided in an embodiment of the present disclosure, including steps S31 to S32.

[0096] S31. In response to a calibration instruction for the video frame timestamps of each video frame in the video stream, the video frame timestamps of each video frame are converted into the first system timestamps corresponding to each video frame.

[0097] The video frame timestamp is used to indicate the display order of each video frame, and the first system timestamp is used to indicate the timestamp of rendering each video frame. The rendering of each video frame is completed within the vertical refresh cycle corresponding to that video frame.

[0098] For example, the first system timestamp corresponding to each video frame can be obtained in the following way: for the first video frame in the video stream, the timestamp when the first video frame is decoded is used as the first system timestamp of the first video frame; for each video frame after the first video frame in the video stream, the difference between the video frame timestamp of the video frame and the video frame timestamp of the first video frame is obtained, and the difference is added to the first system timestamp of the first video frame to obtain the first system timestamp corresponding to the video frame.

[0099] For example, the video frame timestamp is represented by PTS, the first system timestamp is represented by STS, the video frame timestamp of the video frame is represented by PTS, and the video frame timestamp of the Nth video frame is PTS. N Let S1 be the timestamp when the decoding of the first video frame is completed. Then, the first systematic timestamp of the first video frame is S1. For any video frame N after the first video frame in the video stream, the first systematic timestamp of video frame N can be obtained as follows: STS 1N = S1 + (PTSn – PTS1); where S1 is the timestamp when the decoding of the first video frame is completed, and PTS... N PTS1 is the video frame timestamp of the Nth video frame, and PTS2 is the video frame timestamp of the first video frame.

[0100] In the video stream, if it is determined that the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, the process of determining whether the vertical refresh period corresponding to the Mth video frame will coincide with the vertical refresh period corresponding to any video frame after the Mth video frame is a prediction process. The decoding time delay or long decoding time of the Mth video frame causes the time when the Mth video frame arrives at the rendering module after decoding to be later than the display time (first system timestamp). For example, the Mth video frame should have arrived for rendering in the second vertical refresh period, but it is determined that the Mth video frame will arrive in the fourth vertical refresh period, that is, the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to the M+2th video frame. In order to ensure that the Mth video frame is not lost, it is necessary to delay the rendering time of the Mth video frame and the video frames after the Mth video frame, and execute the following step S32.

[0101] S32. Based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period, adjust the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame respectively to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame respectively.

[0102] Wherein, the second system timestamp corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame, and the second system timestamp corresponding to any video frame after the Mth video frame is later than the first system timestamp corresponding to that video frame.

[0103] For example, the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame can be obtained as follows: for each video frame in the Mth video frame and the video frames after the Mth video frame, the sum of the first system timestamp of the video frame and a preset number of vertical refresh periods is obtained as the second system timestamp corresponding to the video frame.

[0104] That is, the second system timestamp (STS) corresponding to the Mth video frame. 2m =S1+(PTS) M –PTS1)+V t Where S1 is the timestamp when the decoding of the first video frame is completed, and PTS M V is the video frame timestamp of the Mth video frame, PTS1 is the video frame timestamp of the first video frame, and V is the video frame timestamp of the Mth video frame. tThe preset number of vertical refresh cycles is t. The principle for determining the preset number of vertical refresh cycles is to not affect the user's viewing experience, that is, the user does not feel any lag. There is an upper limit threshold for the number of vertical refresh cycles to be delayed. For example, if two vertical refresh cycles of blank space will not affect the user's viewing experience, but three vertical refresh cycles of blank space will affect the user's viewing experience, then the preset number of vertical refresh cycles is one or two vertical refresh cycles.

[0105] S33. Render the Mth video frame and the video frames after the Mth video frame based on the second system timestamp, and control the display to show the Mth video frame and the video frames after the Mth video frame.

[0106] For example, refer to Figure 4 As shown, Figure 4 This is a rendering diagram of a video frame provided in another embodiment of the present disclosure. In the first V-sync cycle, two video frames, video frame F1 and video frame F2, are received. In the second V-sync cycle, video frame F3 is received. No video frames are received in the third and fourth V-sync cycles. In the fifth V-sync cycle, video frame F4 is received. The preset number of vertical refresh cycles is 1. After obtaining the second system time based on the above method, when rendering according to the second system timestamp, video frame F1 is rendered in the second V-sync cycle, video frame F2 is rendered in the third V-sync cycle, video frame F3 is rendered in the fourth V-sync cycle, and video frame F4 is rendered in the fifth V-sync cycle.

[0107] In this embodiment of the disclosure, since the second system time corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame, and the second system time corresponding to the video frames after the Mth video frame is later than the first system timestamp corresponding to that video frame, when rendering the Mth video frame and the video frames after the Mth video frame based on the second system timestamp, it is equivalent to delaying the rendering and display time of the Mth video frame and the video frames after the Mth video frame, thus avoiding the loss of data frames when the display device displays the data stream due to the delay in the decoding time of the Mth video frame.

[0108] Following the above embodiments, if, among the video frames after the Mth video frame, it is determined that the vertical refresh period corresponding to the Nth video frame coincides with the vertical refresh period corresponding to any video frame after the Nth video frame, then the following steps S51 to S52 are executed, and the video frames after the Mth video frame and before the Nth video frame are still rendered and displayed based on the second system time. The second system timestamps corresponding to the Nth video frame and the video frames after the Nth video frame are adjusted, and the following steps S51 to S52 are executed, referring to... Figure 5 As shown, Figure 5 This is a flowchart illustrating a display control method according to another embodiment of the present disclosure. Based on the embodiment shown in FIG1, it further includes the following steps S51 to S52. Wherein, any video frame after the Nth video frame can be the (N+1)th video frame, the (N+2)th video frame, the (N+3)th video frame, etc.

[0109] S51. Determine the third system timestamp of the Nth video frame, and adjust the second system timestamps of the video frames after the Nth video frame based on the second system timestamp corresponding to the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames after the Nth video frame respectively.

[0110] Wherein, the third system timestamp corresponding to the Nth video frame is later than the second system timestamp corresponding to the Nth video frame;

[0111] That is, if the Nth video frame arrives at the rendering module too late, still later than the second system timestamp, for example, the second system timestamp corresponds to the 5th vertical refresh cycle when rendering is performed based on the second system timestamp, and the arrival time of the Nth video frame at the rendering module corresponds to the 6th vertical refresh cycle, then the second system timestamp corresponding to the Nth video frame and the video frames after the Nth video frame are adjusted according to the steps of S51 above.

[0112] For example, reusing the Nth video frame as the starting video frame and as the reference frame for other video frames is equivalent to resetting the reference starting frame. The Nth video frame can then be used as the first video frame again, based on... Figure 3 The method in the illustrated embodiment determines the rendering time of the Nth video frame and subsequent video frames. The decoding completion time of the Nth video frame is obtained and used as its third system timestamp. For subsequent video frames, the sum of the third system timestamp of the Nth video frame and a preset number of vertical refresh periods is obtained and used as the corresponding third system timestamp for that video frame.

[0113] S52. Render the Nth video frame and the video frames after the Nth video frame based on the timestamp of the third system, and control the display to show the Nth video frame and the video frames after the Nth video frame.

[0114] For example, refer to Figure 6As shown, the second system timestamp of video frame F4 corresponds to the 5th vertical refresh cycle. However, the time when video frame F4 arrives at the rendering module is later than the second system timestamp, thus missing the 5th vertical refresh cycle. When it arrives at the rendering module, it corresponds to the 6th vertical refresh cycle. Therefore, the rendering time of video frame F4 needs to be adjusted to obtain the third system timestamp of video frame F4, so that rendering can be performed within the 6th vertical refresh cycle. Based on video frame F4 and a preset number of vertical refresh cycles, the third system timestamp corresponding to the video frames after video frame F4 is determined.

[0115] In this embodiment, when a video frame arrives at the rendering module too late, the rendering time is delayed based on the third system timestamp obtained from two adjustments, thus preventing frame drops caused by the video frame arriving at the rendering module too late. This embodiment uses two adjustments as an example, but it should be noted that if a video frame arriving at the rendering module after the Nth video frame still occurs at a time later than the rendering time of that video frame, the reference starting frame can be reset multiple times, i.e., steps S51 to S52 can be executed multiple times.

[0116] In some embodiments, after converting the video frame timestamps of each video frame into their respective first system timestamps in response to a calibration instruction for the video frame timestamps of each video frame in the video stream, if a user's pause playback operation is received, the rendering timestamps of each video frame after the current video frame can be adjusted through the following steps S71 to S73, referring to... Figure 7 As shown, Figure 7 This is a flowchart illustrating a display control method provided in another embodiment of the present disclosure.

[0117] S71. When the video stream is in a paused playback state, in response to the user's pause playback operation for ending the video stream, determine the fourth system timestamp of each video frame after the current video frame displayed on the display device.

[0118] For example, for each video frame after the current video frame, the fourth system timestamp corresponding to each frequency frame is obtained according to the fourth system time of the current video frame and the vertical refresh period.

[0119] The current video frame is the video frame corresponding to the paused playback state. When the pause playback operation ends, the display begins from the current video frame. Taking the current video frame as the starting video frame (the first video frame), the corresponding fourth system timestamp and whether to adjust the fourth system timestamp for other video frames after this starting video frame can be determined according to steps S31 to S52 in the above embodiments, which will not be elaborated here. The function of the fourth system timestamp corresponding to the current video frame is equivalent to the function of the first system timestamp corresponding to the first video frame in the video stream in the above embodiments.

[0120] S72. Render each video frame after the current video frame based on the timestamp of the fourth system, and control the display to show each video frame after the current video frame.

[0121] In some embodiments, after converting the video frame timestamps of each video frame into first system timestamps corresponding to each video frame in response to a calibration instruction for the video frame timestamps of each video frame in the video stream, the method further includes:

[0122] In response to a user's fast-forward playback operation on the video stream, the system determines the fifth system time corresponding to the current video frame displayed on the display device after the fast-forward playback operation ends; for each video frame after the current video frame, the system obtains the fifth system timestamp corresponding to each video frame based on the fifth system time of the current video frame and the vertical refresh period; the system renders the current video frame and each video frame after the current video frame based on the fifth system timestamp, and controls the display to display the current video frame and each video frame after the current video frame.

[0123] That is, it is also the process of resetting the reference starting frame. For example, the current video frame is the first video frame after the fast-forward playback operation ends. For the current video frame, the difference between the first system timestamp corresponding to the current video frame and the duration corresponding to the fast-forward playback state is used as the fifth system timestamp of the current video frame. For other video frames after this video frame, the corresponding fifth system timestamp and whether to adjust the fifth system timestamp can be determined according to the method of steps S31 to S52 in the above embodiment, which will not be repeated here. The function of the fifth system timestamp corresponding to the current video frame is equivalent to the function of the first system timestamp corresponding to the first video frame in the video stream in the above embodiment.

[0124] In some embodiments, in response to a user's rewind operation on the video stream, the sixth system time corresponding to the current video frame displayed on the display device is determined after the rewind operation ends; for each video frame after the current video frame, the sixth system timestamp corresponding to each video frame is obtained according to the sixth system time of the current video frame and the vertical refresh period; the current video frame and each video frame after the current video frame are rendered based on the sixth system timestamp, and the display is controlled to display the current video frame and each video frame after the current video frame.

[0125] That is, it is also the process of resetting the reference starting frame. For example, the current video frame is the first video frame after the rewind playback operation ends. For the current video frame, the sum of the first system timestamp corresponding to the current video frame and the duration corresponding to the rewind playback state is used as the sixth system timestamp of the current video frame. For other video frames after this video frame, the corresponding sixth system timestamp and whether to adjust the sixth system timestamp can be determined according to the method of steps S31 to S52 in the above embodiment, which will not be repeated here. The function of the sixth system timestamp corresponding to the current video frame is equivalent to the function of the first system timestamp corresponding to the first video frame in the video stream in the above embodiment.

[0126] In some embodiments, if the video frame arrives at the rendering module earlier than the rendering timestamp, resulting in an increasing number of unrendered video frames in the buffer queue, a time difference threshold is set, and the video frame is rendered by advancing the rendering timestamp by a preset number of vertical refresh cycles based on the time difference threshold.

[0127] For example, in combination Figure 8 As shown, Figure 8 This is a rendering diagram of a video frame provided in an embodiment of the present disclosure. In response to a calibration command for the video frame timestamp of each video frame in the video stream, after converting the video frame timestamp of each video frame into the first system timestamp corresponding to each video frame, if the first system timestamp (rendering timestamp) of video frame F6 and video frame F7 correspond to the 7th (V7) and 8th V-sync cycles (V8) respectively, and video frames F6 and F7 arrive in the 3rd (V3) and 4th V-sync cycles (V4), which is 4 V-sync cycles earlier than the rendering timestamp, in the case of a preset number of 1 V-sync cycle, the first system timestamps corresponding to video frames F6 and F7 are adjusted to the 7th system timestamp, corresponding to the 6th (V6) and 7th V-sync cycles (V7) respectively. That is, video frames F6 and F7 are adjusted to be rendered in the 6th (V6) and 7th V-sync cycles (V7) respectively, and video frame F6 is adjusted to be the reference starting frame.

[0128] This is to avoid the loss of video frame F5.

[0129] This disclosure provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the virtual meeting scene creation and output method described above, and achieves the same technical effect. To avoid repetition, further details are omitted here.

[0130] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0131] This disclosure provides a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the above-described virtual meeting scene creation method.

[0132] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific application considerations.

Claims

1. A display device, characterized in that, include: The controller is configured as follows: In response to a calibration command for the video frame timestamps of each video frame in the video stream, the video frame timestamps of each video frame are converted into first system timestamps corresponding to each video frame. The video frame timestamps are used to indicate the display order of each video frame, and the first system timestamps are used to indicate the timestamps for rendering each video frame. The rendering of each video frame is completed within the vertical refresh cycle corresponding to the video frame. In the video stream, if it is determined that the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, then the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period, respectively, to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame, respectively, wherein the second system timestamp corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame; Based on the second system timestamp, render the Mth video frame and the video frames following the Mth video frame, and control the display to show the Mth video frame and the video frames following the Mth video frame; The controller is specifically configured as follows: The adjustment of the first system timestamps corresponding to the Mth video frame and subsequent video frames based on the first system timestamp of the first video frame in the video stream and the vertical refresh period to obtain the second system timestamps corresponding to the Mth video frame and subsequent video frames includes: For the Mth video frame and each video frame in the subsequent video frames, the sum of the first system timestamp of the video frame and a preset number of vertical refresh cycles is obtained as the second system timestamp corresponding to the video frame.

2. The display device according to claim 1, characterized in that, The controller is also configured to: Before rendering the Mth video frame and subsequent video frames based on the second system timestamp, and before controlling the display to show the Mth video frame and subsequent video frames, the method further includes: In the video frames after the Mth video frame, if it is determined that the vertical refresh period corresponding to the Nth video frame coincides with the vertical refresh period corresponding to any video frame after the Nth video frame, then the third system timestamp corresponding to the Nth video frame is determined, and the second system timestamp corresponding to the video frames after the Nth video frame is adjusted based on the third system timestamp of the Nth video frame and the vertical refresh period to obtain the third system timestamps corresponding to the video frames after the Nth video frame respectively. The system renders the Nth video frame and subsequent video frames based on the third system timestamp, and controls the display to show the Nth video frame and subsequent video frames.

3. The display device according to claim 1, characterized in that, The controller is specifically configured as follows: The step of converting the video frame timestamps of each video frame into a first system timestamp corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream includes: For the first video frame in the video stream, the timestamp when the decoding of the first video frame is completed is used as the first system timestamp of the first video frame; For each video frame after the first video frame in the video stream, the difference between the video frame timestamp of the video frame and the video frame timestamp of the first video frame is obtained, and the difference is added to the first system timestamp of the first video frame to obtain the first system timestamp corresponding to the video frame.

4. The display device according to claim 2, characterized in that, The controller is specifically configured as follows: The process of determining the third system timestamp corresponding to the Nth video frame, and adjusting the second system timestamps corresponding to the video frames following the Nth video frame based on the third system timestamp of the Nth video frame and the vertical refresh period, to obtain the third system timestamps corresponding to the video frames following the Nth video frame respectively, includes: For the Nth video frame, the decoding completion time of the Nth video frame is used as the third system timestamp of the Nth video frame; For video frames after the Nth video frame, the sum of the third system timestamp of the Nth video frame and a preset number of vertical refresh cycles is obtained as the third system timestamp corresponding to that video frame.

5. The display device according to claim 1, characterized in that, The controller is also configured to: After converting the video frame timestamps of each video frame into the first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes: When the video stream is in a paused playback state, in response to the user's operation to end the pause playback of the video stream, the fourth system time of each video frame after the current video frame displayed on the display device is determined. The system renders each video frame after the current video frame based on the fourth system timestamp, and controls the display to show each video frame after the current video frame.

6. The display device according to claim 5, characterized in that, The controller is specifically configured as follows: The step of determining the fourth system time of each video frame after the current video frame displayed on the display device in response to a user's pause playback operation for ending the video stream includes: For each video frame following the current video frame, the fourth system timestamp corresponding to each video frame is obtained according to the fourth system time of the current video frame and the vertical refresh period.

7. The display device according to claim 1, characterized in that, The controller is also configured to: After converting the video frame timestamps of each video frame into the first system timestamps corresponding to each video frame in response to a calibration command for the video frame timestamps of each video frame in the video stream, the method further includes: In response to a user's fast-forward playback operation on the video stream, the fifth system time corresponding to the current video frame displayed on the display device is determined after the fast-forward playback operation ends; For each video frame following the current video frame, the fifth system timestamp corresponding to each video frame is obtained according to the fifth system time of the current video frame and the vertical refresh period; The current video frame and all subsequent video frames are rendered based on the fifth system timestamp, and the display is controlled to show the current video frame and all subsequent video frames.

8. A display control method, characterized in that, include: In response to a calibration command for the video frame timestamps of each video frame in the video stream, the video frame timestamps of each video frame are converted into first system timestamps corresponding to each video frame. The video frame timestamps are used to indicate the display order of each video frame, and the first system timestamps are used to indicate the timestamps for rendering each video frame. The rendering of each video frame is completed within the vertical refresh cycle corresponding to the video frame. In the video stream, if it is determined that the vertical refresh period corresponding to the Mth video frame coincides with the vertical refresh period corresponding to any video frame after the Mth video frame, then the first system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame are adjusted based on the first system timestamp corresponding to the first video frame in the video stream and the vertical refresh period, respectively, to obtain the second system timestamps corresponding to the Mth video frame and the video frames after the Mth video frame; the second system timestamp corresponding to the Mth video frame is later than the first system timestamp corresponding to the Mth video frame. Based on the second system timestamp, render the Mth video frame and the video frames following the Mth video frame, and control the display to show the Mth video frame and the video frames following the Mth video frame; The method further includes: The adjustment of the first system timestamps corresponding to the Mth video frame and subsequent video frames based on the first system timestamp of the first video frame in the video stream and the vertical refresh period to obtain the second system timestamps corresponding to the Mth video frame and subsequent video frames includes: For the Mth video frame and each video frame in the subsequent video frames, the sum of the first system timestamp of the video frame and a preset number of vertical refresh cycles is obtained as the second system timestamp corresponding to the video frame.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the display control method as described in claim 8.

Images