Video playing method, electronic equipment, chip system and storage medium

Abstract

The invention provides a video playing method, electronic equipment, a chip system and a storage medium, and relates to the technical field of video playing, in the video playing process, the calculation rendering time of a video frame with a system clock as the criterion can be obtained according to the timestamp of the video frame; calculating the incomplete alignment between the rendering time and the vsync signal; after receiving the vsync signal, the rendering thread renders the video frame with the calculated rendering time before the signal and the latest in the queue to be rendered, and if the calculated rendering time of the video frame is close to the edge of the signal, the timestamp jitter of the video frame may cause the video frame to jump to other signal periods, resulting in the problems of frame loss and repeated rendering; in order to solve the problem, the rendering time of the video frame is adjusted to a position close to the middle of the two signals as much as possible, and the video frame does not jump to other signal periods even if jittering exists, so that the situations of frame loss and repeated rendering are reduced.

Description

Technical Field

[0001] This application relates to the field of video playback technology, and in particular to a video playback method, electronic device, chip system, and storage medium. Background Technology

[0002] Electronic devices are becoming increasingly feature-rich. For example, devices equipped with cameras can record videos, play them on their own devices, or send them to other devices or the network. When recording video, the frame rate can be set. For instance, a frame rate of 60fps can be set, meaning the device captures 60 video frames per second, each with a timestamp indicating its playback time.

[0003] However, when electronic devices capture videos, there may be jitter due to uneven time intervals between the capture of each video frame, resulting in uneven timestamps for each video frame. When electronic devices play videos, the screen itself has a refresh rate. If the timestamps of the video frames are uneven (for example, one video frame is earlier or another is later), problems such as frame dropping and repeated rendering may occur, causing stuttering during video playback. Summary of the Invention

[0004] This application provides a video playback method, electronic device, chip system, and storage medium that can reduce stuttering during video playback.

[0005] To achieve the above objectives, the first aspect of this application provides a video playback method, employing the following technical solution:

[0006] During the playback of the first video, the first moment corresponding to the reference vsync signal is obtained, and the reference vsync signal is the first vsync signal;

[0007] The first video frame whose rendering time is calculated after the first moment is determined, and the first video frame is the first video frame;

[0008] Based on the first moment, the first adjustment rendering time of the first video frame is obtained. The first adjustment rendering time is the time offset by 1 / 2T based on the first moment, where T is the period of the real-time detected vsync signal.

[0009] The first video frame is rendered based on the first adjusted rendering time to obtain a rendered image, which is used to display on the screen.

[0010] During video playback, the calculated rendering time of a video frame, based on the system clock, can be obtained from the video frame's timestamp. However, the calculated rendering time and the vsync signal are not perfectly aligned. After receiving the vsync signal, the rendering thread renders the latest video frame in the rendering queue whose calculated rendering time precedes that vsync signal. If the calculated rendering time of a video frame is close to the edge of the vsync signal, timestamp jitter may cause the video frame to jump to other vsync signal cycles, resulting in dropped frames. To solve this problem, the rendering time of the video frame is adjusted to be as close as possible to the middle of the two signals. Even with jitter, the video frame will not jump to other signal cycles, thus reducing dropped frames and duplicate rendering.

[0011] When the rendering time of a video frame is adjusted to be as close as possible to the middle of the two signals, the adjusted rendering time of the video frame can be the time offset from the base vsync signal by 1 / 2 of the signal period.

[0012] As another implementation of the first aspect, obtaining the first adjustment rendering time of the first video frame based on the first moment includes:

[0013] Based on the first moment, the first anchor time is determined, which is the moment offset by multiple vsync signal cycles based on the first moment.

[0014] Divide the difference between the calculation rendering time of the first video frame and the time of the first anchor point by T to obtain the remainder;

[0015] The theoretical time of the first vsync signal is obtained by subtracting the remainder from the calculated rendering time of the first video frame.

[0016] Add 1 / 2T to the theoretical time of the first vsync signal to obtain the first adjustment rendering time of the first video frame.

[0017] In this application, multiple vsync signals are shifted forward as the first anchor time, and then the theoretical time of the first vsync signal is determined based on the first anchor time, thereby adjusting the rendering time of the first video frame to 1 / 2T after the theoretical time of the first vsync signal.

[0018] As another implementation of the first aspect, determining the first anchor point time based on the first moment includes:

[0019] The difference between the first moment and the calculation rendering time of the first video frame is rounded down to obtain the rounded result;

[0020] Add the first value to the rounded result and multiply it by the vsync signal period to obtain the offset period;

[0021] Subtract the offset period from the first moment to obtain the first anchor point time.

[0022] In this application, multiple vsync signal cycles can be shifted forward as anchor time.

[0023] As another implementation of the first aspect, the method further includes:

[0024] Determine a second video frame whose rendering time is after the first moment, wherein the second video frame is a video frame other than the first video frame;

[0025] Based on the first moment, a second anchor time is determined, wherein the second anchor time is the moment offset by multiple vsync signal cycles based on the first moment.

[0026] Divide the difference between the calculation rendering time of the second video frame and the time of the second anchor point by T and obtain the remainder;

[0027] The theoretical time of the second vsync signal is obtained by subtracting the remainder from the calculation and rendering time of the second video frame. The second vsync signal is the latest vsync signal before the calculation and rendering time of the second video frame.

[0028] Add 1 / 2T to the theoretical time of the second vsync signal to obtain the second adjustment rendering time of the second video frame.

[0029] In this application, the difference between the calculation rendering time of the second video frame and the time of the second anchor point is equivalent to including the real vsync signal period and the virtual vsync signal period detected in real time, which can balance the jitter error.

[0030] As another implementation of the first aspect, determining the first anchor point time based on the first moment includes:

[0031] The difference between the calculation and rendering time of the first moment and the second video frame is rounded down to obtain the rounded result;

[0032] Add the first value to the rounded result and multiply it by the vsync signal period to obtain the offset period;

[0033] Subtract the offset period from the first moment to obtain the second anchor point time.

[0034] As another implementation of the first aspect, before rounding down the difference between the calculation and rendering times of the first moment and the second video frame to obtain the rounded result, the method further includes:

[0035] Calculate the difference between the computation and rendering time of the first moment and the second video frame;

[0036] It is determined that the difference between the calculation and rendering time of the first moment and the second video frame is less than nT.

[0037] As another implementation of the first aspect, after calculating the difference between the computation and rendering times of the first moment and the second video frame, it further includes:

[0038] The difference between the calculation and rendering time of the first moment and the second video frame is determined to be no less than nT;

[0039] The first moment is taken as the second anchor point time.

[0040] As another implementation of the first aspect, the reference vsync signal is the latest vsync signal determined according to a first time period; the vsync signal period is the signal period detected in real time according to a second time period.

[0041] In this application, after calculating the adjusted rendering time of multiple video frames and stabilizing the adjusted rendering time, the anchor time is no longer updated in the above manner, and the reference signal can be used as the anchor time.

[0042] As another implementation of the first aspect, the first time period and the second time period are the same.

[0043] As another implementation of the first aspect, the first calculated rendering time of the first video frame is the rendering time based on the system time, calculated according to the timestamp of the first video frame; the timestamp of the first video frame is the playback time of the first video frame in the first video.

[0044] As another implementation of the first aspect, after obtaining the first adjusted rendering time of the first video frame based on the first moment, it further includes:

[0045] Determine whether the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is greater than or equal to the second duration;

[0046] If the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is less than the second duration, then the first adjustment rendering time of the first video frame is extended by T.

[0047] If the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is not less than the second duration, then the first adjustment rendering time of the first video frame will not be delayed.

[0048] In this application, if the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is less than the second duration, it indicates that the current first video frame has jumped to the previous vsync signal cycle, and the adjustment rendering time of the first video frame needs to be delayed by T.

[0049] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0050] Thirdly, a chip system is provided, including a processor coupled to a memory, wherein the processor executes a computer program stored in the memory to cause an electronic device to implement the method of any one of the first aspects of this application.

[0051] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0052] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0053] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0054] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0055] Figure 2 A schematic diagram illustrating the relationship between a video frame's timestamp, calculated rendering time, vsync signal, and actual rendering time, provided for an embodiment of this application.

[0056] Figure 3 A schematic diagram illustrating frame drops and repeated rendering when the timestamp of a video frame is delayed, as provided in an embodiment of this application;

[0057] Figure 4 A schematic diagram illustrating frame drops and repeated rendering when the timestamp of a video frame is advanced, as provided in an embodiment of this application;

[0058] Figure 5The permissible jitter range of the video frame and the permissible jitter range of the vsync signal are provided in the embodiments of this application when the adjustment rendering time is set between two vsync signals;

[0059] Figure 6 The embodiments provided in this application are related to Figure 3 The corresponding rendering diagram shows the rendering situation when the rendering time is adjusted and set between two vsync signals;

[0060] Figure 7 The embodiments provided in this application are related to Figure 4 The corresponding rendering diagram shows the rendering situation when the rendering time is adjusted and set between two vsync signals;

[0061] Figure 8 A schematic diagram illustrating the process of calculating the adjustment rendering time of video frames provided in an embodiment of this application;

[0062] Figure 9 A schematic diagram illustrating the process of calculating the adjustment rendering time of video frames provided in an embodiment of this application;

[0063] Figure 10 A schematic diagram illustrating the process of calculating the adjustment rendering time of video frames provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the technical architecture of the video playback method provided in the embodiments of this application. Detailed Implementation

[0065] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0066] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0067] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0068] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0069] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0070] This application provides a video playback method that can be applied to electronic devices. These electronic devices can be tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not limit the specific type of electronic device.

[0071] Figure 1 A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0072] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0074] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0075] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback). Touch sensor 180K, also called a "touch panel," can be disposed on display screen 194. Touch sensor 180K and display screen 194 together form a touch screen, also called a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor can transmit the detected touch operation to application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, in a different location than display screen 194.

[0076] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0077] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0078] This application does not specifically limit the structure of the execution entity of a video playback method. As long as communication can be performed according to the video playback method provided by this application by running code containing the code of this application's video playback method, it is acceptable. For example, the execution entity of the video playback method provided by this application can be a functional module in an electronic device capable of calling and executing programs, or a communication device applied in an electronic device, such as a chip.

[0079] Electronic devices are becoming increasingly feature-rich. For example, electronic devices equipped with cameras can record videos, play the recorded videos on their own devices, and send the videos to other electronic devices or the network for playback.

[0080] When an electronic device records video, the frame rate can be set. For example, the frame rate can be set to 60fps, meaning that 60 video frames are captured per second. Each video frame carries a timestamp (Presentation Time Stamp, pts). Theoretically, the timestamps of the video frames in the recorded video are as follows: Video frame 0: timestamp 0ms, Video frame 1: timestamp 16.66ms, Video frame 2: timestamp 33.33ms, Video frame 3: timestamp 50ms, Video frame 4: timestamp 66.66ms, Video frame 5: timestamp 83.33ms, Video frame 6: timestamp 100ms, and so on.

[0081] When an electronic device plays this video, it needs to display each video frame sequentially according to its timestamp. For example, decoding and rendering begins at second 0 of playback, displaying video frame 0 with timestamp 0ms; video frame 1 with timestamp 16.66ms is displayed at 16.66ms; video frame 2 with timestamp 33.33ms is displayed at 33.33ms; video frame 3 with timestamp 50ms is displayed at 50ms; video frame 4 with timestamp 66.66ms is displayed at 66.66ms; video frame 5 with timestamp 83.33ms is displayed at 83.33ms; video frame 6 with timestamp 100ms is displayed at 100ms, and so on. In essence, the timestamp of a video frame indicates the specific moment a video frame is displayed or played during video playback.

[0082] When an electronic device displays content on its screen, it refreshes the image at a fixed refresh rate, for example, 60Hz. The timing of this display is controlled by the vsync signal.

[0083] Reference Figure 2This is a schematic diagram illustrating the relationship between the vsync signal and the PTS of a video frame during video playback, provided in an embodiment of this application.

[0084] In this embodiment, the frequency of the vsync signal is 60Hz and the frame rate of the video frame is also 60fps. The time interval of the vsync signal is approximately 16.66ms; the time interval of the timestamps of the video frames is also approximately 16.66ms.

[0085] The vsync signal is based on the electronic device's system clock. Therefore, the rendering time of the video frame, based on the system clock, also needs to be calculated using the video frame's timestamp. Typically, at a playback rate of 1, the rendering time of a video frame is equal to the sum of the video frame's timestamp and the time when playback begins. Of course, in practical applications, there are more precise methods for calculating the rendering time of video frames. This application does not limit the process of calculating the rendering time of a video frame based on its timestamp using the system clock.

[0086] In practical applications, the computational rendering time of a video frame and the vsync signal (used to trigger the rendering thread to execute rendering tasks) may not be perfectly aligned. Therefore, in actual applications, the computational rendering time and the actual rendering time of a video frame may not be perfectly aligned. Typically, after receiving the vsync signal, the rendering thread can select the video frame whose computational rendering time is before this vsync signal and has the latest computational rendering time for rendering, and it will then be sent to the screen for display.

[0087] Reference Figure 2 Calculations show that the rendering time for video frame 0 is 12ms based on the electronic device's system clock, the rendering time for video frame 1 is 28.66ms, the rendering time for video frame 2 is 45.33ms, the rendering time for video frame 3 is 62ms, and so on. It should be noted that the times in this embodiment are only used to briefly describe the relationship between the vsync signal and the timestamps (or rendering times) of the video frames, and are not intended to impose any limitations.

[0088] After receiving the vsync1 signal at 20.66ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 0) from the rendering queue and calculates the latest video frame (i.e., video frame 0) whose rendering time is before the currently received vsync signal.

[0089] After receiving the vsync2 signal at 37.33ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 1) from the rendering queue and calculates the latest video frame (i.e., video frame 1) whose rendering time is before the currently received vsync signal.

[0090] After receiving the vsync3 signal at 54ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 2) from the queue to be rendered and calculates the latest video frame (i.e., video frame 2) that was received before the currently received vsync signal.

[0091] ...

[0092] In this way, after receiving the vsync signal, the rendering thread retrieves the video frames that were rendered before the current vsync signal from the queue to be rendered, calculates the video frame with the latest rendering time, and renders it; subsequently, the rendered images can be sent to the screen for display in sequence.

[0093] Reference Figure 2 As shown, theoretically, when the video frame rate and screen refresh rate are the same, the calculation and rendering time of the video frames in the video are located in different vsync signal cycles. The rendering thread can select a video frame for rendering based on the vsync signal, so that each video frame in the video is displayed on the screen in sequence.

[0094] However, in practical applications, video recording via electronic devices may experience jitter. For example, the capture time of a video frame might be too early or too late, resulting in an excessively early or late timestamp for that frame. Consequently, when playing back the video containing that frame, its calculation and rendering time might be too early or too late. This could cause the calculation and rendering time of a video frame to fall within the previous or next vsync signal cycle. When the rendering thread selects a video frame from the rendering queue, it might choose the frame with the latest calculation and rendering time before the currently received vsync signal, leading to dropped frames and repeated rendering during video playback. For details, please refer to... Figure 3 and Figure 4 As shown.

[0095] Reference Figure 3 As shown, according to the frame rate when the electronic device shoots the video, the timestamp of video frame 2 should theoretically be 33.3ms. However, due to the shaking during shooting, the acquisition time of video frame 2 is delayed, so the timestamp of video frame 2 is delayed by 38ms.

[0096] The calculated rendering times for each video frame during this playback, based on the system clock, are as follows: Frame 0 is at 18ms, Frame 1 is at 34.66ms, Frame 2 is at 56ms, Frame 3 is at 68ms, Frame 4 is at 84.66ms, and Frame 5 is at 101.33ms.

[0097] If the overall video frame's computation and rendering time is located on the left edge of the vsync signal (located to the left of the vsync signal and very close to it), and the timestamp of a certain video frame is delayed, the computation and rendering time of that video frame is likely to exceed the vsync signal and fall into the next vsync signal cycle.

[0098] For example, Figure 3 As shown, the calculation and rendering time of the video frames are uniformly located on the left edge of the vsync signal. However, the calculation and rendering time of video frame 2, which should have been 33.33ms + 18ms = 51.33ms when there was no jitter during acquisition, should be located to the left of vsync3 (54ms). However, when there was jitter during acquisition, the calculation and rendering time of video frame 2 was 38ms + 18ms = 56ms, which fell to the right of vsync3.

[0099] The following details the frame loss situation in this case.

[0100] After receiving the vsync1 signal at 20.66ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 0) from the rendering queue and calculates the latest video frame (i.e., video frame 0) whose rendering time is before the currently received vsync signal.

[0101] After receiving the vsync2 signal at 37.33ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 1) from the rendering queue and calculates the latest video frame (i.e., video frame 1) whose rendering time is before the currently received vsync signal.

[0102] After receiving the vsync3 signal at 54ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and calculates the latest video frame (still video frame 1) that was rendered before the currently received vsync signal. Since video frame 1 has already been consumed, it is not rendered.

[0103] After receiving the vsync4 signal at 70.66ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 3) from the rendering queue and calculates the latest video frame (i.e., video frame 3) whose rendering time is before the currently received vsync signal.

[0104] It's understandable that the failure to render video frame 2 resulted in frame dropping, causing the video to stutter from the user's perspective.

[0105] Of course, in practical applications, the timestamps of video frames may be earlier than expected.

[0106] Reference Figure 4As shown, according to the frame rate when the electronic device shoots the video, the timestamp of video frame 2 should theoretically be 33.3ms. However, due to the shaking during shooting, the acquisition time of video frame 2 is advanced, so the timestamp of video frame 2 is advanced by 28ms.

[0107] The calculated rendering times for each video frame during this playback, based on the system clock, are as follows: Frame 0: 23ms; Frame 1: 39.66ms; Frame 2: 51ms; Frame 3: 73ms; Frame 4: 89.66ms; Frame 5: 106.33ms.

[0108] If the overall video frame's calculation and rendering time is located on the right edge of the vsync signal (located to the right of the vsync signal and very close to it), and the timestamp of a certain video frame is advanced, the calculation and rendering time of that video frame is likely to exceed the vsync signal and fall within the previous vsync signal.

[0109] For example, Figure 4 As shown, the calculation and rendering time of the video frames are uniformly located on the right edge of the vsync signal. However, the calculation and rendering time of video frame 2, which should have been 33.33ms + 23ms = 56.33ms when there was no jitter during acquisition, should be located to the right of vsync3 (54ms). However, when there was jitter during acquisition, the calculation and rendering time of video frame 2 was 28ms + 23ms = 51ms, which fell to the left of vsync3.

[0110] The following details the frame loss situation in this case.

[0111] After receiving the vsync2 signal at 37.33ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 0) from the queue to be rendered and calculates the latest video frame (i.e., video frame 0) that was received before the currently received vsync signal.

[0112] After receiving the vsync3 signal at 54ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and calculates the latest video frame (i.e., video frame 2) that was received before the currently received vsync signal, and renders it, resulting in the loss of video frame 1.

[0113] After receiving the vsync4 signal at 70.66ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and calculates the latest video frame whose rendering time is before the currently received vsync signal. Since video frame 2 has already been consumed, it is not rendered.

[0114] against Figure 3 and Figure 4 As shown, the time interval jitter during video capture is usually small. The specific jitter values ​​listed in the above examples are only used to illustrate frame dropping and duplicate rendering issues and do not impose any limitations on this application. Of course, the timestamps of the video frames in the above examples are based on 0ms, but in practical applications, they can be any video frame during video playback; the system clock time in the above examples starts from 4ms, but in practical applications, it can also be any period of the system clock.

[0115] In order to solve Figure 3 and Figure 4 The video stuttering phenomenon shown can be addressed by adjusting the calculated rendering time obtained from the timestamps of the video frames, so that the adjusted rendering time of the video frames is generally located in the middle area of ​​the two vsync signals.

[0116] Reference Figure 5 As shown, the overall adjustment and rendering time of the video frame is located in the middle area of ​​the two vsync signals. Taking video frame 2 as an example, the allowable jitter range of video frame 2 is relatively large, which is nearly one vsync signal cycle. Even if video frame 2 jitters, it is relatively difficult for it to jitter outside the vsync signal cycle.

[0117] Reference Figure 6 As shown, Figure 3 The example shown illustrates the video playback process after adjusting the rendering time of the video frame, based on the system clock, to the exact midpoint between the two vsync signals.

[0118] In this example, the calculated rendering time of each video frame is uniformly reduced by 5.66ms to obtain the adjusted rendering time. During this playback, the adjusted rendering times of each video frame based on the system clock are as follows: the adjusted rendering time of video frame 0 is 12.33ms, the adjusted rendering time of video frame 1 is 29ms, the adjusted rendering time of video frame 2 is 50.33ms, the adjusted rendering time of video frame 3 is 62.33ms, the adjusted rendering time of video frame 4 is 79ms, and the adjusted rendering time of video frame 5 is 95.66ms.

[0119] After receiving the vsync1 signal at 20.66ms of the system clock, the rendering thread retrieves the video frame (i.e., video frame 0) from the rendering queue and calculates the latest video frame (i.e., video frame 0) whose rendering time is before the currently received vsync signal.

[0120] After receiving the vsync2 signal at 37.33ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 1) that was received before the currently received vsync signal.

[0121] After receiving the vsync3 signal at 54ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 2) that was received before the currently received vsync signal.

[0122] After receiving the vsync4 signal at 70.66ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 3) that was received before the currently received vsync signal.

[0123] It is understandable that the jitter in video frame 2 did not cross the vsync signal into the cycle of other vsync signals, thus avoiding the phenomenon of video stuttering.

[0124] Reference Figure 7 As shown, Figure 4 The example shown illustrates the video playback process after adjusting the rendering time of the video frame, based on the system clock, to the exact midpoint between the two vsync signals.

[0125] In this example, the calculated rendering time of each video frame is uniformly increased by 6ms to obtain the adjusted rendering time. During this playback, the adjusted rendering times of each video frame based on the system clock are as follows: the adjusted rendering time of video frame 0 is 29ms, the adjusted rendering time of video frame 1 is 45.66ms, the adjusted rendering time of video frame 2 is 56ms, the adjusted rendering time of video frame 3 is 79ms, and the adjusted rendering time of video frame 4 is 95.66ms.

[0126] After receiving the vsync2 signal at 37.33ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 0) that was received before the currently received vsync signal.

[0127] After receiving the vsync3 signal at 54ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 1) that was received before the currently received vsync signal.

[0128] After receiving the vsync4 signal at 70.66ms of the system clock, the rendering thread retrieves the video frame to be rendered from the queue and adjusts the rendering time to the latest video frame (i.e., video frame 2) that was received before the currently received vsync signal.

[0129] It is understandable that the jitter in video frame 2 did not cross the vsync signal into the cycle of other vsync signals, thus avoiding the phenomenon of video stuttering.

[0130] In the above embodiments, the overall video frame rendering time is adjusted (added to or subtracted from) by the same amount of time. This method requires determining the overall adjustment time.

[0131] For example, calculate the average computation rendering time of multiple consecutive (e.g., 5) video frames in history; find the period of the vsync signal in which the computation rendering time of the first video frame in the multiple (e.g., 5) video frames is located; calculate the average of the median values ​​of multiple consecutive (e.g., 5) vsync signal periods starting from this vsync signal period; calculate the difference between the two average values ​​to obtain the time to be adjusted each time the computation rendering time is adjusted subsequently.

[0132] Of course, in practical applications, in order to avoid severe jitter in multiple consecutive video frames, the above adjustment process can also be performed in a loop.

[0133] For example, after receiving a vsync signal, the system can: average the calculated rendering times of multiple consecutive (e.g., 5) video frames whose rendering times precede the current vsync signal; find the period of the vsync signal containing the rendering time of the first video frame in a sequence of m (e.g., 5) consecutive video frames; calculate the average of the median values ​​of multiple consecutive (e.g., 5) vsync signal periods starting from this period; calculate the difference between the two averages; and add this difference to the rendering time of the first video frame whose rendering time precedes the currently received vsync signal. The multiple consecutive (e.g., 5) video frames whose rendering times precede the current vsync signal are: the i-th video frame from the end and the (i+5-1)-th video frame from the end before the currently received vsync signal. i is an integer greater than or equal to 1.

[0134] Of course, in practical applications, there may be other ways to calculate the computational rendering time of video frames and adjust the rendering time accordingly.

[0135] As another example, after receiving the vsync signal, the fifth vsync signal before the current vsync signal can be used as the reference signal. Then, five cycles can be added to the fifth vsync signal to obtain the theoretical time of the current vsync signal. The first video frame whose rendering time is calculated after the current vsync signal can be adjusted to the theoretical time of the current vsync signal plus 1 / 2T.

[0136] Of course, you can also take the last 5 Vsync signals before the current Vsync signal as reference signals, and then add the corresponding number of periods to each reference signal (for example, the last 5 Vsync signal + 5T, the last 5-1 Vsync signal + (5-1)T, ...), to obtain the theoretical time of the current Vsync signal corresponding to the multiple reference signals. Calculate the average of the theoretical time corresponding to the multiple reference signals, and then adjust the first video frame after the current Vsync signal to the time when the rendering time is calculated to the average value plus 1 / 2T.

[0137] As yet another example, see Figure 8 This diagram illustrates the process of calculating the rendering time of the rendering thread in an embodiment of this application. In a specific implementation, the frame scheduling algorithm module calculates the adjusted rendering time for each video frame.

[0138] Reference Figure 8 In (a), the frame scheduling algorithm module can obtain the latest vsync signal at regular intervals (e.g., 1 second), and then use the vsync signal as a reference signal to adjust the adjustment rendering time of the video frame.

[0139] Taking the vsync6 signal as the reference signal as an example, first obtain the first video frame whose rendering time is calculated after the corresponding time of the vsync signal; then subtract the calculated rendering time from the reference time and round the result. This number is usually 0.

[0140] Reference Figure 8 In step (b), the rounded result is added to 5 and then multiplied by the vsync signal period (this period can be detected in real time, for example, once per second). This is equivalent to subtracting 5 T from the reference time corresponding to the reference signal to obtain the anchor time. It should be noted that this anchor time may not correspond to the actual occurrence time of the vsync1 signal.

[0141] Reference Figure 8 In (c), the remainder of the result of dividing the difference between the calculated rendering time and the anchor time by the vsync signal period is taken.

[0142] Reference Figure 8 In (d), the theoretical time for the vsync6 signal is obtained by subtracting the remainder from the calculated rendering time.

[0143] Reference Figure 8 In (e), the theoretical time of the vsync6 signal is added to 1 / 2T to obtain the adjusted rendering time of the first video frame after vsync6.

[0144] Of course, in practical applications, the actual time corresponding to the vsync6 signal 6 can also be taken as the theoretical time of the vsync6 signal.

[0145] Reference Figure 9 This refers to the process of adjusting the rendering time of other video frames after the first video frame following the reference signal provided in the embodiments of this application.

[0146] Reference Figure 9 In (a) of this example, the vsync6 signal remains the reference signal. First, obtain the second video frame whose rendering time is calculated after the corresponding time of the vsync signal; then subtract the calculated rendering time from the reference time and round the result. This number is typically -1.

[0147] Reference Figure 9 In step (b), the rounded result is added to 5 and then multiplied by the vsync signal period. The anchor time is obtained by subtracting 4 T from the reference time corresponding to the reference signal. It should be noted that this anchor time may not correspond to the actual occurrence time of the vsync2 signal.

[0148] Reference Figure 9 In (c), the remainder of the result of dividing the difference between the rendering time and the anchor time (the difference between vsync signal 7 and vsync 6) by the period of the vsync signal is taken.

[0149] Reference Figure 9 In (d), the theoretical time of the vsync7 signal is obtained by subtracting the remainder from the calculated rendering time. It should be noted that the theoretical time and the actual time of the vsync7 signal may differ because the current calculation process considers one real vsync signal cycle (the difference between vsync signal 7 and vsync6) and four virtual vsync signal cycles detected in real time according to the cycle, so as to balance the error.

[0150] Reference Figure 9 In (e), the theoretical time of the vsync7 signal is added to 1 / 2T to obtain the adjusted rendering time of the first video frame after vsync7.

[0151] Of course, the rendering time can also be calculated for other non-first frames after the reference signal. Figure 9 The method shown illustrates how to calculate the adjusted rendering time. Essentially, when calculating the adjusted rendering time for each video frame, the anchor point is shifted back by one vsync signal cycle. Furthermore, when calculating the remainder, one more actual vsync signal cycle is considered compared to the previous video frame. For example, when calculating the adjusted rendering time for the third video frame, the anchor point time is near the vsync 3 signal. When calculating the remainder, the two actual vsync signal cycles corresponding to vsync8 to vsync6 are used in the jitter balancing calculation.

[0152] In practical applications, if the difference between the rendering time and the reference time is greater than 4T, the variable anchor time can be replaced by the reference signal. The process of calculating the remainder is similar to that of calculating the remainder in non-first frames.

[0153] To better understand the execution logic above, refer to... Figure 10 The flowchart is shown below.

[0154] S101, after the frame scheduling calculation module receives the latest vsync signal at a period (e.g., 1 second), it obtains the calculation and rendering time of the first video frame after the vsync signal.

[0155] S102, records the time and vsync cycle of the reference vsync signal (real-time detection).

[0156] S103, determine whether the current vsync signal period is within the preset range.

[0157] The implementation of this application allows for a small range of jitter in the vsync signal. If the vsync signal jitter is too large, frame drops and repeated rendering will be unavoidable. Therefore, the video frame calculation and rendering time adjustment process provided in this application embodiment is implemented by setting the vsync signal period within a preset range. This preset range can be related to the method of calculating the current vsync signal period, or it can be calculated based on the maximum jitter range of the vsync signal. For example, if the maximum range of a vsync signal period is set to 1 / 4T left and right, the vsync signal at 16.6ms can jitter to 12.5ms or 20.83ms, then the allowable difference between two consecutive vsync signals is 12.5ms (corresponding to 80Hz) or 20.83ms (48Hz). This example is only used as an example and does not impose any limitations on this application.

[0158] If the current vsync signal period is outside the preset range, the video frame calculation and rendering time will not be adjusted.

[0159] S104, if the current vsync signal period is within a preset range, determine whether the difference between the baseline vsync signal and the first subsequent calculation rendering time is less than 4 vsync signal periods.

[0160] S105, the difference between the baseline vsync signal and the first subsequent computed render time is less than 4 vsync signal cycles, according to Figure 8 The anchor point time is determined based on the calculated rendering time and the reference signal. Of course, for non-first video frames, the anchor point time is determined using... Figure 9The anchor point time is determined by this method.

[0161] S106, the difference between the reference vsync signal and the first calculated rendering time thereafter is not less than 4 vsync signal cycles, and the reference vsync signal is used to determine the anchor point time.

[0162] S107, based on the anchor point time, obtain the adjusted rendering time.

[0163] S108, determine whether the time interval between the adjustment rendering time of the previous video frame and the latest calculated adjustment rendering time is less than 2ms.

[0164] S109, if it is less than 2ms, the calculated adjustment rendering time will be delayed by one vsync signal cycle.

[0165] If it is less than 2ms, it means that one of the two video frames has too much jitter. The adjustment rendering time of the latest determined video frame can be delayed by one vsync signal cycle as the adjustment rendering time.

[0166] Of course, if the time is not less than 2ms, then no further adjustment to the rendering time is needed.

[0167] Normally, after adjusting the rendering time sequentially, the adjusted rendering time of the previous video frame is in the middle of two vsync signal cycles. If the adjusted rendering time of the latest video frame is less than 2ms, it means that the latest video frame should have been in the next cycle, but the timestamp was advanced by one cycle, causing the calculated rendering time to be in the previous cycle, and the adjusted rendering time to also be in the previous cycle. Therefore, the adjusted rendering time of this video frame should be delayed by one cycle.

[0168] In the embodiments of this application, Figure 10 The flowchart shown is implemented in the frame scheduling algorithm module, which is used to calculate and adjust the rendering time based on the timestamps of the video frames.

[0169] Reference Figure 11 As shown, the frame scheduling algorithm module is set in the unified playback engine, which is located in the application framework layer. The unified playback engine can be called by applications in the application layer. For example, it can be called by any application with video playback function, such as gallery applications, video playback applications, video editing applications, etc.

[0170] The unified playback engine includes an interface layer related to the video playback process, a buffer management module, an encoding / decoding module, an output buffer management module, an audio / video synchronization module, a rendering module, and a frame scheduling algorithm module.

[0171] The interface layer is used to be called by the upper-layer application, so that the upper-layer application can call the modules below the interface layer to implement the corresponding functions.

[0172] During video playback, the encoding / decoding module decodes the video file to obtain decoded data, which is placed in a buffer. The rendering thread retrieves the decoded data from the buffer, renders the decoded data to obtain rendered image data, which is placed in the output buffer and then sent to the screen for display.

[0173] Buffer management is used to manage and store decoded data.

[0174] The video data decoding module in the encoding / decoding module is used to decode video frames, while the audio data decoding module in the encoding module is used to decode audio frames.

[0175] The audio rendering module is used to render audio frames; the video rendering module is used to render video frames.

[0176] Output buffer management is used to manage and store rendered image data.

[0177] Audio-video synchronization is used to synchronize audio and video in time.

[0178] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0179] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0180] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0182] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0185] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video playback method, characterized in that, The method includes: During the playback of the first video, the first moment corresponding to the reference vsync signal is obtained, and the reference vsync signal is the first vsync signal; The first video frame whose rendering time is calculated after the first moment is determined, and the first video frame is the first video frame; Based on the first moment, the first adjustment rendering time of the first video frame is obtained. The first adjustment rendering time is the time offset by 1 / 2T based on the first moment, where T is the period of the real-time detected vsync signal. The first video frame is rendered based on the first adjusted rendering time to obtain a rendered image, which is used to display on the screen.

2. The method as described in claim 1, characterized in that, The step of obtaining the first adjusted rendering time of the first video frame based on the first moment includes: Based on the first moment, the first anchor time is determined, which is the moment offset by multiple vsync signal cycles based on the first moment. Divide the difference between the calculation rendering time of the first video frame and the time of the first anchor point by T to obtain the remainder; The theoretical time of the first vsync signal is obtained by subtracting the remainder from the calculated rendering time of the first video frame. Add 1 / 2T to the theoretical time of the first vsync signal to obtain the first adjustment rendering time of the first video frame.

3. The method as described in claim 2, characterized in that, The step of determining the first anchor point time based on the first moment includes: The difference between the first moment and the calculation rendering time of the first video frame is rounded down to obtain the rounded result; Add the first value to the rounded result and multiply it by the vsync signal period to obtain the offset period; Subtract the offset period from the first moment to obtain the first anchor point time.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Determine a second video frame whose rendering time is after the first moment, wherein the second video frame is a video frame other than the first video frame; Based on the first moment, a second anchor time is determined, wherein the second anchor time is the moment offset by multiple vsync signal cycles based on the first moment. Divide the difference between the calculation rendering time of the second video frame and the time of the second anchor point by T and obtain the remainder; The theoretical time of the second vsync signal is obtained by subtracting the remainder from the calculation and rendering time of the second video frame. The second vsync signal is the latest vsync signal before the calculation and rendering time of the second video frame. Add 1 / 2T to the theoretical time of the second vsync signal to obtain the second adjustment rendering time of the second video frame.

5. The method as described in claim 4, characterized in that, The step of determining the first anchor point time based on the first moment includes: The difference between the calculation and rendering time of the first moment and the second video frame is rounded down to obtain the rounded result; Add the first value to the rounded result and multiply it by the vsync signal period to obtain the offset period; Subtract the offset period from the first moment to obtain the second anchor point time.

6. The method as described in claim 4 or 5, characterized in that, Before rounding down the difference between the calculation and rendering times of the first moment and the second video frame to obtain the rounded result, the process also includes: Calculate the difference between the computation and rendering time of the first moment and the second video frame; It is determined that the difference between the calculation and rendering time of the first moment and the second video frame is less than nT.

7. The method as described in claim 6, characterized in that, After calculating the difference between the computation and rendering times of the first moment and the second video frame, the method further includes: The difference between the calculation and rendering time of the first moment and the second video frame is determined to be no less than nT; The first moment is taken as the second anchor point time.

8. The method according to any one of claims 1 to 7, characterized in that, The reference vsync signal is the latest vsync signal determined according to the first time period; the vsync signal period is the signal period detected in real time according to the second time period.

9. The method as described in claim 8, characterized in that, The first time period and the second time period are the same.

10. The method according to any one of claims 1 to 9, characterized in that, The first calculated rendering time of the first video frame is the rendering time based on the system time, calculated according to the timestamp of the first video frame; the timestamp of the first video frame is the playback time of the first video frame in the first video.

11. The method according to any one of claims 1 to 10, characterized in that, After obtaining the first adjusted rendering time of the first video frame based on the first moment, the method further includes: Determine whether the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is greater than or equal to the second duration; If the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is less than the second duration, then the first adjustment rendering time of the first video frame is extended by T. If the difference between the first adjustment rendering time of the first video frame and the calculated adjustment rendering time of the previous video frame is not less than the second duration, then the first adjustment rendering time of the first video frame will not be delayed.

12. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, which, when executed by the one or more processors, causes the electronic device to perform the method as described in any one of claims 1-11.

13. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-11.

Images