A frame playing method, an electronic device, a chip system and a storage medium

Abstract

The application provides a frame playing method, an electronic device, a chip system and a storage medium, and relates to the technical field of electronic devices. When frames are played, a touch position is collected, a touch time is determined according to the touch position, too dense touch times can be filtered, and multiple touch times are prevented from being repeatedly mapped to the same video frame when dragging is slow. One video frame is displayed in each screen refresh cycle, and when multiple touch times are concentrated in one refresh cycle, the redundant touch times also need to be filtered. After filtering, the touch times are mapped to the nearest video frame or I frame, and the I frame is mapped when dragging is fast, which can reduce the decoding time. Since decoding is in a grouped form and can only be decoded to a specified frame in decoding order, in the case that a video frame to be rendered and displayed is not decoded in time, a substitute rendering display or a rendering display abandonment mode can be used to avoid the phenomenon of always waiting for the video frame to be decoded successfully before rendering and displaying.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a frame playback method, electronic equipment, chip system, and storage medium. Background Technology

[0002] Gallery apps on electronic devices can provide image preview interfaces, allowing users to view various images stored on the device. Gallery apps also provide video playback interfaces, allowing users to play various videos stored on the device.

[0003] The video playback interface can also display the video playback progress, for example, through a progress bar, progress ring, or progress display frame. User drag gestures (press, drag, and release) on the progress bar, progress ring, or progress display frame can trigger the electronic device to play video frames in frame playback mode. However, currently, stuttering often occurs during frame playback due to user drag gestures. Summary of the Invention

[0004] This application provides a frame playback method, electronic device, chip system, and storage medium, which can reduce stuttering during frame playback.

[0005] To achieve the above objectives, the first aspect of this application provides a frame playback method, employing the following technical solution:

[0006] The touch point time is obtained, which is the playback time of the video file mapped from the touch point position during frame playback;

[0007] The touch point time is mapped to candidate video frames, wherein each candidate video frame has: a first timestamp, a decoding state, and a rendering state, and the first timestamp is the system detection time of the touch point position corresponding to the candidate video frame;

[0008] Update the rendering status of the candidate video frame to be rendered, and send the identifier of the candidate video frame updated to be rendered to the queue to be rendered.

[0009] The candidate video frame is decoded, and after the candidate video frame is successfully decoded, the decoding status of the candidate video frame is updated to "decoded".

[0010] According to the rendering and display strategy, candidate video frames in the queue to be rendered are either abandoned for rendering or rendered. The rendering and display strategy includes: if the decoding status of the first candidate video frame in the waiting state is not updated to "decoded" within a first time period, the first candidate video frame is abandoned for rendering; if the decoding status of the first candidate video frame in the waiting state is updated to "decoded" within the first time period, the first candidate video frame is rendered.

[0011] In this application, the frame playback process is random, meaning the video frame to be displayed is related to the user's finger touch position. Even if the touch time is determined based on the user's finger touch position and mapped to a video frame, the video frame needs to undergo decoding and rendering before being displayed on the screen. However, the decoding process doesn't involve immediately decoding a specified video frame. Video frames are decoded in groups, with each frame in its own decoding order, which is not based on timestamps. Therefore, after determining which video frame to display, its decoding order might be the last one decoded in the group. This could lead to situations where, after a certain period, the video frame hasn't been successfully decoded. A rendering strategy can be set, including: if the decoding status of a video frame to be rendered hasn't been updated to "decoded" within a certain time, rendering and displaying that video frame can be abandoned; if the decoding status of a video frame to be rendered is updated to "decoded" within a certain time, rendering and displaying that video frame can be performed. This method avoids the problem of delayed rendering caused by waiting indefinitely for a video frame to be decoded, thus reducing stuttering caused by delayed rendering.

[0012] As one implementation of the first aspect, the step of discarding the rendering and display of candidate video frames in the queue to be rendered or rendering and displaying them according to the rendering and display strategy includes:

[0013] After the first target video frame is successfully rendered, the timer restarts. The first target video frame is the candidate video frame determined for rendering and display in the queue to be rendered.

[0014] After the timing restarts, check the queue for decoded and ready-to-be-rendered candidate video frames.

[0015] If there are decoded and unrendered candidate video frames, the last candidate video frame sent to the rendering queue among the decoded and unrendered candidate video frames will be used as the second target video frame.

[0016] Update the rendering status of the second target video frame to rendered, and update the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the second target video frame to abandoned rendering.

[0017] Render the second target video frame;

[0018] The timing restarts after the second target video frame is successfully rendered.

[0019] In this application, the timer restarts whenever a video frame is successfully rendered. Since video frames are decoded in groups, when a certain video frame a is decoded, other video frames (including video frame b) in the same video frame group are also successfully decoded. If another video frame b needs to be rendered later, it has already been successfully decoded. Therefore, in practical applications, we can first check in the rendering queue whether there is a candidate video frame that has been decoded and is ready to be rendered. If such a video frame (e.g., video frame b) exists, we can directly obtain video frame b for rendering.

[0020] Of course, if the user drags their finger slowly, the multiple video frames that need to be displayed may belong to the same GOP group. In this case, the differences between the video frames within the same GOP group are usually small. At the same time, the electronic device screen has a refresh cycle (the number of video frames rendered and displayed within a certain time is limited). If multiple frames are rendered and displayed at the current time, the number of video frames rendered and displayed later will be reduced (which may result in a large difference between the displayed video frame and the user's drag position). Therefore, even if there are multiple video frames within the same GOP group in the queue to be rendered, the candidate video frame that was last sent to the queue to be rendered from the decoded and unrendered candidate video frames can be selected as the video frame to be rendered and displayed.

[0021] In addition, even if they are not in the same GOP group, some video frames can be decoded faster if they are decoded earlier (e.g., the first I-frame in the decoding order). They may have been successfully decoded during the rendering of the previous video frame. Therefore, after the timer restarts, if there are decoded candidate video frames in the rendering queue that are waiting to be rendered, these video frames can be directly obtained for rendering.

[0022] Similarly, even if there are video frames that do not belong to the same GOP group among the decoded and unrendered candidate video frames, the electronic device screen has a refresh cycle. If multiple frames are rendered and displayed at the current time, the number of video frames rendered and displayed will be reduced in the future (which may result in frame loss and screen tearing). Therefore, even if there are video frames that do not belong to the same GOP group in the queue to be rendered, the last candidate video frame sent to the queue to be rendered from the decoded and unrendered candidate video frames (regardless of whether it belongs to the same GOP group as other decoded and unrendered video frames) can be selected as the video frame to be rendered and displayed.

[0023] After each successful rendering, the timer restarts, entering the next timing cycle.

[0024] Additionally, before each rendering, the rendering status of the video frame to be rendered in the current batch can be updated to "rendered," while the rendering status of the video frames before this current video frame (those sought before this current video frame, determined by the system time corresponding to the touch point) can be updated to "abandon rendering." This is because, in practical applications, video frames before this current video frame have already passed their rendering window.

[0025] As another implementation of the first aspect, after checking whether there are decoded and unrendered candidate video frames in the queue to be rendered, the method further includes:

[0026] If there are no decoded and unrendered candidate video frames in the queue to be rendered, then the decoding status of the second candidate video frame in the queue to be rendered is monitored. The second candidate video frame is the first candidate video frame in the queue to be decoded and unrendered.

[0027] If the decoding status of the second candidate video frame is updated to "decoded" within the second time period, then the second candidate video frame is used as the third target video frame.

[0028] The rendering status of the third target video frame is updated to "rendered", and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the third target video frame is updated to "abandon rendering".

[0029] Render the third target video frame;

[0030] The timer restarts after the third target video frame is successfully rendered.

[0031] In this application, at the very beginning of the re-timing, if there are no decoded video frames to be rendered in the rendering queue, the decoding status of the first undecoded video frame to be rendered can be monitored within the current timing period. If it is within the allowed time range (the second duration), the video frame is updated to be decoded, and then the video frame can be rendered and displayed. This rendering and display process will not be in a state of waiting for decoding indefinitely, avoiding the rendering and display delay (stuttering phenomenon) caused by waiting for decoding indefinitely.

[0032] As another implementation of the first aspect, the process of monitoring the decoding status of the second candidate video frame in the queue to be rendered further includes:

[0033] If the decoding status of the second candidate video frame is not updated to decoded within the second time period, it is determined whether the second candidate video frame and the third candidate video frame in the queue to be rendered belong to the same video frame group. The third candidate video frame is the candidate video frame that is adjacent to and follows the second candidate video frame in the queue to be rendered.

[0034] If the second candidate video frame and the third candidate video frame belong to the same video frame group, then the third candidate video frame replaces the second candidate video frame, and the decoding status of the third candidate video frame continues to be monitored within the current timing period.

[0035] If the decoding status of the third candidate video frame is updated to "decoded" within the third time period, then the third candidate video frame is used as the fourth target video frame. The starting point of the third time period is the same as the starting point of the second time period.

[0036] The rendering status of the fourth target video frame is updated to "rendered", and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the fourth target video frame is updated to "abandon rendering".

[0037] Render the fourth target video frame;

[0038] The timing restarts after the fourth target video frame is successfully rendered.

[0039] In this application, if a video frame has not been successfully decoded within the allowed time range, it indicates that the decoding order of the video frame may be relatively late. If the next video frame in the rendering queue happens to belong to the same GOP group as the current video frame, then the next video frame may have been successfully decoded or its decoding order may be earlier than the current video frame. The next video frame can replace the current video frame (the content difference within the same GOP group is not significant), and the decoding status of the next video frame can continue to be monitored within the current timing period. That is, there is no need to restart the timing, but to continue timing. If the decoding status of the next video frame has changed to "decoded", the next video frame can be immediately obtained for rendering and display. If it is also "undecoded", the decoding status of the next video frame can continue to be monitored. If the next video frame is successfully decoded within the third time period, the next video frame can be rendered and displayed.

[0040] As another implementation of the first aspect, the method further includes, during the process of monitoring the decoding status of the third candidate video frame:

[0041] If the decoding status of the third candidate video frame is not updated to decoded within the third time period, the abandonment flag of the candidate video frame from the second candidate video frame to the fourth candidate video frame in the queue to be rendered is obtained. The fourth candidate video frame is the first candidate video frame in the queue to be rendered that is located after the second candidate video frame and is not in the same video frame group as the second candidate video frame.

[0042] The decoding status of the candidate video frame corresponding to the abandonment flag is updated to abandon decoding, and the rendering status is updated to abandon rendering. The rendering status of the candidate video frame whose first timestamp is before the first timestamp of the second candidate video frame is updated to abandon rendering.

[0043] After updating to abandon rendering, the timer restarts.

[0044] In this application, if, after extending to the third duration, another video frame within the same GOP group also fails to be decoded successfully, it indicates that there may be a problem with the decoding of this group of video frames. In this case, all other video frames in the same GOP group as the currently monitored video frame in the rendering queue will be abandoned for decoding and rendering. After abandoning the decoding of this group of video frames, the decoder can continue to decode the next group of video frames. Of course, the decoder does not need to continue decoding the video frame group that has been abandoned for decoding, but instead selects the video frame group containing the video frame that is currently in the state of being decoded for decoding.

[0045] In addition, even if rendering is abandoned, the timing needs to be restarted to enter the next timing cycle and determine whether to render and display the next video frame or abandon it.

[0046] As another implementation of the first aspect, after determining whether the second candidate video frame and the third candidate video frame in the queue to be rendered belong to the same video frame group, the method further includes:

[0047] If the second candidate video frame and the third candidate video frame do not belong to the same video frame group, the decoding status of the second candidate video frame will continue to be monitored within the current timing period.

[0048] If the decoding status of the second candidate video frame is updated to "decoded" within the third time period, then the second candidate video frame is used as the fifth target video frame.

[0049] The rendering status of the fifth target video frame is updated to "rendered", and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the fifth target video frame is updated to "abandon rendering".

[0050] Render the fifth target video frame;

[0051] The timer restarts after the fifth target video frame is successfully rendered.

[0052] In this application, if the next candidate video frame and the currently monitored candidate video frame do not belong to the same GOP group, the difference in the picture content may be large. Therefore, the next candidate video frame cannot be used to replace the current candidate video frame for rendering and display, as the next candidate video frame has its own timing period. Instead, the current video frame is monitored. If the video frame has been successfully decoded before the third duration of monitoring is reached, the currently monitored video frame can be rendered and displayed.

[0053] As another implementation of the first aspect, in the process of continuing to monitor the decoding status of the second candidate video frame within the current timing period, it also includes:

[0054] If the decoding status of the second candidate video frame is not updated to decoded within the third time period, the rendering status of the second candidate video frame is updated to abandon rendering, and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the second candidate video frame is updated to abandon rendering.

[0055] After updating to abandon rendering, the timer restarts.

[0056] In this application, since the next candidate video frame cannot be used to replace the current video frame for rendering and display, if the current video frame is not updated to be decoded within a certain period of time, the current video frame will be abandoned for rendering and display to avoid rendering delays caused by waiting indefinitely and resulting in stuttering.

[0057] As another implementation of the first aspect, the method further includes, before mapping the touch point moment to the candidate video frame:

[0058] Get the screen refresh rate;

[0059] The first moment is taken as the starting point of the first filtering cycle, and the duration of the first filtering cycle is equal to the duration of the screen refresh cycle.

[0060] If any contact moment falls within the time range of the first filtering cycle, then that contact moment is retained;

[0061] If at least two contact moments fall within the time range of the first filtering cycle, one contact moment will be retained from the multiple contact moments within the time range of the first filtering cycle, and the other contact moments other than the retained contact moment will be deleted.

[0062] In this application, if the touch moments are too dense, they may be mapped to the same video frame. To avoid repeated mapping to the same video frame, some touch moments can be deleted. On the other hand, since the screen has a refresh cycle, that is, the number of video frames displayed on the screen in 1 second is limited, it is meaningless to retain too many video frames mapped by touch moments. Therefore, the screen refresh cycle can be used as a benchmark so that basically one touch moment (mapped to one video frame) is retained in each screen refresh cycle.

[0063] The first moment is the end of the previous filtering cycle or the contact moment determined in the previous filtering cycle.

[0064] When the first moment is the end of the previous filtering cycle, the start time of the first filtering cycle can be 0ms.

[0065] As another implementation of the first aspect, the video file includes multiple video frames, each video frame in the video file carrying a second timestamp, the second timestamp of the video frame representing the playback time point of the video frame in the video file; the step of mapping the touch point time to candidate video frames includes:

[0066] The video frame with the smallest difference between the second timestamp and the touch point time in the video file is selected as the candidate video frame for the touch point time mapping.

[0067] As another implementation of the first aspect, during the forward sliding process, the second timestamps of the candidate video frames mapped by two adjacent touch points are increased from small to large; when the video frame mapped by one of the touch points is the fifth candidate video frame, the next adjacent candidate video frame of the fifth candidate video frame is the sixth candidate video frame, and the second timestamp of the sixth candidate video frame is greater than the second timestamp of the fifth candidate video frame.

[0068] If the fifth candidate video frame and the sixth candidate video frame are located in different video frame groups and there is a key frame between the fifth candidate video frame and the sixth candidate video frame, then the fifth candidate video frame is adjusted to be the key frame with the smallest difference between the second timestamp of the fifth candidate video frame and the second timestamp of the fifth candidate video frame among the key frames between the fifth candidate video frame and the sixth candidate video frame.

[0069] As another implementation of the first aspect, during the reverse sliding process, the second timestamps of the candidate video frames mapped by two adjacent touch points are from large to small; when the video frame mapped by one of the touch points is the seventh candidate video frame, the preceding candidate video frame adjacent to the seventh candidate video frame is the eighth candidate video frame, and the second timestamp of the eighth candidate video frame is greater than the second timestamp of the seventh candidate video frame.

[0070] If the seventh candidate video frame and the eighth candidate video frame are located in different video frame groups and there is a key frame between the seventh candidate video frame and the eighth candidate video frame, then the seventh candidate video frame is adjusted to be the key frame with the smallest difference between the second timestamp of the seventh candidate video frame and the second timestamp of the seventh candidate video frame among the key frames between the seventh candidate video frame and the eighth candidate video frame.

[0071] In this application, some video frames (with fast scrolling and sparse touch points) can be replaced with adjacent I-frames (the first video frame decoded in each GOP group) to reduce decoding time and reduce stuttering.

[0072] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0073] Thirdly, a chip system is provided, including a processor coupled to a memory, wherein the processor executes a computer program stored in the memory to cause an electronic device to implement the method of any one of the first aspects of this application.

[0074] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0075] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0076] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0077] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0078] Figure 2 A schematic diagram of an interface for a normal playback mode and a frame playback mode provided in an embodiment of this application;

[0079] Figure 3 A schematic diagram illustrating the relationship between the playback order and decoding order of video frames in a video frame group, provided for an embodiment of this application;

[0080] Figure 4 A flowchart illustrating the normal playback mode and frame playback mode in the playback engine provided in this application embodiment;

[0081] Figure 5 A timing diagram of a frame playback process with buffering provided for an embodiment of this application;

[0082] Figure 6 A schematic diagram illustrating the timing of the filtering contact points provided in an embodiment of this application;

[0083] Figure 7 This is a schematic diagram illustrating how a touch point moment is mapped to a video frame, as provided in an embodiment of this application.

[0084] Figure 8 This is a schematic diagram illustrating how the video frame mapped to the touch point moment is adjusted to a nearby I-frame, as provided in an embodiment of this application.

[0085] Figure 9 A timing diagram of a frame playback process without buffering, provided for an embodiment of this application;

[0086] Figure 10 Provided for the embodiments of this application Figure 9 A flowchart illustrating the display strategy in the document;

[0087] Figure 11 Provided for the embodiments of this application Figure 10 A schematic diagram of a display strategy corresponding to A4 size;

[0088] Figure 12 A schematic diagram illustrating another display strategy provided in an embodiment of this application;

[0089] Figure 13 This is a schematic diagram illustrating another display strategy provided in an embodiment of this application. Detailed Implementation

[0090] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0091] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0092] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0093] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0094] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0095] This application provides a frame playback method that can be applied to electronic devices. These electronic devices can be tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not limit the specific type of electronic device.

[0096] Figure 1A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0097] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0098] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0099] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0100] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback). Touch sensor 180K, also called a "touch panel," can be disposed on display screen 194. Touch sensor 180K and display screen 194 together form a touch screen, also called a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor can transmit the detected touch operation to application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, in a different location than display screen 194.

[0101] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0102] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0103] This application does not specifically limit the structure of the execution subject of a frame playback method. As long as the code recording a frame playback method according to this application is executed, communication can be performed according to the frame playback method provided in this application. For example, the execution subject of a frame playback method provided in this application can be a functional module in an electronic device capable of calling and executing a program, or a communication device applied in an electronic device, such as a chip.

[0104] Users can use the gallery app on their electronic devices to view various photos stored on the device and play various videos stored on the device.

[0105] Reference Figure 2 This is a schematic diagram of an interface for playing videos stored on an electronic device through a gallery application, as provided in an embodiment of this application.

[0106] Reference Figure 2 (a) in the image gallery is a grid of images and videos stored on an electronic device, displayed by the image gallery application of the electronic device. The grid can display images and videos, where the cell containing a video displays the cover and duration of the video, and the cell containing an image displays a thumbnail of the image.

[0107] The grid image can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app, or it can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app and triggers more steps.

[0108] When displaying images and / or videos stored on an electronic device, this grid chart can be presented in various ways, for example, Figure 2 The method shown in (a) displays photos and videos in a photo group in reverse chronological order; in practical applications, gallery applications can also set up multiple object groups, for example, through... Figure 2 The control 20 shown in (a) can view multiple groups in the album: image group, video group, screenshot and screen recording group, etc. Users can also display pictures through the image group in the gallery application, videos through the video group in the gallery application, and screenshots and screen recordings through the screenshot and screen recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the grid diagram shown above.

[0109] This application does not limit the number of user operations required from the system desktop to the display of the grid chart in the gallery application, nor does it limit the way the grid chart displays images and videos.

[0110] Reference Figure 2 (b) in the text represents the user's click. Figure 2 Following the cover of video A in the grid diagram shown in (a), the electronic device displays the video playback interface of video A in response to the operation. The video playback interface includes a video playback area 21 and a progress display area 22.

[0111] The video playback area 21 is used to display the video frame during video playback. After entering the video playback interface from the grid interface, the control 211 is displayed in the video playback area. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the 0th second. When the video is paused, the control 211 is also displayed in the video playback area 21. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the current paused moment.

[0112] The progress display area 22 is used to display the video playback progress. The progress display area 22 includes a progress display frame 221 and a progress indicator line 222. The position of the progress indicator line 222 can remain unchanged. The user can drag the progress display frame 221 located below the progress indicator line 222 so that the progress display frame 221 moves left or right without moving the progress indicator line 222. The position of the progress indicator line 222 above the progress display frame indicates the current playback progress of the video.

[0113] As an example, the total playback duration of the current video is 15 seconds, and the total length of the current video's progress display frames is represented by L pixels. Based on these two parameters, we can obtain the playback duration corresponding to a unit length (or the length corresponding to a unit of time). For example, divide the playback duration 15 by the total length L. Therefore, if the leftmost edge of the progress display frame (the position corresponding to the first pixel) corresponds to second 0 of the video, then different positions of the progress indicator line on the progress display frame will correspond to different playback times. For example, if the distance between the position of the progress indicator line on the progress display frame and the leftmost edge of the progress display frame is X, then the playback time corresponding to the progress indicator line is: X*15 / L.

[0114] For ease of subsequent description, the embodiments in this application are described in the manner of "the time corresponding to the progress indicator line" or "the contact time".

[0115] Due to space limitations in the progress display area 22, only a portion of the video frames or a portion of the video frames from the progress display frame 221 can be displayed in the progress display area 22. For example... Figure 2 As shown in (b), the progress display frame 221 displayed in the progress display area 22 includes: thumbnails of three complete video frames and a partial area of ​​a thumbnail of a video frame.

[0116] The number of thumbnails displayed in the progress bar is usually related to the video length. Generally, the longer the video, the more thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames), and the shorter the video, the fewer thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames).

[0117] Typically, the video playback area 21 displays the video frame corresponding to the time indicated by the progress indicator line 222 in the progress display area 22.

[0118] In a specific implementation, there is a situation where the time corresponding to the progress indicator line 222 is the time between the playback times of two adjacent video frames, that is, there is no video frame in the video file corresponding to the time of the progress indicator line 222.

[0119] As an example, in a video frame sorted by playback time (denoted as timestamp) from smallest to largest, the playback time of the i-th video frame is ti, and the playback time of the (i+1)-th video frame is t(i+1); the time t corresponding to the progress indicator line 222 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval to the time t corresponding to the progress indicator line 221 can be taken as the video frame corresponding to the time of the progress indicator line 222.

[0120] Of course, in practical applications, if there are no video frames in the video file whose timestamps and progress indicator lines correspond to the same time, the video frame preceding time t (the video frame corresponding to ti) of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 221; or the video frame following time t (the video frame corresponding to t(i+1)) of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 222. In practical applications, the progress display area 22 can display the playback progress in many ways. For example, it can also display the video playback progress using a progress bar or a progress ring.

[0121] This application provides two playback modes for video playback: normal playback and frame-by-frame playback. The embodiments in this application only illustrate... Figure 2 This serves as an example to illustrate the difference between normal playback and frame playback.

[0122] Reference Figure 2 (c) in the text represents the user's click. Figure 2 After the control 211 (which can also be referred to as the playback control) in the interface shown in (b) is displayed, the electronic device responds to the operation by displaying an interface diagram during the playback of video A. This interface diagram indicates that the video A has been played to the 5th second, and the video frame corresponding to the 5th second is displayed in the video playback area 21. At the same time, the progress display frame 221 in the progress display area 22 moves its position so that the time corresponding to the progress indicator line 222 is the 5th second. Among them, from Figure 2 (b) in Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0123] A video file consists of multiple video frames, each with a timestamp (also known as a second timestamp). The timestamp records the playback time of each video frame during the video file's playback process. For example, arranged by timestamp from smallest to largest, the video frames in the file are: Video Frame 0 (timestamp t0), Video Frame 1 (timestamp t1), Video Frame 2 (timestamp t2), Video Frame 3 (timestamp t3), Video Frame 4 (timestamp t4), Video Frame 5 (timestamp t5), Video Frame 6 (timestamp t6), Video Frame 7 (timestamp t7)... Normal playback involves displaying each video frame sequentially in the playback area according to its corresponding timestamp, from smallest to largest.

[0124] As an example of a normal playback process, video frame 0 is displayed in video playback area 21 at time t0 after playback begins; video frame 1 is displayed in video playback area 21 at time t1 after playback begins; video frame 2 is displayed in video playback area 21 at time t2 after playback begins; video frame 3 is displayed in video playback area 21 at time t3 after playback begins; video frame 4 is displayed in video playback area 21 at time t4 after playback begins; video frame 5 is displayed in video playback area 21 at time t5 after playback begins; video frame 6 is displayed in video playback area 21 at time t6 after playback begins; video frame 7 is displayed in video playback area 21 at time t7 after playback begins…

[0125] from Figure 2 (b) in Figure 2 (c) shows the normal playback process of video A: starting from the 0th second of video A, each video frame is displayed sequentially according to the playback time indicated by the timestamp; Figure 2 (c) in the diagram is a schematic of the interface displaying the video frame up to the 5th second.

[0126] Reference Figure 2 In (d), the user drags... Figure 2 The progress display frame of the interface shown in (c) is in response to... Figure 2 As shown in (d) above, the electronic device displays the corresponding screen in the video playback area for the drag operation.

[0127] As an example, refer to Figure 2 In step (e), when the user drags the progress bar to display a frame such that the progress indicator line corresponds to the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be a video frame in video A with a timestamp of 14 seconds; it can also be a video frame with a timestamp closest to 14 seconds; it can also be a video frame before 14 seconds whose time interval between timestamp and 14 seconds is closest; or it can be a video frame after 14 seconds whose time interval between timestamp and 14 seconds is closest.

[0128] from Figure 2 (d) in Figure 2 The playback process shown in (e) is the frame playback process of the video.

[0129] As the user drags the progress display frame 221, a relative position is formed between the progress indicator line 222 and the progress display frame 221. The position of the progress indicator line 222 on the progress display frame 221 is related to the time corresponding to the progress indicator line 222.

[0130] In the specific implementation, assuming that the time indicated by the progress indicator line 222 is t1 before the user moves in the area where the progress display frame 221 is located on the screen. When the user moves left or right in the area where the progress display frame is located, the electronic device can detect the user's real-time touch position at certain time intervals. The electronic device determines the user's movement direction and distance based on the continuous real-time touch positions. The electronic device determines the time change Δt based on the user's movement distance; and determines the calculation method for the touch time t2 based on the user's movement direction. For example, if the movement direction is left, the touch time t2 is obtained by subtracting the time change Δt from the starting time t1; if the movement direction is right, the touch time t2 is obtained by adding the time change Δt to the starting time t1. After obtaining the touch time t2, the electronic device searches for the video frame corresponding to the touch time t2 and displays the video frame corresponding to the touch time t2 in the video playback area.

[0131] It's understandable that the time or touch point time corresponding to the progress indicator line is not the system time, but rather the playback time in the video file where the touch point position is mapped. The video frame corresponding to touch point time t2 can be found in the detailed explanation of "the video frame corresponding to the 14th second". The video frame corresponding to touch point time t2 can also be understood as the video frame mapped to touch point time t2.

[0132] On the other hand, the electronic device obtains the movement direction of the progress display frame based on the movement direction determined by the real-time touch point position, and determines the movement distance of the progress display frame based on the movement distance of the real-time touch point position; the electronic device moves the progress display frame in the progress display area according to the movement direction and movement distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0133] As another example, the electronic device also determines the moving direction (left if t2 is greater than t1, right if t2 is less than t1) and moving distance of the progress display frame based on the start time t1 and the touch time t2; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0134] It should be noted that the above process is only an example. In actual applications, there are multiple ways to determine the touch point time corresponding to the progress indicator line 222, the position of the progress display frame 221, and the video frame displayed in the video playback area 21 when the user drags the progress display frame 121.

[0135] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. It may move to the left, it may move to the right, it may be dragged quickly, or it may be dragged slowly. Accordingly, the video frame images displayed in the video playback area 21 have randomness.

[0136] As an example, before the user moves within the area of ​​the progress display frame on the screen, the starting time indicated by the progress indicator is t1. During the user's dragging of the progress display frame, the electronic device detects the user's real-time touch position at regular time intervals, thus obtaining multiple consecutive time periods corresponding to the real-time times: t2, t3, t7, t4, and t3. In the video playback area, video frames are played in the following order: starting with the video frame corresponding to time t1, the video frames corresponding to time t2, t3, t7, t4, and t3 are played sequentially.

[0137] It can be understood that the process of a user clicking control 21 to trigger the electronic device to play video in the video playback area is normal playback; the process of a user dragging the progress display frame to trigger the electronic device to display the video frame corresponding to the progress indicator line in the video playback area is frame playback.

[0138] Figure 2 The example shown is intended to illustrate the difference between normal playback and frame playback, and does not imply that the interface during normal playback and frame playback must be identical. Figure 2 Consistent with the above.

[0139] In the above embodiments, whether it is normal playback or frame playback, the video frames need to be decoded before they are displayed in the video playback area. Usually, the video frames are decoded in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a keyframe (I-frame), at least one forward reference frame (P-frame), and at least two bidirectional reference frames (B-frames).

[0140] Reference Figure 3 This is a schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application. A set of video frames can also be referred to as a group of pictures (GOP).

[0141] Each group of video frames, in playback order, includes: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. Among these, the I-frame, as a keyframe, can be decoded to obtain a complete image, but decoding is slower. The P-frame records the changes relative to the previous frame (not a B-frame); information from the previous frame is required to decode the P-frame. Similarly, information from previous P-frames (which may also be I-frames) and subsequent P-frames is needed to decode the B-frame. P-frames and B-frames improve video smoothness while reducing the file size.

[0142] Whether it's an I-frame, P-frame, or B-frame, they are all video frames. When decoding a video frame at a specific moment, you don't directly decode the video frame at that moment. Instead, you decode the group of video frames that the video frame at that moment belongs to. When decoding video frames within a group of video frames, you decode them one by one in the decoding order.

[0143] As an example, if we need to decode the video frame at time t1, and the video frame at time t1 is the 6th video frame in the playback order of a group of video frames: frame B; then the decoding order is:

[0144] First decoding: the video frame played in the first playback order: I-frame;

[0145] The second decoding: the 4th video frame in playback order: P-frame;

[0146] The third decoding step: the second video frame played in the playback order: frame B;

[0147] The fourth decoding: the third video frame in playback order: frame B;

[0148] The fifth decoding: the 7th video frame in playback order: P-frame;

[0149] The sixth decoding: the 5th video frame in playback order: frame B;

[0150] The seventh decoding: the 6th video frame in playback order: frame B;

[0151] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding operation to decode the video frame at time t1.

[0152] Currently, in normal playback scenarios, the video frames in each video frame group are parsed, decoded, rendered, and displayed sequentially according to the playback order. Of course, when decoding the video frames in each video frame group, the process is as follows: Figure 3 The decoding order shown decodes each video frame in a video frame group sequentially. Normally, in a continuous playback scenario, when displaying frames from the previous video frame group, all frames in the next video frame group have already been decoded. However, when switching from frame playback mode to normal playback mode, it's necessary to determine the video frame group containing the latest frame based on its playback time, and then decode the video frames sequentially according to the above decoding order.

[0153] In a frame playback scenario, the process of a user dragging the progress bar to display a frame is uncertain. The user may drag the progress bar to the left, or to the right, or drag it quickly or slowly. Consequently, the video frame images displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display a frame, the electronic device needs to determine the moment the progress indicator line points based on the real-time detected touch point position, and then decode the video frames sequentially according to the above decoding order.

[0154] However, since the user's dragging process may be fast or slow, if the method described in the above embodiment is followed exactly: each time a touch point is detected, the touch point is mapped to a video frame, the video frame is decoded (in groups and in the order of the group's decoding), and then rendered and displayed, there may be a phenomenon where the video frame displayed in frame playback does not correspond to the real-time touch point position of the user's finger, resulting in screen stuttering.

[0155] This application provides a frame playback method. During frame playback, touch point positions are collected, and touch point moments are determined based on the touch point positions. This can filter out some overly dense touch point moments, avoiding the repeated mapping of multiple touch point moments to the same video frame when dragging is slow. It also avoids the situation where multiple video frames mapped to touch point moments exist within each refresh cycle, causing some video frames to not be rendered even if they are successfully decoded, resulting in wasted decoding resources. After filtering, the touch point moments are mapped to the nearest video frame or I-frame. Mapping to I-frames when dragging is fast can reduce decoding time. Since decoding is done in groups, and the specified frame can only be decoded in the order of decoding, if a video frame that is about to be rendered is not decoded in time, alternative rendering or abandoning rendering can be used to reduce the stuttering phenomenon that occurs when waiting for the video frame to be successfully decoded before rendering.

[0156] The frame playback method provided in the embodiments of this application is described in detail below. Both the normal playback and frame playback provided in the embodiments of this application are located in the playback engine.

[0157] Reference Figure 4 In the playback engine, both normal playback and frame-by-frame playback have their own corresponding players. For example, the normal player is used to parse, decode, and render video frames during normal playback; the frame player is used to parse, decode, and render video frames during frame-by-frame playback. The parsing, decoding, and rendering processes for normal playback and frame-by-frame playback can be completely independent. Of course, frame-by-frame playback can also reuse the parser and decoder from normal playback, or reuse the parsed data and decoded data successfully parsed and decoded during normal playback.

[0158] In normal playback, the data source is a video file. The video file is first parsed to obtain the audio stream and video frames. Specifically, the audio stream is decoded by an audio decoder, and the video stream is decoded by a video decoder. After synchronizing the decoded audio stream and video frames, the video frames in the video stream are rendered and displayed, and the audio frames in the audio stream are rendered and played.

[0159] Additionally, dragging the progress frame displayed in the progress display area by the user triggers frame playback; however, during normal playback, the progress frame in this area also moves as normal playback progresses. Therefore, both normal playback and frame playback require generating thumbnails of multiple video frames to display the progress frame in the progress display area.

[0160] The thumbnail management module can determine the number of video frames to be extracted and the specific video frames based on the length of the video file. Then, it retrieves the video frames recorded in the data table that are used to generate progress display frames. These video frames can be parsed and decoded in the frame player to obtain decoded YUV data, generate thumbnails and cache them. Subsequently, during normal playback and frame playback, the cached data is used to display these progress display frames in the progress display area.

[0161] During frame playback, the user slides or drags their finger on the progress display frame in the progress display area. The seek manager can maintain a seek data table, which records the following information: the time of each touch point in the sliding or dragging operation, the system time (which can also be understood as the detection time) for capturing each touch point, the video frame mapped to each touch point (which can be recorded as a candidate video frame), the information of each video frame, and the decoding and rendering status of each video frame.

[0162] Touch point time: The time when the progress indicator line points to is determined based on the touch point position, indicating the moment when the user drags their finger to the current touch point position and expects to display the image from the video file at that moment; for example, if the touch point time is 1015ms, it means that the user expects to display the image from the 1015ms position in the video file.

[0163] System detection time at the moment of contact: The system time at which the contact moment is captured. For example, if the contact moment (at 1015ms) is captured at 13:54:24 on the electronic device, then the system time of the contact moment (at 1015ms) is 13:54:24. Therefore, it can also be recorded as system time, detection time, or first timestamp.

[0164] Video frame mapped to touch point time: A video file contains multiple video frames, and each video frame also has another time point, that is, the playback time point of the video frame in the video file, which can be recorded as the second timestamp. Among the second timestamps of the video frames in the video file, the video frame with the smallest difference from the touch point time is selected. For example, the video frame mapped to the touch point time (1015ms) is the video frame with the second timestamp of 1016.67ms.

[0165] Information about each video frame: for example, whether the video frame is an I-frame, a P-frame, or a B-frame.

[0166] Decoding status of each video frame: Decoding status includes pending decoding, decoded, and abandoned decoding;

[0167] Rendering status of each video frame: The rendering status includes pending rendering, rendered, and abandoned rendering.

[0168] For ease of understanding and description, the system detection time will be used to represent the first timestamp, and the second timestamp will be used to represent the second timestamp.

[0169] The video frames mapped to each touchpoint in the seek data table (initially in a state awaiting decoding) also need to be parsed, decoded, and rendered before display. Before decoding, the cache manager can check whether the video frame is cached. If the video frame is not cached, it is handed over to the decoder by the decoding manager, and the rendering status of the video frame in the seek data table is updated to await rendering. Then, the identifier of the video frame is sent to the await-rendering queue. The display manager maintains the await-rendering queue and can detect the decoding status of the video frames in the await-rendering queue in the seek data table. Based on the decoding status, it determines whether to abandon rendering or display; if rendering is to be performed, the video frame is rendered. Of course, regardless of the display strategy used, the status of the video frames in the seek data table needs to be updated according to the result of the strategy. In addition, if the video frame is cached, the cached video frame can be retrieved (without decoding) and directly rendered. Of course, the status of the video frames in the seek data table also needs to be updated after rendering.

[0170] To make the frame playback process in the playback engine clearer, the frame playback process will be described in detail through subsequent embodiments.

[0171] First, we will describe the normal playback process, without using illustrations as examples.

[0172] S101, while the gallery application is displaying the video playback interface of video A, it receives a click operation on a regular playback control. For example, Figure 2 The click operation on the control shown in (b) is shown in the image.

[0173] S102, the gallery application launches the playback engine.

[0174] S103, the playback engine sends a playback command for video A to a regular player.

[0175] S104: A standard media player parses video file A, obtaining H.264 format video frames (for example only). It then decodes these H.264 frames to obtain YUV data, and renders the YUV data to produce a rendered image. This rendered image can be displayed in the video preview area of ​​the gallery application.

[0176] This process corresponds to Figure 2 (b) to Figure 2 (c) in the middle.

[0177] While video A is playing in normal playback mode in the gallery app, a progress display area is shown at the bottom of the screen. The user can drag the progress frames in this area to trigger the electronic device to play video A frame by frame. The touchpad of the electronic device monitors and reports the touch point position at a certain frequency, and the touch point position of the user's finger can determine the touch time.

[0178] The following is through Figure 5 Describe the frame playback process in detail. This process corresponds to... Figure 2 (d) to Figure 2 (e) in the middle.

[0179] S201, the playback engine receives the reported touch time t1 and the corresponding system time T1 (or detection time T1).

[0180] In this embodiment of the application, if the reported touch point time t1 is the user's first touch point time, the playback engine also needs to send a pause command to the ordinary player, and the ordinary player stops parsing, decoding and rendering according to the pause command.

[0181] S202, the playback engine sends the touch point time t1 and system time T1 to the seek management module.

[0182] S203, the playback engine sends a parsing instruction to the parser, carrying the contact time t1.

[0183] S204, the parser parses the video file and obtains the parsed data of the video frames in the GOP where t1 is located, for example, H.264 format data.

[0184] In practical applications, video frames in a video file are divided into multiple Groups of Pictures (GOPs). Each GOP has a playback start time (e.g., the timestamp of the first video frame in the current GOP) and a playback end time (the timestamp of the first video frame in the next GOP). Of course, this is just an example. In practice, there is a time interval between two adjacent video frames in two GOPs (e.g., 16.67ms in a 60FPS scenario). Therefore, there are multiple ways to determine the playback time range for each GOP. The video frames in the GOP containing t1 can be parsed. In practice, the parser can also parse one or two more GOPs. For example, upon receiving a parsing instruction carrying t1, it can parse the GOP containing t1, as well as the video frames in the GOPs before and after it. If the parser is fast enough, it can even parse the entire video frame file.

[0185] S205, the playback engine receives the reported touch point time t2 and the corresponding system time T2.

[0186] Contact time t2 and contact time t1 are in the same GOP group.

[0187] S206, the playback engine sends t2 and T2 to the seek management module.

[0188] S207, the seek management module sends a parsing instruction to the parser, carrying the contact time t2.

[0189] S208, the parser determines that the video frames in the GOP group where t2 is located have been parsed, and will not repeat the parsing.

[0190] Similarly, when a parsing instruction carrying t2 is received, if t2 and t1 are in the same GOP group, the parsing will not be repeated. If t2 and t1 are not in the same GOP group, the video frames of the GOP group where t2 is located, as well as the video frames of the GOP groups before and after it, will be parsed. Of course, if any of the three GOP groups corresponding to t2 have been successfully parsed, only the GOP groups that have not been successfully parsed need to be parsed.

[0191] S209, the seek manager retains a contact time within the range of the refresh cycle in each refresh cycle based on the received contact time. For example, t1 and t2 retain t2.

[0192] Reference Figure 6 This is the process provided in the embodiments of this application for retaining a contact moment within the range of that refresh cycle in each refresh cycle.

[0193] The frequency of contact reporting is usually high, generally at least 120Hz. The contact time is related to the real-time position of the user's finger. When the user's finger slides faster, the contact time distribution is more scattered (the difference between adjacent contact times is larger). When the user's finger slides slower, the contact time distribution is more concentrated (the difference between adjacent contact times is smaller).

[0194] With a screen refresh rate of 60Hz, this means that one video frame is rendered and displayed every 16.67ms. When the user's finger swipes slowly, multiple touch points may be received within a single 16.67ms filtering cycle. In other words, the duration of the filtering cycle can be equal to the duration of the screen refresh cycle.

[0195] like Figure 6 As shown, when the user's finger drags slowly, the touch point may be at 4ms, 6ms, 13ms, 20ms, 23ms, and 35ms.

[0196] In practical applications, it is not necessary to map every touch point moment to a specific video frame. For example, 4ms and 6ms might map to the same video frame, and 13ms, 20ms, and 23ms might also map to the same video frame. Therefore, it is necessary to filter out some touch point moments.

[0197] Reference Figure 6 (a) illustrates a filtering method provided in this application embodiment. This method aligns the filtering period T with the timestamps of video frames. For example, if the video file has a frame rate of 60 FPS, using the timestamps of two adjacent video frames as the dividing points between different periods, any touchpoint moment before 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. Alternatively, the first touchpoint moment before 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. Furthermore, the last touchpoint moment between 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. For instance, in the illustration, the last touchpoint moment between 0ms and 16.67ms, at 13ms, is retained, and the last touchpoint moment between 16.67ms and 33.33ms, at 23ms, is retained. Other touchpoint moments are deleted.

[0198] This application embodiment does not limit which touch moment to retain in each cycle. The purpose of this step is to retain one touch moment within the cycle (the filtering cycle converted from the screen refresh rate).

[0199] Of course, if the video file's frame rate is 120 FPS, then the timestamps of the video frames are 0ms, 8.33ms, 16.67ms, 25ms, 33.33ms, and so on. Since the screen refresh rate is 60Hz, the filtering period is still calculated using the screen refresh rate conversion cycle from 0ms to 16.67ms as one filtering period, from 16.67ms to 33.33ms as another filtering period, and so on.

[0200] Reference Figure 6 (b) in this embodiment represents another filtering method provided by this application, in which the two endpoints of the period and the timestamp of the video frame may not be aligned. Taking a filtering period of 16.67ms as an example, the period from 5ms to 21.67ms is one period, the period from 21.67ms to 38.33ms is another period, and so on.

[0201] Within the period from 5ms to 21.67ms, any one contact moment can be retained while other contact moments are filtered out. For example, the first or last contact moment can be retained while other contact moments are filtered out.

[0202] Of course, continuing to take the example of retaining the last contact moment, the retained contact moments are 4ms, 20ms and 35ms respectively.

[0203] Reference Figure 6 (c) in this application provides another filtering method. The previously determined contact time (or the first contact time) of 4ms is taken as the starting point of this filtering cycle, and then extended by 16.67ms to 20.67ms. The last contact time (20ms) in the filtering cycle from 4ms to 20.67ms is retained, and other contact times between 4ms and 20ms are deleted. Continuing, the previously determined contact time of 20ms is taken as the starting point of this filtering cycle, and then extended by 16.67ms to 36.67ms. The last contact time (35ms) between 20ms and 36.67ms is retained, and other contact times between 20ms and 36.67ms are deleted.

[0204] Although the touch point times determined by the three methods may be different, a video frame is retained within 16.67ms. Even if different touch point times are mapped to different video frames, the difference in display is only between the preceding and following frames. The frame playback process itself is the process of displaying video frames related to the drag position as the user drags their finger. The difference between the preceding and following frames will not cause a visual difference for the user.

[0205] Of course, if there is no contact moment or only one contact moment in a certain period, the above filtering process is not required, and the contact moment is directly retained.

[0206] In this embodiment, t1 and t2 are within the same cycle, and the contact time t2 is retained while the contact time t1 is filtered out.

[0207] After filtering out redundant contact times, the contact times can be stored in the seek data table.

[0208] S210, map the touch point time t2 within the current filtering period T to the video frame with the most recent timestamp and touch point time t2.

[0209] In this application, the timestamps of two adjacent video frames at the touch point time t2 may be calculated with t2 respectively, and the adjacent video frame with the smallest difference may be used as the mapped video frame.

[0210] Reference Figure 7 As shown, the touch point time t2 is the 13ms. The timestamps of the two video frames adjacent to the 13ms are the 0ms and 16.67ms. The difference between the 0ms and 13ms (13ms) and the difference between the 16.67ms and 13ms (3.67ms) are calculated respectively. Therefore, the touch point time t2 (the 13ms) is mapped to the video frame with the timestamp of the 16.67ms.

[0211] Of course, in practical applications, if two consecutive touch points are mapped to the same video frame, one of the touch points in the seek data table can be deleted.

[0212] S211, determine whether the mapped video frame meets the condition for adjustment to an I-frame. If it does, adjust the mapped video frame to an I-frame.

[0213] In this step, when the current video frame and the adjacent video frame belong to different GOP groups, it usually occurs in scenarios where the user drags relatively quickly. It is necessary to decode the video frames in the GOP group where the current video frame belongs and the video frames in the GOP group where the adjacent video frame belongs. Both of the video frames that are rendered and displayed consecutively need to be decoded in their respective GOP groups. Especially when the decoding order of the video frame in the GOP group is later, the decoding speed may be slower, and it may even lead to frame loss.

[0214] To quickly respond to user dragging operations, the current video frame can be adjusted to an I-frame between adjacent video frames. This means that when decoding this video frame, the first video frame in the GOP group containing it is the mapped video frame, resulting in faster decoding speeds.

[0215] Reference Figure 8As shown in (a), during the user's forward swipe, i.e., when the user's finger swipes from right to left, the touch point time corresponding to the progress indicator line becomes larger and larger. The timestamp of the video frame mapped to the touch point time becomes larger and larger. During the forward swipe, the current mapped video frame and the next mapped video frame are usually regarded as two adjacent mapped video frames.

[0216] If the touch point of the current video frame is 33.3ms and the next neighboring video frame is 216.6ms, the GOPs at 33.3ms and 216.6ms are different GOPs, and there is an I-frame between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the middle I-frame (116.6ms).

[0217] In this way, when decoding the video frame corresponding to the touch point t2 (adjusted from 33.3ms to 116.6ms), it is only necessary to decode one video frame in the GOP containing 116.6ms to achieve successful decoding; compared to decoding the video frame in 33.3ms, which requires decoding the fourth video frame to achieve successful decoding, the decoding speed is faster.

[0218] In current practical applications, when the touch point of the current video frame is 33.3ms and the next neighboring video frame is 266.6ms, the GOPs at 33.3ms and 266.6ms are different GOPs, and there are two I-frames between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the I-frame (116.6ms) that is closest to the current video frame (33.3ms) in the middle I-frames.

[0219] Therefore, during forward sliding, regardless of how many I-frames exist between the current video frame and the next adjacent frame, the current video frame will be replaced with the nearest I-frame.

[0220] Reference Figure 8 As shown in (b), during the user's reverse swipe, i.e., when the user's finger slides from left to right, the touch point time corresponding to the progress indicator line becomes smaller and smaller. The timestamp of the video frame mapped to the touch point time becomes smaller and smaller. During the reverse swipe, the current mapped video frame and the previous mapped video frame are usually regarded as two adjacent mapped video frames.

[0221] If the touch point of the current video frame is 33.3ms and the previous neighboring video frame is 216.6ms, the GOP at 33.3ms and the GOP at 216.6ms are different GOPs, and there is an I-frame between the two video frames, then the current video frame (33.3ms) can be adjusted to the middle I-frame (116.6ms).

[0222] In this way, when decoding the video frame corresponding to the touch point t2 (adjusted from 33.3ms to 116.6ms), it is only necessary to decode one video frame in the GOP containing 116.6ms to achieve successful decoding; compared to decoding the video frame in 33.3ms, which requires decoding the fourth video frame to achieve successful decoding, the decoding speed is faster.

[0223] In current practical applications, when the touch point of the current video frame is 33.3ms and the previous neighboring video frame is 266.6ms, the GOP at 33.3ms and the GOP at 266.6ms are different GOPs, and there are two I-frames between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the I-frame (116.6ms) that is closest to the current video frame (33.3ms) in the middle I-frame.

[0224] Therefore, during reverse scrolling, regardless of how many I-frames exist between the current video frame and the previous adjacent frame, the current video frame is replaced with the nearest I-frame.

[0225] S212, the seek management module sends a query command for mapped video frames to the decoder manager.

[0226] In this embodiment, if the target I-frame is not reached, the most recent mapped video frame is taken as the determined video frame. If the target I-frame is reached, the reached I-frame is taken as the mapped video frame (which can also be understood as a candidate video frame in the seek data table), and this mapped video frame is denoted as video frame b. Of course, the mapped video frames in the seek data table can also be denoted as candidate video frames.

[0227] S213, the decoder manager sends a query command for video frame b to the decoder buffer.

[0228] S214, if the decoder finds cached video frame b, then execute S215; if it does not cache video frame a, then execute S219.

[0229] S215, the decoder buffer sends the rendering instruction for video frame b to the renderer.

[0230] Of course, in practical applications, the decoding buffer can also send the rendering instruction for video frame a to the renderer through the display manager. When sending the rendering instruction for video frame b to the renderer through the display manager, on the one hand, the rendering status of the video frame in the seek data table can be updated to pending rendering, and on the other hand, the identifier of video frame b can be sent to the pending rendering queue maintained by the display manager. This application embodiment does not limit the specific implementation method.

[0231] S216, the renderer retrieves video frame b from the cache and renders video frame b.

[0232] In this embodiment of the application, the cached video frame b may come from various sources.

[0233] For example, after the playback engine is started, during the normal playback mode, the YUV data of video frame b decoded during normal playback can be cached. For instance, from second 0 to second 5, the decoded data from second 0 to second 5 can be cached. When the user drags their finger over the video frames between second 0 and second 5, the corresponding video frames can be retrieved from the cache.

[0234] For example, during frame playback, if the mapped video frame is any video frame in a GOP, when decoding a specific video frame, each video frame in that GOP can be decoded. These decoded video frames can be cached. When scrolling backwards later, the mapped video frame may also be in that GOP, and it can be directly retrieved from the cache.

[0235] Of course, during the idle time of electronic devices, a portion of video frames from video files that users are highly interested in can be extracted and decoded and cached.

[0236] This application does not limit the source of the cached video frames in its embodiments.

[0237] S217, after determining the cached video frame b, the decoding buffer can also send an update to the seek management module to show that the rendering status of video frame b has been rendered.

[0238] S218, after the seek manager receives the information that video frame b has been rendered, it updates the rendering status of video frame b to rendered and updates the rendering status of each video frame previously sought to abandoned rendering.

[0239] As mentioned earlier, the decoding buffer executes S219 if it determines that no video frame b is cached.

[0240] Reference Figure 9 S219, the decoder buffer sends information about the unbuffered video frame b to the decoder manager.

[0241] S220, after receiving the information of the unbuffered video frame b, the decoding manager sends the decoding instruction of video frame a to the decoder.

[0242] S221, after receiving information about the unbuffered video frame b, the decoding manager sends an instruction to the seek manager to update the rendering status of the video frame to be rendered.

[0243] S222, the seek manager changes the rendering status of video frame b in the seek data table to pending rendering.

[0244] S223, after the seek manager updates the status of video frame b to pending rendering, it sends the identifier of video frame b to the pending rendering queue maintained by the display manager.

[0245] Before executing S223, there may or may not be video frames in the queue to be rendered. In this embodiment, "*" is used instead.

[0246] After S223, the rendering queue maintained by the display manager includes the identifier (*) of the original video frame and the identifier of video frame b.

[0247] After S220, the decoder continues to execute S224.

[0248] S224, the decoder obtains the parsing data of the video frame group where video frame b is located from the buffer data corresponding to the parser.

[0249] S225, the decoder decodes according to the decoding order.

[0250] S226, during the decoding process, sends information about successfully decoded video frames to the seek manager.

[0251] S227, the seek manager updates the decoding status of video frame b in the seek data table to "decoded". It can also add information about other video frames in the video frame group where video frame b is located, and update the decoding status of other decoded video frames to "decoded".

[0252] In this embodiment, the decoded data can also be cached. When subsequently mapped to a cached video frame, it does not need to be decoded again.

[0253] S228, the display manager determines the video frame Z to be rendered from the queue according to the display strategy (video frame z may be any video frame in the current queue, for example, it may be the video frame in * or it may be video frame b).

[0254] S229, The display manager sends a rendering instruction for video frame z to the renderer.

[0255] S230, the renderer renders video frame z.

[0256] Reference Figure 10 This refers to the display strategy of the display manager provided in the embodiments of this application, namely the execution flow of S228 in the above embodiments.

[0257] The following rules were set for the display strategy:

[0258] After any video frame (e.g., video frame a) is rendered or rendering of any video frame is abandoned, the timer restarts from 0ms to measure whether rendering is replaced, abandoned, or rendered for display.

[0259] Furthermore, since the process of sending video frame identifiers to the rendering queue is not periodic, and each time a video frame identifier is retrieved from the rendering queue for display, the identifiers of the displayed video frame and the previous video frames in the rendering queue are deleted. Therefore, in practical applications, at any given moment, the rendering queue may contain no video frame identifiers, may contain one video frame identifier, or may contain multiple video frame identifiers.

[0260] When the timer restarts, if there is no video frame in the rendering queue, it can continue to wait. If no new video frame is sent to the rendering queue after the third time interval, the timer can restart. Of course, if a new video frame has been sent to the rendering queue before the third time interval, the display strategy can continue to be executed within the current time interval, or the timer can be restarted and the display strategy executed within the restarted time interval.

[0261] As another example, if the queue for rendering has not received a new video frame by the third time interval, the timer can continue until a new video frame is received in the queue for rendering, at which point the timer restarts and the display strategy is executed within the restarted timer period.

[0262] The embodiments of this application do not limit the above content.

[0263] Typically, the number of identifiers for video frames in the rendering queue is greater than 0, provided that the user can experience dragging.

[0264] The following describes the case where the number of identifiers for video frames in the queue to be rendered is greater than 0.

[0265] A1: Render any video frame completely or abandon the rendering of any video frame.

[0266] A2 restarts the timer, usually starting from 0.

[0267] A3. First, check if there are any decoded video frames to be rendered in the queue.

[0268] A4, if it exists, will be the last video frame in the decoded video frames to be rendered that was sent to the rendering queue, and will be used as the video frame to be rendered (or the target video frame); video frames in the seek data table can be called candidate video frames. Candidate video frames can be sent to the rendering queue, but this does not mean that every candidate video frame can be rendered and displayed as the target video frame.

[0269] After determining the video frame to be displayed, the rendering status of the video frame in the seek data table is updated to rendered. The rendering status of the candidate video frames that were previously in the rendering state (measured by the system clock corresponding to the touch point of the video frame) is updated to abandoned rendering. The current video frame is removed from the rendering queue, and the rendering instruction for the current video frame is sent to the renderer, which can then render the video frame.

[0270] Additionally, after the second target video frame is successfully rendered, the timer restarts, essentially entering the next loop. A5, if it does not exist, monitors the decoding status of the first candidate video frame in the rendering queue that is awaiting decoding and rendering.

[0271] A6. If the decoding status of the first video frame is updated to "decoded" within the second time period, then that video frame is taken as the target video frame.

[0272] Update the rendering status of the currently determined target video frame to "rendered", and update the rendering status of the video frames in the seek data table that were previously in a pending state to "abandon rendering".

[0273] Remove the current target video frame from the rendering queue and render the currently determined target video frame; after the target video frame is successfully rendered, restart the timer, which is equivalent to entering the next loop.

[0274] A7. If the decoding status of the first video frame is not updated to "decoded" within the second time period, is the video frame and the next video frame in the rendering queue in the same GOP group?

[0275] If A8 does not belong to the same GOP group (or there is no next video frame in the queue to be rendered), then continue to monitor the first video frame within the current timing period and determine whether the decoding of the first video frame has been updated to "decoded" within the third time period.

[0276] A9, if the first video frame is updated to be decoded within the third time period, then the first video frame is determined as the target video frame.

[0277] Update the rendering status of the currently determined target video frame to "rendered", and update the rendering status of the video frames in the seek data table that were previously in a pending state to "abandon rendering".

[0278] Remove the current target video frame from the rendering queue and render the currently determined target video frame; after the target video frame is successfully rendered, restart the timer, which is equivalent to entering the next loop.

[0279] If A10 fails to detect the first video frame being updated to be decoded within the third time period, then the rendering of that first video frame will be abandoned.

[0280] Update the rendering status of the currently determined target video frame in the seek data table to "abandon rendering", and update the rendering status of the video frames in the previous pending state in the seek data table to "abandon rendering"; remove the current target video frame from the pending rendering queue and restart the timer, which is equivalent to entering the next loop.

[0281] A11. If the video frame and the next video frame in the rendering queue are in the same GOP group, then the next video frame replaces the first video frame, and the decoding status of the next video frame continues to be monitored within the current timing period.

[0282] A12, if the next video frame to be replaced has been successfully decoded within the third time period, then the next video frame to be replaced is determined as the target video frame.

[0283] Update the rendering status of the currently determined target video frame to "rendered", and update the rendering status of the video frames in the seek data table that were previously in a pending state to "abandon rendering".

[0284] Remove the current target video frame from the rendering queue and render the currently determined target video frame; after the target video frame is successfully rendered, restart the timer, which is equivalent to entering the next loop.

[0285] In this step, the third duration starts from the restart time corresponding to the first video frame that is replaced. That is, the third duration does not restart the timing after the second duration is replaced with the next video frame, but continues the timing after the second duration is replaced with the next video frame. Therefore, the starting time of the third duration and the second duration are the same.

[0286] A13. If the next video frame to be replaced within the third duration has not been successfully decoded, then discard the video frame in the rendering queue that is consecutive to the first (or next) video frame and in the same GOP group.

[0287] The following examples illustrate this. Figure 10This section outlines display strategies for various scenarios. In the examples below, 75ms is used as an example of the second duration, and 150ms as an example of the third duration. In practical applications, other durations can be set as the second and third durations, for example, by setting the second and third durations to be shorter. Alternatively, the first duration (or the second or third duration) can be used to determine whether to render or abandon the display.

[0288] Reference Figure 11 As shown, after restarting the timer, the state of each video frame in the rendering queue is retrieved; the state of these video frames is referenced... Figure 11 The seek data table in the context of the current rendering queue contains multiple decoded video frames awaiting rendering. The last video frame (video frame g) added to the rendering queue can be rendered. After rendering, the timer restarts. This process can be described as... Figure 10 A4 in the middle.

[0289] If, when the timing restarts, video frame h is still pending decoding and rendering, the decoding status of video frame h is monitored. If video frame h has been updated to be decoded and ready for rendering (successfully decoded during the rendering of video frame g), but video frame i has not been updated to be pending decoding, then video frame h in the rendering queue is rendered directly.

[0290] Other possible scenarios will not be exemplified.

[0291] Reference Figure 12 As shown in (a), after video frame a is rendered, the timer restarts to monitor the decoding status of video frame b to be decoded and rendered. Before 75ms, video frame b has been decoded; therefore, video frame b can be rendered. After video frame b is rendered, the timer restarts to continue monitoring the decoding status of video frame c to be decoded and rendered. This process can correspond to... Figure 10 A6 in the middle.

[0292] Reference Figure 12 As shown in (b), after video frame a is rendered, the timing restarts to monitor the decoding status of video frame b to be decoded and rendered. If video frame b is not decoded before 75ms, the timing continues to monitor the next video frame c in the same GOP group as video frame b, i.e., video frame c replaces video frame b for rendering and display. Before 150ms, if video frame c is detected as successfully decoded, it can be rendered. After video frame c is rendered, the timing restarts. This process can correspond to... Figure 10 A12 in the middle.

[0293] Reference Figure 12As shown in (c), after video frame a is rendered, timing restarts to monitor the decoding status of video frame b to be decoded and rendered. If video frame b is not decoded before 75ms, the timing cycle continues to monitor the next video frame c, which belongs to the same GOP group as video frame b, and video frame c replaces video frame e for rendering and display. If video frame c is not successfully decoded before 150ms, rendering of video frame c, as well as b and d in the same video frame, is abandoned. After abandoning rendering, timing restarts, and the decoding status of video frame e is monitored in the next timing cycle. This process can be corresponding to... Figure 10 A13 in the middle.

[0294] Of course, in practical applications, if video frame j and video frame c belong to the same GOP group, video frame j may be the video frame mapped to the touch point position corresponding to the touch point time generated by the user dragging in the opposite direction. In this case, there is no need to abandon the display of video frame j. Therefore, the video frame to be abandoned is the first video frame in the rendering queue that does not belong to the same GOP group.

[0295] according to Figure 12 As shown in (d), after video frame a is rendered, timing restarts to monitor the decoding status of video frame b to be decoded and rendered. If video frame b is not decoded before 75ms, and the next video frame c does not belong to the same GOP group as video frame b, it cannot be displayed as a replacement. Therefore, monitoring of video frame b continues within the current timing cycle. If video frame b is detected as undecoded before 150ms, rendering of video frame b is abandoned. After abandoning rendering, timing restarts, and the decoding status of video frame c is monitored in the next timing cycle. This process can be corresponding to... Figure 10 A10 in the middle.

[0296] Of course, if video frame b is successfully decoded before 150ms, then video frame b is rendered and displayed. After successfully rendering video frame b, the timing restarts, and the decoding status of video frame c is monitored in the next timing cycle. This process can correspond to... Figure 10 A9 in the middle.

[0297] like Figure 9 As shown, when S214 does not cache video frames, the display manager performs rendering display, replacement rendering display, and abandon rendering display on the video frames in the rendering queue according to the display policy.

[0298] like Figure 5 As shown, with video frames cached in S214, rendering commands are sent directly to the renderer to execute rendering and display.

[0299] For a renderer, it may receive rendering commands from two branches. To manage these commands uniformly, a method can be adopted. Figure 5 Method two.

[0300] S215, the decoding buffer sends the decoding status of video frame b to the seek manager as decoded and the rendering status as pending.

[0301] S216, the seek manager updates the decoding status of video frame b in the seek data table to "decoded" and the rendering status to "to be rendered".

[0302] S217, the seek manager sends the identifier of video frame b to the display manager, and video frame b is sent to the rendering queue.

[0303] After S217, Figure 5 and Figure 9 The corresponding solutions are unified:

[0304] The queue to be rendered may contain video frame identifiers from the cache or from the decoder; the display manager performs replacement rendering, abandon rendering, and rendering display on the video frames in the queue to be rendered according to the display strategy in the example above.

[0305] In practical applications, the following situations may exist in the queue to be rendered.

[0306] Reference Figure 13 As shown, the order in which the data is sent into the rendering queue is basically consistent with the order of the system time corresponding to the touch point.

[0307] For an unbuffered video frame e, after S214, it is necessary to execute S219 to return the unbuffered information, S221 to notify the seek manager, and S223 to send the seek manager into the rendering queue.

[0308] For the cached video frame f, after S214, S215 needs to be executed to notify the seek manager and S217 the seek manager to send it to the rendering queue.

[0309] S219 executes very quickly. Therefore, when video frame e in the seek data table is before video frame f (the system time corresponding to the touch point of these two video frames is mapped respectively), video frame e will also be before video frame f in the rendering queue. Under normal circumstances, video frame e will not appear after video frame f in the rendering queue.

[0310] In this case, the rendering queue will contain two consecutive video frames, e and f, with the first frame awaiting decoding and the second frame already decoded. It is still possible to proceed according to... Figure 10In step A4, the last decoded video frame (g) is rendered, and the previous video frames are updated to be discarded for rendering. Of course, to improve the decoding speed of subsequent video frames, previous video frames to be decoded can also be updated to be discarded for decoding.

[0311] This application provides a frame playback method, including:

[0312] The touch point time is obtained, which is the playback time of the video file mapped from the touch point position detected during frame playback;

[0313] The touch point time is mapped to candidate video frames, wherein each candidate video frame has: a first timestamp, a decoding state, and a rendering state, and the first timestamp is the system detection time of the touch point position corresponding to the candidate video frame;

[0314] Update the rendering status of the candidate video frame to be rendered, and send the identifier of the candidate video frame updated to be rendered to the queue to be rendered.

[0315] The candidate video frame is decoded, and after the candidate video frame is successfully decoded, the decoding status of the candidate video frame is updated to "decoded".

[0316] According to the rendering and display strategy, candidate video frames in the queue to be rendered are either abandoned for rendering or rendered. The rendering and display strategy includes: if the decoding status of the first candidate video frame in the waiting state is not updated to "decoded" within a first time period, the first candidate video frame is abandoned for rendering; if the decoding status of the first candidate video frame in the waiting state is updated to "decoded" within the first time period, the first candidate video frame is rendered.

[0317] This application uses a gallery application to implement video playback using a video playback engine as an example to describe the switching process between normal playback and frame playback. In practical applications, it can also be applied to other applications, such as video editing applications and video playback applications, to achieve the same function using the playback engine.

[0318] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0319] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0320] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0321] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0322] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0323] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0324] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0325] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A frame playback method, characterized in that, include: The touch point time is obtained, which is the playback time of the video file mapped from the touch point position detected during frame playback; The touch point time is mapped to candidate video frames, wherein each candidate video frame has: a first timestamp, a decoding state, and a rendering state, and the first timestamp is the system detection time of the touch point position corresponding to the candidate video frame; Update the rendering status of the candidate video frame to be rendered, and send the identifier of the candidate video frame updated to be rendered to the queue to be rendered. The candidate video frame is decoded, and after the candidate video frame is successfully decoded, the decoding status of the candidate video frame is updated to "decoded". According to the rendering and display strategy, candidate video frames in the queue to be rendered are either abandoned for rendering or rendered. The rendering and display strategy includes: if the decoding status of the first candidate video frame in the waiting state is not updated to "decoded" within a first time period, the first candidate video frame is abandoned for rendering; if the decoding status of the first candidate video frame in the waiting state is updated to "decoded" within the first time period, the first candidate video frame is rendered.

2. The method as described in claim 1, characterized in that, The step of discarding or rendering candidate video frames in the rendering queue according to the rendering display strategy includes: After the first target video frame is successfully rendered, the timer restarts. The first target video frame is the candidate video frame determined for rendering and display in the queue to be rendered. After the timing restarts, the system checks the queue of candidate video frames to be rendered to see if there are any that have been decoded and are ready to be rendered. If there are decoded and unrendered candidate video frames, the last candidate video frame sent to the rendering queue among the decoded and unrendered candidate video frames will be used as the second target video frame. Update the rendering status of the second target video frame to rendered, and update the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the second target video frame to abandoned rendering. Render the second target video frame; The timing restarts after the second target video frame is successfully rendered.

3. The method as described in claim 2, characterized in that, After checking the queue of candidate video frames to be rendered to see if there are any decoded and ready-to-be-rendered frames, the method further includes: If there are no decoded and rendered candidate video frames in the queue to be rendered, then the decoding status of the second candidate video frame in the queue to be rendered is monitored. The second candidate video frame is the first candidate video frame in the queue to be decoded and rendered. If the decoding status of the second candidate video frame is updated to "decoded" within the second time period, then the second candidate video frame is used as the third target video frame. The rendering status of the third target video frame is updated to "rendered", and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the third target video frame is updated to "abandon rendering". Render the third target video frame; The timer restarts after the third target video frame is successfully rendered.

4. The method as described in claim 3, characterized in that, The process of monitoring the decoding status of the second candidate video frame in the queue to be rendered also includes: If the decoding status of the second candidate video frame is not updated to decoded within the second time period, it is determined whether the second candidate video frame and the third candidate video frame in the queue to be rendered belong to the same video frame group. The third candidate video frame is the candidate video frame in the queue to be rendered that is adjacent to and follows the second candidate video frame. If the second candidate video frame and the third candidate video frame belong to the same video frame group, then the third candidate video frame replaces the second candidate video frame, and the decoding status of the third candidate video frame continues to be monitored within the current timing period. If the decoding status of the third candidate video frame is updated to "decoded" within the third time period, then the third candidate video frame is used as the fourth target video frame. The starting point of the third time period is the same as the starting point of the second time period. The rendering status of the fourth target video frame is updated to "rendered", and the rendering status of the candidate video frames whose unrendered state is before the first timestamp of the fourth target video frame is updated to "abandon rendering". Render the fourth target video frame; The timing restarts after the fourth target video frame is successfully rendered.

5. The method as described in claim 4, characterized in that, During the process of monitoring the decoding status of the third candidate video frame, the method further includes: If the decoding status of the third candidate video frame is not updated to decoded within the third time period, the abandonment flag of the candidate video frame from the second candidate video frame to the fourth candidate video frame in the queue to be rendered is obtained. The fourth candidate video frame is the first candidate video frame in the queue to be rendered that is located after the second candidate video frame and is not in the same video frame group as the second candidate video frame. The decoding status of the candidate video frame corresponding to the abandonment flag is updated to abandon decoding, and the rendering status is updated to abandon rendering. The rendering status of the candidate video frame whose first timestamp is before the first timestamp of the second candidate video frame is updated to abandon rendering. After updating to abandon rendering, the timer restarts.

6. The method as described in claim 4 or 5, characterized in that, After determining whether the second candidate video frame and the third candidate video frame in the queue to be rendered belong to the same video frame group, the process further includes: If the second candidate video frame and the third candidate video frame do not belong to the same video frame group, the decoding status of the second candidate video frame will continue to be monitored within the current timing period. If the decoding status of the second candidate video frame is updated to "decoded" within the third time period, then the second candidate video frame is used as the fifth target video frame. The rendering status of the fifth target video frame is updated to "rendered", and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the fifth target video frame is updated to "abandon rendering". Render the fifth target video frame; The timer restarts after the fifth target video frame is successfully rendered.

7. The method as described in claim 6, characterized in that, The process of continuing to monitor the decoding status of the second candidate video frame within the current timing period also includes: If the decoding status of the second candidate video frame is not updated to decoded within the third time period, the rendering status of the second candidate video frame is updated to abandon rendering, and the rendering status of the candidate video frames whose first timestamp is before the first timestamp of the second candidate video frame is updated to abandon rendering. After updating to abandon rendering, the timer restarts.

8. The method according to any one of claims 1 to 7, characterized in that, Before mapping the touch point moments to candidate video frames, the method further includes: Get the screen refresh rate; The first moment is taken as the starting point of the first filtering cycle, and the duration of the first filtering cycle is equal to the duration of the screen refresh cycle. If any contact moment falls within the time range of the first filtering cycle, then that contact moment is retained; If at least two contact moments fall within the time range of the first filtering cycle, one contact moment will be retained from the multiple contact moments within the time range of the first filtering cycle, and the other contact moments other than the retained contact moment will be deleted.

9. The method as described in claim 8, characterized in that, The first moment is the end of the previous filtering cycle or the contact moment determined in the previous filtering cycle.

10. The method according to any one of claims 1 to 9, characterized in that, The video file includes multiple video frames, and each video frame in the video file carries a second timestamp, which represents the playback time of the video frame in the video file. The step of mapping the touch point time to candidate video frames includes: The video frame with the smallest difference between the second timestamp and the touch point time in the video file is selected as the candidate video frame for the touch point time mapping.

11. The method as described in claim 10, characterized in that, During the forward sliding process, the second timestamps of the candidate video frames mapped by two adjacent touch points are increased from small to large; when the video frame mapped by one of the touch points is the fifth candidate video frame, the next candidate video frame adjacent to the fifth candidate video frame is the sixth candidate video frame, and the second timestamp of the sixth candidate video frame is greater than the second timestamp of the fifth candidate video frame. If the fifth candidate video frame and the sixth candidate video frame are located in different video frame groups and there is a key frame between the fifth candidate video frame and the sixth candidate video frame, then the fifth candidate video frame is adjusted to be the key frame with the smallest difference between the second timestamp of the fifth candidate video frame and the second timestamp of the fifth candidate video frame among the key frames between the fifth candidate video frame and the sixth candidate video frame.

12. The method as described in claim 10, characterized in that, During the reverse sliding process, the second timestamps of the candidate video frames mapped by two adjacent touch points decrease from large to small; when the video frame mapped by one of the touch points is the seventh candidate video frame, the preceding candidate video frame adjacent to the seventh candidate video frame is the eighth candidate video frame, and the second timestamp of the eighth candidate video frame is greater than the second timestamp of the seventh candidate video frame. If the seventh candidate video frame and the eighth candidate video frame are located in different video frame groups and there is a key frame between the seventh candidate video frame and the eighth candidate video frame, then the seventh candidate video frame is adjusted to be the key frame with the smallest difference between the second timestamp of the seventh candidate video frame and the second timestamp of the seventh candidate video frame among the key frames between the seventh candidate video frame and the eighth candidate video frame.

13. An electronic device, characterized in that, The device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when executed by the one or more processors, causes the electronic device to perform the method as described in any one of claims 1-12.

14. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-12.

16. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-12.

Images