A frame playing method, an electronic device, a chip system and a storage medium

Abstract

The application provides a frame playing method, an electronic device, a chip system and a storage medium, and relates to the technical field of electronic devices. In the frame playing or ordinary playing process, a part of successfully decoded video frames can be selected for caching. For example, a part of video frames located at the tail of a video frame group being currently decoded can be cached, and the part of video frames can be released when decoding video frames in the next video frame group. In addition, video frames adjacent to the touch moment when the user drags can be cached, and the part of video frames is released following the principle of first in first out. In this way, memory resource shortage caused by caching too many video frames can be avoided. In addition, when reverse sliding, less memory resource can be used to try to obtain decoded video frames from the cached video frames, so that the decoding waiting time of the video frames mapped by the touch moment is reduced, and the picture fluency during frame playing is improved.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a frame playback method, electronic equipment, chip system, and storage medium. Background Technology

[0002] Gallery apps on electronic devices can provide image preview interfaces, allowing users to view various images stored on the device. Gallery apps also provide video playback interfaces, allowing users to play various videos stored on the device.

[0003] The video playback interface can also display the video playback progress, for example, through a progress bar, progress ring, or progress display frame. User drag gestures (press, drag, and release) on the progress bar, progress ring, or progress display frame can trigger the electronic device to play video frames in frame playback mode. However, currently, stuttering often occurs during frame playback due to user drag gestures. Summary of the Invention

[0004] This application provides a frame playback method, electronic device, chip system, and storage medium, which can reduce stuttering during frame playback.

[0005] To achieve the above objectives, the first aspect of this application provides a frame playback method, employing the following technical solution:

[0006] The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file;

[0007] Decode each video frame in the first frame group containing the first video frame in the decoding order;

[0008] During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and the decoding data of the display video frame is cached; some video frames in the first frame group are marked as non-display video frames, and the decoding data of the non-display video frames is cached.

[0009] The second touch point moment is mapped to the second video frame. The second touch point moment and the first touch point moment are different touch point moments in the same sliding operation. The detection time of the second touch point moment is later than the detection time of the first touch point moment. The cached decoding data includes the decoding data of the second video frame or includes decoding data whose timestamp difference from the second video frame is less than a threshold.

[0010] The decoded data of the third video frame is obtained from the cache, and the decoded data of the third video frame is rendered and displayed. The third video frame is either the second video frame or a video frame whose timestamp difference with the second video frame is less than a threshold.

[0011] In this application, the frame playback process is random, that is, the video frame to be displayed is related to the position of the user's finger touch. Even if the touch time is determined based on the position of the user's finger touch, and the touch time is mapped to the video frame, the video frame still needs to go through the decoding and rendering process before it is displayed on the screen. The decoding process doesn't involve immediately decoding any video frame specified for decoding. Video frames are decoded in groups, with each frame in its own decoding order. Furthermore, the decoding order isn't based on timestamps. Therefore, when a specific video frame (the one mapped to the touch point) needs to be displayed, its decoding order might be the last frame decoded in the group (or later in the decoding order). This could result in the frame not being successfully decoded after a certain time. To avoid screen stuttering caused by long decoding times delaying subsequent rendering, a portion of the video frame's decoding data can be cached during frame playback. In subsequent frame playback, if the decoding data for the required video frame has already been cached (or if there are timestamp differences less than a threshold), the decoding data can be directly retrieved from the cache for rendering, thus avoiding screen stuttering caused by waiting for decoding.

[0012] In addition, in practical applications, since the video frames that need to be displayed are usually successfully decoded, the video frames that need to be displayed can be marked as display video frames, and the decoded data of the display video frames can be cached. In addition, since the decoding is done in groups, other video frames in the same frame group as the video frame that needs to be displayed may also be decoded when decoding the video frame that needs to be displayed. Therefore, a portion of the video frames in the same video frame group can also be selected for caching.

[0013] As one implementation of the first aspect, after marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes:

[0014] If the first video frame is successfully rendered and displayed, the decoded data of the first video frame in the first cache queue is marked as invalid. The first cache queue is used to cache the decoded data of the displayed video frame.

[0015] If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame;

[0016] If the first video frame marked as invalid is not marked as a non-display video frame, then the decoded data of the first video frame in the first buffer queue is released.

[0017] In this application, if a video frame marked as a display video frame has been successfully rendered and displayed, it means that the video frame was successfully rendered this time and can be in an invalid state; a video frame in an invalid state is also marked as a non-display video frame and needs to be cached as a non-display video frame. When caching a non-display video frame, the decoded data of the video frame is cached in the cache queue of non-display video frames.

[0018] Since the decoding data of video frames is image data, which consumes a lot of memory space, in order to avoid consuming memory resources, this application embodiment needs to release the currently consumed display video frames (which are not marked as non-display video frames) in order to achieve caching of video frames under the limitation of limited memory resources.

[0019] As one implementation of the first aspect, after marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes:

[0020] If the first video frame is not rendered and displayed, the cache duration of the decoded data of the first video frame in the first cache queue is calculated.

[0021] If the buffer duration of the decoded data of the first video frame is longer than the first duration, the decoded data of the first video frame is marked as invalid.

[0022] If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame;

[0023] If the first video frame marked as invalid is not marked as a non-display video frame, then the first video frame is released from the first buffer queue.

[0024] In this application, even if the first video frame is determined to be the display video frame, in practical applications, the detection frequency of user touch points is relatively high, usually higher than the rendering frame rate. In this case, the renderer typically selects the most recently sent decoded video frame for rendering. Therefore, even if the video frame mapped to certain touch points is determined to be the video frame to be displayed, it may still fail to be rendered successfully. In this situation, such video frames can be cached for a period of time to avoid needing to display the video frame again when swiping in the opposite direction. If the video frame needs to be displayed again, it can be rendered directly without decoding, using the decoded data of the video frame. Of course, if the cache duration of such video frames exceeds a certain period, in order to save memory resources, such video frames can be marked as invalid and processed according to the invalid state handling strategy.

[0025] As one implementation of the first aspect, marking a portion of the video frames in the first frame group as non-display video frames and caching the decoded data of the non-display video frames includes:

[0026] Select a portion of video frames from the end of the first frame group and mark them as preset cached video frames. Cache the preset cached video frames from the first frame group in the second cache queue. The non-display video frames include the preset cached video frames.

[0027] A portion of video frames adjacent to the first video frame in the first frame group are selected and marked as temporary cache video frames. The temporary cache video frames in the first frame group are cached in the third cache queue. The non-display video frames include the temporary cache video frames.

[0028] In this application, the non-display video frames are selected into two main categories. One category consists of video frames located at the end of a group of video frames (GOPs). Video frames at the end of a GOP are typically decoded later, resulting in longer decoding times. These frames are most prone to screen stuttering due to the waiting time for decoding. Therefore, these video frames are cached. The other category consists of video frames adjacent to the currently determined display video frame. During user swiping, when the user switches from a forward swipe to a reverse swipe, the video frames most likely to be mapped by the touch point are those near the current display video frame. Therefore, to reduce screen stuttering in this scenario, video frames near the currently likely rendering display video frame (the first video frame) are used as temporary cached video frames.

[0029] As one implementation of the first aspect, before caching the preset cached video frames in the first frame group in the second cache queue, the method further includes:

[0030] Release the preset cached video frames in the video frame group before the second frame group from the second cache queue. The second frame group is a video frame group in the video file that is before the first frame group. The video frame group before the second frame group is the video frame group in the video file that is before the second frame group.

[0031] Before caching the temporary buffered video frames in the first frame group in the third buffer queue, the method further includes:

[0032] If there is a temporary cached video frame in the third cache queue with a timestamp greater than the timestamp of the first video frame, the temporary cached video frame with a timestamp greater than the timestamp of the first video frame will be released.

[0033] After releasing the temporary cached video frames whose timestamps are greater than the timestamp of the first video frame, the temporary cached video frames in the first frame group are cached in the third cache queue according to the first-in-first-out principle.

[0034] In this application, due to limited memory resources, the embodiments of this application are applied when switching from forward sliding to reverse sliding or during reverse sliding. In order to retain the preset cached video frames in the previous frame group and delete the preset cached video frames before that when caching the preset cached video frames of the current frame group, under the premise of a limited number of cached video frames, the temporary cached video frames with timestamps greater than the current first video frame are deleted. And when the length of the third cache queue is determined, a portion of the temporary cached video frames are released according to the first-in-first-out principle.

[0035] According to the above method, the maximum cache duration of the display video frames in the first cache queue is the first duration, the second cache queue retains a maximum of two preset cached video frames from the video frame groups, and the third cache queue retains a maximum of a certain length of temporary cached video frames. Thus, when memory resources are limited, the phenomenon of screen stuttering can be minimized when applied to forward sliding to reverse sliding scenarios or reverse sliding scenarios.

[0036] As one implementation of the first aspect, selecting a portion of video frames from the tail of the first frame group and marking them as preset buffered video frames includes:

[0037] Calculate the first number interval when caching the preset cached video frames in the first frame group, the first number interval being related to the frame rate and rendering frame rate of the video file;

[0038] Calculate the first quantity length when caching the preset cached video frames in the first frame group. The first quantity length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate.

[0039] Based on the first quantity interval and the first quantity length, the first cache number of the video frames marked as preset cache in the first frame group is obtained;

[0040] Starting from the end of the first frame group, a portion of video frames are selected and marked as preset cached video frames based on the first cache quantity and the first quantity interval.

[0041] In this application, when the frame rate of the video file is greater than the rendering frame rate, the probability of consecutive video frames in the video file being rendered and displayed is relatively small. Therefore, the frame rate of the video file can be set to be greater than the rendering frame rate, and the interval between frames can be larger.

[0042] When sliding in the opposite direction, you can also set a setting related to the sliding speed. In the reverse sliding scenario, the greater the sliding speed, the greater the interval of the preset buffered video frames at the end, which can make the span of the buffered video frames larger.

[0043] As one implementation of the first aspect, the first quantity interval for calculating the preset cached video frames in the first frame group includes:

[0044]

[0045] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0046] As one implementation of the first aspect, the first quantity length when calculating the preset cached video frames in the first frame group includes:

[0047]

[0048] Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps The number of video frames the decoder decodes per second.

[0049] As one implementation of the first aspect, the step of obtaining the first cache number of pre-buffered video frames in the first frame group based on the first quantity interval and the first quantity length includes:

[0050]

[0051] Among them, D PCF L represents the first quantity interval. PCF The first quantity length is indicated by the `round()` function, which is used to round floating-point numbers to integers.

[0052] As one implementation of the first aspect, the step of selecting a portion of video frames from adjacent positions of the first video frame in the first frame group and marking them as temporary buffer video frames includes:

[0053] Calculate a second quantity interval when caching temporary cached video frames in the first frame group, the second quantity interval being related to the frame rate and rendering frame rate of the video file;

[0054] During reverse sliding, the second number length of temporary cached video frames in the first frame group is calculated. The second number length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate. Based on the second number interval and the second number length, the second number of cached video frames marked as temporary cached in the first frame group is obtained. The third cache queue caches a maximum of the second number length of video frames. During forward sliding, the second number of cached frames is obtained.

[0055] Starting from the first video frame in the first frame group, based on the second cache quantity and the second quantity interval, a portion of video frames with timestamps less than the timestamp of the first video frame are selected and marked as the temporary cache video frames.

[0056] As one implementation of the first aspect, the second quantity interval for calculating the temporary cached video frames in the first frame group includes:

[0057]

[0058] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0059] The second number length of the temporarily cached video frames in the first frame group includes:

[0060]

[0061] Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps The number of video frames the decoder decodes per second.

[0062] The step of obtaining the second buffer quantity of video frames marked as temporary buffers in the first frame group based on the second quantity interval and the second quantity length includes:

[0063]

[0064] Among them, D TCF L represents the first quantity interval. TCF The first quantity length is indicated by the `round()` function, which is used to round floating-point numbers to integers.

[0065] As one implementation of the first aspect, the method further includes:

[0066] During the first frame playback after the video file is opened this time, and during subsequent playback, the steps of marking and displaying video frames for caching and marking and not displaying video frames for caching are executed.

[0067] In this application, the steps of marking displayed video frames for caching and marking non-displayed video frames for caching can be performed during frame playback after the first triggering of frame playback and during subsequent playback.

[0068] As an example, if a user presses and releases their finger in the progress display area, the system switches to normal playback. However, it still needs to determine the mapped video frame based on the detected touch point position when the finger was pressed and cache the decoding data of the relevant video frame.

[0069] As one implementation of the first aspect, the method further includes:

[0070] If the video file's playback interface is closed after detection, and the video file's cache queue contains the video file's video frame decoding data, then the video file's video frame decoding data is released.

[0071] After a second duration following the completion of the frame playback operation of the video file, if the video file's cache queue contains the decoded data of the video frame, then the decoded data of the video frame is released.

[0072] In this application, to avoid wasting memory resources, the decoded data of video frames can be released in certain scenarios.

[0073] As one implementation of the first aspect, after decoding each video frame in the first frame group according to the decoding order, the method further includes:

[0074] In a forward playback scenario, the timing of the next touch point is predicted based on the sliding speed of the sliding operation during frame playback;

[0075] Decode the video frames in the third frame group containing the video frame mapped at the next touch moment according to the decoding order;

[0076] During the decoding of video frames in the third frame group, a portion of video frames are selected from the end of the third frame group and marked as preset cached video frames. The decoding data of the preset cached video frames in the third frame group is cached in the second cache queue.

[0077] In this application, in order to improve the decoding speed in the forward playback scenario, the next touch point can also be predicted, thereby decoding video frames in advance, reducing the waiting time for decoding, and improving the smoothness of the picture when playing frames.

[0078] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0079] Thirdly, a chip system is provided, including a processor coupled to a memory, wherein the processor executes a computer program stored in the memory to cause an electronic device to implement the method of any one of the first aspects of this application.

[0080] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0081] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0082] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0083] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0084] Figure 2 A schematic diagram of an interface for a normal playback mode and a frame playback mode provided in an embodiment of this application;

[0085] Figure 3 A schematic diagram illustrating the relationship between the playback order and decoding order of video frames in a video frame group, provided for an embodiment of this application;

[0086] Figure 4 A schematic diagram illustrating a scenario of performing frame caching and using cached video frames, provided for an embodiment of this application;

[0087] Figure 5 A timing diagram of a frame playback process without buffering, provided for an embodiment of this application;

[0088] Figure 6 A schematic diagram illustrating the timing of the filtering contact points provided in an embodiment of this application;

[0089] Figure 7 This is a schematic diagram illustrating how a touch point moment is mapped to a video frame, as provided in an embodiment of this application.

[0090] Figure 8 This is a schematic diagram illustrating how the video frame mapped to the touch point moment is adjusted to a nearby I-frame, as provided in an embodiment of this application.

[0091] Figure 9 This is a schematic diagram illustrating the rendering or abandonment of video frames in a cache queue, as provided in an embodiment of this application.

[0092] Figure 10 A schematic diagram illustrating the marking and failure conditions of a display video frame provided in an embodiment of this application;

[0093] Figure 11 A schematic diagram illustrating the determination of a preset buffered video frame, provided as an embodiment of this application;

[0094] Figure 12 A schematic diagram illustrating another method for determining preset buffered video frames, provided in an embodiment of this application;

[0095] Figure 13 A schematic diagram illustrating the release of a preset buffered video frame, provided as an embodiment of this application;

[0096] Figure 14 A schematic diagram illustrating the determination of temporary buffered video frames provided in an embodiment of this application;

[0097] Figure 15 A schematic diagram illustrating another method for determining temporary buffered video frames, provided as an embodiment of this application;

[0098] Figure 16 A schematic diagram illustrating the release of a temporary buffered video frame, provided as an embodiment of this application;

[0099] Figure 17 This is a timing diagram of a frame playback process with buffering provided in an embodiment of this application. Detailed Implementation

[0100] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0101] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0102] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0103] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0104] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0105] This application provides a frame playback method that can be applied to electronic devices. These electronic devices can be tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not limit the specific type of electronic device.

[0106] Figure 1A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0107] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0108] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0109] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0110] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback). Touch sensor 180K, also called a "touch panel," can be disposed on display screen 194. Touch sensor 180K and display screen 194 together form a touch screen, also called a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor can transmit the detected touch operation to application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, in a different location than display screen 194.

[0111] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0112] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0113] This application does not specifically limit the structure of the execution subject of a frame playback method. As long as the code recording a frame playback method according to this application is executed, communication can be performed according to the frame playback method provided in this application. For example, the execution subject of a frame playback method provided in this application can be a functional module in an electronic device capable of calling and executing a program, or a communication device applied in an electronic device, such as a chip.

[0114] Users can use the gallery app on their electronic devices to view various photos stored on the device and play various videos stored on the device.

[0115] Reference Figure 2 This is a schematic diagram of an interface for playing videos stored on an electronic device through a gallery application, as provided in an embodiment of this application.

[0116] Reference Figure 2 (a) in the image gallery is a grid of images and videos stored on an electronic device, displayed by the image gallery application of the electronic device. The grid can display images and videos, where the cell containing a video displays the cover and duration of the video, and the cell containing an image displays a thumbnail of the image.

[0117] The grid image can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app, or it can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app and triggers more steps.

[0118] When displaying images and / or videos stored on an electronic device, this grid chart can be presented in various ways, for example, Figure 2 The method shown in (a) displays photos and videos in a photo group in reverse chronological order; in practical applications, gallery applications can also set up multiple object groups, for example, through... Figure 2 The control 20 shown in (a) can view multiple groups in the album: image group, video group, screenshot and screen recording group, etc. Users can also display pictures through the image group in the gallery application, videos through the video group in the gallery application, and screenshots and screen recordings through the screenshot and screen recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the grid diagram shown above.

[0119] This application does not limit the number of user operations required from the system desktop to the display of the grid chart in the gallery application, nor does it limit the way the grid chart displays images and videos.

[0120] Reference Figure 2 (b) in the text represents the user's click. Figure 2 Following the cover of video A in the grid diagram shown in (a), the electronic device displays the video playback interface of video A in response to the operation. The video playback interface includes a video playback area 21 and a progress display area 22.

[0121] The video playback area 21 is used to display the video frame during video playback. After entering the video playback interface from the grid interface, the control 211 is displayed in the video playback area. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the 0th second. When the video is paused, the control 211 is also displayed in the video playback area 21. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the current paused moment.

[0122] The progress display area 22 is used to display the video playback progress. The progress display area 22 includes a progress display frame 221 and a progress indicator line 222. The position of the progress indicator line 222 can remain unchanged. The user can drag the progress display frame 221 located below the progress indicator line 222 so that the progress display frame 221 moves left or right without moving the progress indicator line 222. The position of the progress indicator line 222 above the progress display frame indicates the current playback progress of the video.

[0123] As an example, the total playback duration of the current video is 15 seconds, and the total length of the current video's progress display frames is represented by L pixels. Based on these two parameters, we can obtain the playback duration corresponding to a unit length (or the length corresponding to a unit of time). For example, divide the playback duration 15 by the total length L. Therefore, if the leftmost edge of the progress display frame (the position corresponding to the first pixel) corresponds to second 0 of the video, then different positions of the progress indicator line on the progress display frame will correspond to different playback times. For example, if the distance between the position of the progress indicator line on the progress display frame and the leftmost edge of the progress display frame is X, then the playback time corresponding to the progress indicator line is: X*15 / L.

[0124] For ease of subsequent description, the embodiments in this application are described in the manner of "the time corresponding to the progress indicator line" or "the contact time".

[0125] Due to space limitations in the progress display area 22, only a portion of the video frames or a portion of the video frames from the progress display frame 221 can be displayed in the progress display area 22. For example... Figure 2 As shown in (b), the progress display frame 221 displayed in the progress display area 22 includes: thumbnails of three complete video frames and a partial area of ​​a thumbnail of a video frame.

[0126] The number of thumbnails displayed in the progress bar is usually related to the video length. Generally, the longer the video, the more thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames), and the shorter the video, the fewer thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames).

[0127] Typically, the video playback area 21 displays the video frame corresponding to the time indicated by the progress indicator line 222 in the progress display area 22.

[0128] In a specific implementation, there is a situation where the time corresponding to the progress indicator line 222 is the time between the playback times of two adjacent video frames, that is, there is no video frame in the video file corresponding to the time of the progress indicator line 222.

[0129] As an example, in a video frame sorted by playback time (denoted as timestamp) from smallest to largest, the playback time of the i-th video frame is ti, and the playback time of the (i+1)-th video frame is t(i+1); the time t corresponding to the progress indicator line 222 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval to the time t corresponding to the progress indicator line 221 can be taken as the video frame corresponding to the time of the progress indicator line 222.

[0130] Touch point time: The time when the progress indicator line points to is determined based on the touch point position, indicating the moment when the user drags their finger to the current touch point position and expects to display the image from the video file at that moment; for example, if the touch point time is 1015ms, it means that the user expects to display the image from the 1015ms position in the video file.

[0131] System time at the moment of contact: The system time at which the contact moment is acquired. For example, if the contact moment (at 1015ms) is acquired at 13:54:24 on the electronic device, then the system time at the contact moment (at 1015ms) is 13:54:24.

[0132] The video frame mapped to the touch point: The video frame in the video file whose timestamp is the smallest difference from 1015ms, which is the 1016ms video frame. For example, the video frame mapped to the touch point (1015ms) is the video frame with timestamp 1016.67ms.

[0133] Of course, in practical applications, if there are no video frames in the video file whose timestamps and progress indicator lines correspond to the same time, the video frame preceding time t (the video frame corresponding to ti) of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 221; or the video frame following time t (the video frame corresponding to t(i+1)) of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 222. In practical applications, the progress display area 22 can display the playback progress in many ways. For example, it can also display the video playback progress using a progress bar or a progress ring.

[0134] This application provides two playback modes for video playback: normal playback and frame-by-frame playback. The embodiments in this application only illustrate... Figure 2 This serves as an example to illustrate the difference between normal playback and frame playback.

[0135] Reference Figure 2 (c) in the text represents the user's click. Figure 2 After the control 211 (which can also be referred to as the playback control) in the interface shown in (b) is displayed, the electronic device responds to the operation by displaying an interface diagram during the playback of video A. This interface diagram indicates that the video A has been played to the 5th second, and the video frame corresponding to the 5th second is displayed in the video playback area 21. At the same time, the progress display frame 221 in the progress display area 22 moves its position so that the time corresponding to the progress indicator line 222 is the 5th second. Among them, from Figure 2 (b) in Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0136] A video file consists of multiple video frames, each with a timestamp. The timestamp records the playback time of each video frame during the video file's playback process. For example, arranged by timestamp from smallest to largest, the video frames in the file are: Video Frame 0 (timestamp t0), Video Frame 1 (timestamp t1), Video Frame 2 (timestamp t2), Video Frame 3 (timestamp t3), Video Frame 4 (timestamp t4), Video Frame 5 (timestamp t5), Video Frame 6 (timestamp t6), Video Frame 7 (timestamp t7)... Normal playback involves displaying each video frame sequentially in the playback area according to its corresponding timestamp, from smallest to largest.

[0137] As an example of a normal playback process, video frame 0 is displayed in video playback area 21 at time t0 after playback begins; video frame 1 is displayed in video playback area 21 at time t1 after playback begins; video frame 2 is displayed in video playback area 21 at time t2 after playback begins; video frame 3 is displayed in video playback area 21 at time t3 after playback begins; video frame 4 is displayed in video playback area 21 at time t4 after playback begins; video frame 5 is displayed in video playback area 21 at time t5 after playback begins; video frame 6 is displayed in video playback area 21 at time t6 after playback begins; video frame 7 is displayed in video playback area 21 at time t7 after playback begins…

[0138] from Figure 2 (b) in Figure 2 (c) shows the normal playback process of video A: starting from the 0th second of video A, each video frame is displayed sequentially according to the playback time indicated by the timestamp; Figure 2 (c) in the diagram is a schematic of the interface displaying the video frame up to the 5th second.

[0139] Reference Figure 2 In (d), the user drags... Figure 2 The progress display frame of the interface shown in (c) is in response to... Figure 2 As shown in (d) above, the electronic device displays the corresponding screen in the video playback area for the drag operation.

[0140] As an example, refer to Figure 2 In step (e), when the user drags the progress bar to display a frame such that the progress indicator line corresponds to the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be a video frame in video A with a timestamp of 14 seconds; it can also be a video frame with a timestamp closest to 14 seconds; it can also be a video frame before 14 seconds whose time interval between timestamp and 14 seconds is closest; or it can be a video frame after 14 seconds whose time interval between timestamp and 14 seconds is closest.

[0141] from Figure 2 (d) in Figure 2 The playback process shown in (e) is the frame playback process of the video.

[0142] As the user drags the progress display frame 221, a relative position is formed between the progress indicator line 222 and the progress display frame 221. The position of the progress indicator line 222 on the progress display frame 221 is related to the time corresponding to the progress indicator line 222.

[0143] In the specific implementation, assuming that the time indicated by the progress indicator line 222 is t1 before the user moves in the area where the progress display frame 221 is located on the screen. When the user moves left or right in the area where the progress display frame is located, the electronic device can detect the user's real-time touch position at certain time intervals. The electronic device determines the user's movement direction and distance based on the continuous real-time touch positions. The electronic device determines the time change Δt based on the user's movement distance; and determines the calculation method for the touch time t2 based on the user's movement direction. For example, if the movement direction is left, the touch time t2 is obtained by subtracting the time change Δt from the starting time t1; if the movement direction is right, the touch time t2 is obtained by adding the time change Δt to the starting time t1. After obtaining the touch time t2, the electronic device searches for the video frame corresponding to the touch time t2 and displays the video frame corresponding to the touch time t2 in the video playback area.

[0144] It's understandable that the time or touch point time corresponding to the progress indicator line is not the system time, but rather the playback time in the video file where the touch point position is mapped. The video frame corresponding to touch point time t2 can be found in the detailed explanation of "the video frame corresponding to the 14th second". The video frame corresponding to touch point time t2 can also be understood as the video frame mapped to touch point time t2.

[0145] On the other hand, the electronic device obtains the movement direction of the progress display frame based on the movement direction determined by the real-time touch point position, and determines the movement distance of the progress display frame based on the movement distance of the real-time touch point position; the electronic device moves the progress display frame in the progress display area according to the movement direction and movement distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0146] As another example, the electronic device also determines the moving direction (left if t2 is greater than t1, right if t2 is less than t1) and moving distance of the progress display frame based on the start time t1 and the touch time t2; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0147] It should be noted that the above process is only an example. In actual applications, there are multiple ways to determine the touch point time corresponding to the progress indicator line 222, the position of the progress display frame 221, and the video frame displayed in the video playback area 21 when the user drags the progress display frame 121.

[0148] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. It may move to the left, it may move to the right, it may be dragged quickly, or it may be dragged slowly. Accordingly, the video frame images displayed in the video playback area 21 have randomness.

[0149] As an example, before the user moves within the area of ​​the progress display frame on the screen, the starting time indicated by the progress indicator is t1. During the user's dragging of the progress display frame, the electronic device detects the user's real-time touch position at regular time intervals, thus obtaining multiple consecutive time periods corresponding to the real-time times: t2, t3, t7, t4, and t3. In the video playback area, video frames are played in the following order: starting with the video frame corresponding to time t1, the video frames corresponding to time t2, t3, t7, t4, and t3 are played sequentially.

[0150] It can be understood that the process of a user clicking control 21 to trigger the electronic device to play video in the video playback area is normal playback; the process of a user dragging the progress display frame to trigger the electronic device to display the video frame corresponding to the progress indicator line in the video playback area is frame playback.

[0151] Figure 2 The example shown is intended to illustrate the difference between normal playback and frame playback, and does not imply that the interface during normal playback and frame playback must be identical. Figure 2 Consistent with the above.

[0152] In the above embodiments, whether it is normal playback or frame playback, the video frames need to be decoded before they are displayed in the video playback area. Usually, the video frames are decoded in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a keyframe (I-frame), at least one forward reference frame (P-frame), and at least two bidirectional reference frames (B-frames).

[0153] Reference Figure 3 This is a schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application. A set of video frames can also be referred to as a group of pictures (GOP).

[0154] Each group of video frames, in playback order, includes: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. Among these, the I-frame, as a keyframe, can be decoded to obtain a complete image, but decoding is slower. The P-frame records the changes relative to the previous frame (not a B-frame); information from the previous frame is required to decode the P-frame. Similarly, information from previous P-frames (which may also be I-frames) and subsequent P-frames is needed to decode the B-frame. P-frames and B-frames improve video smoothness while reducing the file size.

[0155] Whether it's an I-frame, P-frame, or B-frame, they are all video frames. When decoding a video frame at a specific moment, you don't directly decode the video frame at that moment. Instead, you decode the group of video frames that the video frame at that moment belongs to. When decoding video frames within a group of video frames, you decode them one by one in the decoding order.

[0156] As an example, if we need to decode the video frame at time t1, and the video frame at time t1 is the 6th video frame in the playback order of a group of video frames: frame B; then the decoding order is:

[0157] First decoding: the video frame played in the first playback order: I-frame;

[0158] The second decoding: the 4th video frame in playback order: P-frame;

[0159] The third decoding step: the second video frame played in the playback order: frame B;

[0160] The fourth decoding: the third video frame in playback order: frame B;

[0161] The fifth decoding: the 7th video frame in playback order: P-frame;

[0162] The sixth decoding: the 5th video frame in playback order: frame B;

[0163] The seventh decoding: the 6th video frame in playback order: frame B;

[0164] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding operation to decode the video frame at time t1.

[0165] Currently, in normal playback scenarios, the video frames in each video frame group are parsed, decoded, rendered, and displayed sequentially according to the playback order. Of course, when decoding the video frames in each video frame group, the process is as follows: Figure 3The decoding order shown decodes each video frame in a video frame group sequentially. Normally, in a continuous playback scenario, when displaying frames from the previous video frame group, all frames in the next video frame group have already been decoded. However, when switching from frame playback mode to normal playback mode, it's necessary to determine the video frame group containing the latest frame based on its playback time, and then decode the video frames sequentially according to the above decoding order.

[0166] In a frame playback scenario, the process of a user dragging the progress bar to display a frame is uncertain. The user may drag the progress bar to the left, or to the right, or drag it quickly or slowly. Consequently, the video frame images displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display a frame, the electronic device needs to determine the moment the progress indicator line points based on the real-time detected touch point position, and then decode the video frames sequentially according to the above decoding order.

[0167] However, since the user's dragging process may be fast or slow, if the method described in the above embodiment is followed exactly: each time a touch point is detected, the touch point is mapped to a video frame, the video frame is decoded (in groups and in the order of the group's decoding), and then rendered and displayed, there may be a phenomenon where the video frame displayed in frame playback does not correspond to the real-time touch point position of the user's finger, resulting in screen stuttering.

[0168] This application provides a frame playback method. During frame playback (or normal playback), the YUV data of some successfully decoded video frames can be cached. In subsequent frame playback, if the video frame mapped at the touch point is a cached video frame, or if there is a video frame in the cache whose timestamp difference with the video frame mapped at the touch point is less than a threshold, the YUV data of the mapped video frame (or YUV data with timestamps less than the threshold) can be retrieved from the cache and rendered directly. Since no decoding is required, the stuttering caused by decoding time is avoided, and the smoothness of the frame playback is improved.

[0169] Reference Figure 4 This is a schematic diagram illustrating a scenario where frame caching and YUV data retrieval from the cache are performed, as provided in an embodiment of this application.

[0170] and Figure 2 In the scenario shown, after the video file is opened, it first plays normally from the 0th second; then it plays normally until the 5th second, when the user drags the progress display frame in the progress display area to trigger a switch to frame playback; after the user slides the progress display frame to trigger playback of the 14th second video frame, the user lifts their finger to trigger a switch back to normal playback, and the electronic device continues to play the video frame normally from the 14th second.

[0171] Assuming the video file has a playback duration of 50 seconds, the electronic device plays the video from the 14th to the 19th second in normal playback mode. When the user drags the progress display frame, it triggers a switch to frame playback. When the user drags the progress display frame, the time the progress line points to changes from the 19th second to the 28th second. After the user lifts their finger, the electronic device switches back to normal playback and continues playing from the 28th second in normal playback mode.

[0172] The video frame caching method used in this application primarily caches the decoded data (e.g., YUV data) of video frames for use during frame playback. The cached YUV data can be directly rendered; therefore, the caching process is typically executed only when a user's intention to play the frame is detected. However, caching the YUV data in memory can strain memory resources. To optimize memory utilization, the video frame caching method provided in this application can be initiated after the first frame playback is triggered when the video file is opened.

[0173] The playback mode is from second 0 to second 5 of the video file.

[0174] From the 5th to the 14th second of the video file, it is in frame playback mode, which is also the first frame playback triggered after the video file is opened. Therefore, during frame playback, some video frames can be selected for caching, and under appropriate conditions, the cached video frames (without decoding) can be used for rendering.

[0175] From the 14th to the 19th second of the video file, it is in normal playback mode, which is also the playback state after the first triggered frame after the video file is opened locally. Therefore, during normal playback, some video frames can be selected for caching. However, since normal playback decodes in advance in sequence, it is not necessary to use the cached video frames for rendering.

[0176] From the 19th to the 28th second of the video file, it is in frame playback mode. During frame playback, some video frames can be selected for caching, and under appropriate conditions, the cached video frames can be used for rendering (without decoding).

[0177] Of course, the above scenarios are for illustrative purposes only, and the frame playback scenarios are all forward sliding scenarios (the time of the next touch point is greater than the time of the previous touch point). In practical applications, it can be applied to reverse frame playback scenarios (the time of the next touch point is less than the time of the previous touch point).

[0178] After describing the above scenarios, it can be understood that both frame playback scenarios and normal playback scenarios may execute the frame caching process.

[0179] The processes of normal playback and frame playback are described below.

[0180] S101, while the gallery application is displaying the video playback interface of video A, it receives a click operation on a regular playback control. For example, Figure 2 The click operation on the control shown in (b) is shown in the image.

[0181] S102, the gallery application launches the playback engine.

[0182] S103, the playback engine sends a playback command for video A to a regular player.

[0183] S104: A standard media player parses video file A, obtaining H.264 format video frames (for example only). It then decodes these H.264 frames to obtain YUV data, and renders the YUV data to produce a rendered image. This rendered image can be displayed in the video preview area of ​​the gallery application.

[0184] This process corresponds to Figure 2 (b) to Figure 2 (c) In this context, after decoding the H.264 format video frames to obtain YUV data, a portion of the video frames can be selected as buffered video frames for caching.

[0185] While video A is playing in normal playback mode in the gallery app, a progress display area is shown at the bottom of the screen. The user can drag the progress frames in this area to trigger the electronic device to play video A frame by frame. The touchpad of the electronic device monitors and reports the touch point position at a certain frequency, and the touch point position of the user's finger can determine the touch time.

[0186] The following is through Figure 5 Describe the frame playback process in detail. This process corresponds to... Figure 2 (d) to Figure 2 (e) in the middle.

[0187] S201, the playback engine receives the reported touch point time t1 and the corresponding system time T1.

[0188] In this embodiment of the application, if the reported touch point time t1 is the user's first touch point time, the playback engine also needs to send a pause command to the ordinary player, and the ordinary player stops parsing, decoding and rendering according to the pause command.

[0189] S202, the playback engine sends the touch point time t1 and system time T1 to the seek management module.

[0190] S203, the playback engine sends a parsing instruction to the parser, carrying the contact time t1.

[0191] S204, the parser parses the video file and obtains the parsed data of the video frames in the GOP where t1 is located, for example, H.264 format data.

[0192] In practical applications, video frames in a video file are divided into multiple Groups of Pictures (GOPs). Each GOP has a playback start time (e.g., the timestamp of the first video frame in the current GOP) and a playback end time (the timestamp of the first video frame in the next GOP). Of course, this is just an example. In practice, there is a time interval between two adjacent video frames in two GOPs (e.g., 16.67ms in a 60FPS scenario). Therefore, there are multiple ways to determine the playback time range for each GOP. The video frames in the GOP containing t1 can be parsed. In practice, the parser can also parse one or two more GOPs. For example, upon receiving a parsing instruction carrying t1, it can parse the GOP containing t1, as well as the video frames in the GOPs before and after it. If the parser is fast enough, it can even parse the entire video file.

[0193] S205, the playback engine receives the reported touch point time t2 and the corresponding system time T2.

[0194] Contact time t2 and contact time t1 are in the same GOP group.

[0195] S206, the playback engine sends t2 and T2 to the seek management module.

[0196] S207, the seek management module sends a parsing instruction to the parser, carrying the contact time t2.

[0197] S208, the parser determines that the video frames in the GOP group where t2 is located have been parsed, and will not repeat the parsing.

[0198] Similarly, when a parsing instruction carrying t2 is received, if t2 and t1 are in the same GOP group, the parsing will not be repeated. If t2 and t1 are not in the same GOP group, the video frames of the GOP group where t2 is located, as well as the video frames of the GOP groups before and after it, will be parsed. Of course, if any of the three GOP groups corresponding to t2 have been successfully parsed, only the GOP groups that have not been successfully parsed need to be parsed.

[0199] S209, the seek manager retains a contact time within the range of the refresh cycle in each refresh cycle based on the received contact time. For example, t1 and t2 retain t2.

[0200] Reference Figure 6This is the process provided in the embodiments of this application for retaining a contact moment within the range of that refresh cycle in each refresh cycle.

[0201] The frequency of contact reporting is usually high, generally at least 120Hz. The contact time is related to the real-time position of the user's finger. When the user's finger slides faster, the contact time distribution is more scattered (the difference between adjacent contact times is larger). When the user's finger slides slower, the contact time distribution is more concentrated (the difference between adjacent contact times is smaller).

[0202] With a screen refresh rate of 60Hz, this means that one video frame is rendered and displayed every 16.67ms. When the user's finger swipes slowly, multiple touch points may be received within a single 16.67ms filtering cycle. In other words, the duration of the filtering cycle can be equal to the duration of the screen refresh cycle.

[0203] like Figure 6 As shown, when the user's finger drags slowly, the touch point may be at 4ms, 6ms, 13ms, 20ms, 23ms, and 35ms.

[0204] In practical applications, it is not necessary to map every touch point moment to a specific video frame. For example, 4ms and 6ms might map to the same video frame, and 13ms, 20ms, and 23ms might also map to the same video frame. Therefore, it is necessary to filter out some touch point moments.

[0205] Reference Figure 6 (a) illustrates a filtering method provided in this application embodiment. This method aligns the filtering period T with the timestamps of video frames. For example, if the video file has a frame rate of 60 FPS, using the timestamps of two adjacent video frames as the dividing points between different periods, any touchpoint moment before 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. Alternatively, the first touchpoint moment before 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. Furthermore, the last touchpoint moment between 0ms and 16.67ms can be retained, while other touchpoint moments are filtered out. For instance, in the illustration, the last touchpoint moment between 0ms and 16.67ms, at 13ms, is retained, and the last touchpoint moment between 16.67ms and 33.33ms, at 23ms, is retained. Other touchpoint moments are deleted.

[0206] This application embodiment does not limit which touch moment to retain in each cycle. The purpose of this step is to retain one touch moment within the cycle (the filtering cycle converted from the screen refresh rate).

[0207] Of course, if the video file's frame rate is 120 FPS, then the timestamps of the video frames are 0ms, 8.33ms, 16.67ms, 25ms, 33.33ms, and so on. Since the screen refresh rate is 60Hz, the filtering period is still calculated using the screen refresh rate conversion cycle from 0ms to 16.67ms as one filtering period, from 16.67ms to 33.33ms as another filtering period, and so on.

[0208] Reference Figure 6 (b) in this embodiment represents another filtering method provided by this application, in which the two endpoints of the period and the timestamp of the video frame may not be aligned. Taking a filtering period of 16.67ms as an example, the period from 5ms to 21.67ms is one period, the period from 21.67ms to 38.33ms is another period, and so on.

[0209] Within the period from 5ms to 21.67ms, any one contact moment can be retained while other contact moments are filtered out. For example, the first or last contact moment can be retained while other contact moments are filtered out.

[0210] Of course, continuing to take the example of retaining the last contact moment, the retained contact moments are 4ms, 20ms and 35ms respectively.

[0211] Reference Figure 6 (c) in this application provides another filtering method. The previously determined contact time (or the first contact time) of 4ms is taken as the starting point of this filtering cycle, and then extended by 16.67ms to 20.67ms. The last contact time (20ms) in the filtering cycle from 4ms to 20.67ms is retained, and other contact times between 4ms and 20ms are deleted. Continuing, the previously determined contact time of 20ms is taken as the starting point of this filtering cycle, and then extended by 16.67ms to 36.67ms. The last contact time (35ms) between 20ms and 36.67ms is retained, and other contact times between 20ms and 36.67ms are deleted.

[0212] Although the touch point times determined by the three methods may be different, a video frame is retained within 16.67ms. Even if different touch point times are mapped to different video frames, the difference in display is only between the preceding and following frames. The frame playback process itself is the process of displaying video frames related to the drag position as the user drags their finger. The difference between the preceding and following frames will not cause a visual difference for the user.

[0213] Of course, if there is no contact moment or only one contact moment in a certain period, the above filtering process is not required, and the contact moment is directly retained.

[0214] In this embodiment, t1 and t2 are within the same cycle, and the contact time t2 is retained while the contact time t1 is filtered out.

[0215] After filtering out redundant contact times, the contact times can be stored in the seek data table.

[0216] S210, map the touch point time t2 within the current filtering period T to the video frame with the most recent timestamp and touch point time t2.

[0217] In this application, the timestamps of two adjacent video frames at the touch point time t2 may be calculated with t2 respectively, and the adjacent video frame with the smallest difference may be used as the mapped video frame.

[0218] Reference Figure 7 As shown, the touch point time t2 is the 13ms. The timestamps of the two video frames adjacent to the 13ms are the 0ms and 16.67ms. The difference between the 0ms and 13ms (13ms) and the difference between the 16.67ms and 13ms (3.67ms) are calculated respectively. Therefore, the touch point time t2 (the 13ms) is mapped to the video frame with the timestamp of the 16.67ms.

[0219] Of course, in practical applications, if two consecutive touch points are mapped to the same video frame, one of the touch points in the seek data table can be deleted.

[0220] S211, determine whether the mapped video frame meets the condition for adjustment to an I-frame. If it does, adjust the mapped video frame to an I-frame.

[0221] In this step, when the current video frame and the adjacent video frame belong to different GOP groups, it usually occurs in scenarios where the user drags relatively quickly. It is necessary to decode the video frames in the GOP group where the current video frame belongs and the video frames in the GOP group where the adjacent video frame belongs. Both of the video frames that are rendered and displayed consecutively need to be decoded in their respective GOP groups. Especially when the decoding order of the video frame in the GOP group is later, the decoding speed may be slower, and it may even lead to frame loss.

[0222] To quickly respond to user dragging operations, the current video frame can be adjusted to an I-frame between adjacent video frames. This means that when decoding this video frame, the first video frame in the GOP group containing it is the mapped video frame, resulting in faster decoding speeds.

[0223] Reference Figure 8As shown in (a), during the user's forward swipe, i.e., when the user's finger swipes from right to left, the touch point time corresponding to the progress indicator line becomes larger and larger. The timestamp of the video frame mapped to the touch point time becomes larger and larger. During the forward swipe, the current mapped video frame and the next mapped video frame are usually regarded as two adjacent mapped video frames.

[0224] If the touch point of the current video frame is 33.3ms and the next neighboring video frame is 216.6ms, the GOPs at 33.3ms and 216.6ms are different GOPs, and there is an I-frame between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the middle I-frame (116.6ms).

[0225] In this way, when decoding the video frame corresponding to the touch point t2 (adjusted from 33.3ms to 116.6ms), it is only necessary to decode one video frame in the GOP containing 116.6ms to achieve successful decoding; compared to decoding the video frame in 33.3ms, which requires decoding the fourth video frame to achieve successful decoding, the decoding speed is faster.

[0226] In current practical applications, when the touch point of the current video frame is 33.3ms and the next neighboring video frame is 266.6ms, the GOPs at 33.3ms and 266.6ms are different GOPs, and there are two I-frames between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the I-frame (116.6ms) that is closest to the current video frame (33.3ms) in the middle I-frames.

[0227] Therefore, during forward sliding, regardless of how many I-frames exist between the current video frame and the next adjacent frame, the current video frame will be replaced with the nearest I-frame.

[0228] Reference Figure 8 As shown in (b), during the user's reverse swipe, i.e., when the user's finger slides from left to right, the touch point time corresponding to the progress indicator line becomes smaller and smaller. The timestamp of the video frame mapped to the touch point time becomes smaller and smaller. During the reverse swipe, the current mapped video frame and the previous mapped video frame are usually regarded as two adjacent mapped video frames.

[0229] If the touch point of the current video frame is 33.3ms and the previous neighboring video frame is 216.6ms, the GOP at 33.3ms and the GOP at 216.6ms are different GOPs, and there is an I-frame between the two video frames, then the current video frame (33.3ms) can be adjusted to the middle I-frame (116.6ms).

[0230] In this way, when decoding the video frame corresponding to the touch point t2 (adjusted from 33.3ms to 116.6ms), it is only necessary to decode one video frame in the GOP containing 116.6ms to achieve successful decoding; compared to decoding the video frame in 33.3ms, which requires decoding the fourth video frame to achieve successful decoding, the decoding speed is faster.

[0231] In current practical applications, when the touch point of the current video frame is 33.3ms and the previous neighboring video frame is 266.6ms, the GOP at 33.3ms and the GOP at 266.6ms are different GOPs, and there are two I-frames between the two video frames. In this case, the current video frame (33.3ms) can be adjusted to the I-frame (116.6ms) that is closest to the current video frame (33.3ms) in the middle I-frame.

[0232] Therefore, during reverse scrolling, regardless of how many I-frames exist between the current video frame and the previous adjacent frame, the current video frame is replaced with the nearest I-frame.

[0233] S212, the seek management module sends a query command for mapped video frames to the decoder manager.

[0234] In this embodiment, if the target I-frame is not reached, the most recent mapped video frame is taken as the determined video frame. If the target I-frame is reached, the reached I-frame is taken as the mapped video frame (which can also be understood as a candidate video frame in the seek data table), and this mapped video frame is denoted as video frame b. Of course, the mapped video frames in the seek data table can also be denoted as candidate video frames.

[0235] In this embodiment, the determined video frame can be understood as the video frame that needs to be displayed. Of course, in practical applications, determining that a video frame needs to be displayed does not mean that it will definitely be rendered and displayed later; it may also be abandoned from rendering and display. Please refer to the following embodiments for details.

[0236] S213, the decoder manager sends a query command for video frame b to the decoder buffer.

[0237] S214, if the decoding buffer finds cached video frame b or cached timestamp and video frame b timestamp difference less than the threshold, then execute S215 and S216 (in subsequent embodiments). If video frame a is not cached and there is no cached timestamp and video frame b timestamp difference less than the threshold, then execute S219.

[0238] S219, the decoder buffer sends information about the unbuffered video frame b to the decoder manager.

[0239] S220, after receiving the information of the unbuffered video frame b, the decoding manager sends the decoding instruction of video frame a to the decoder.

[0240] S224, the decoder obtains the parsing data of the video frame group where video frame b is located from the buffer data corresponding to the parser.

[0241] S225, the decoder decodes according to the decoding order. The identifier of video frame b is sent to the rendering queue maintained by the display manager.

[0242] S226, the decoder uses the seek manager or display manager to determine whether to display video frame b or not.

[0243] In practice, the step of determining which video frame b to display and which to not display can be performed in any module preceding the renderer. The display and non-display video frames will be described in subsequent embodiments.

[0244] S227, the decoder sends the YUV data of video frame b and non-display video frames into the buffer queue.

[0245] In practice, non-display video frames are a subset of video frames within the same video frame group as video frame b. Display video frame b and non-display video frames can be placed in different buffer queues.

[0246] S228, the display manager determines the currently ready-to-render video frame b from the cache queue of cached display video frames according to the display strategy.

[0247] In this embodiment, with a screen refresh rate of 60Hz, which is equivalent to displaying 16.6ms per second, it is not necessary to render every video frame in practical applications. Therefore, if there are multiple decoded video frames in the rendering queue, the most recently added video frame may be selected for rendering. This results in some video frames in the cache queue being rendered and displayed, while others may not be rendered.

[0248] This application embodiment illustrates a situation in which video frames in the cache queue (a cache queue used to store display video frames) are not rendered and displayed.

[0249] The rendering frame rate is set to 60Hz. Every 16.6ms, the display manager retrieves the latest video frame from the cache queue and sends it to the renderer for decoding. Although the touch moments have been filtered, in actual applications, the number of touch moments per second may still exceed 60. Therefore, there may be situations where some video frames mapped to certain touch moments are successfully decoded and sent to the cache queue as video frames to be displayed, but are not rendered.

[0250] Reference Figure 9 As shown, after receiving vsync1, the latest decoded video frame b is retrieved from the cache queue, and the rendering of video frame b begins. At the same time, video frame b is released. Video frames a before video frame b are abandoned for rendering and display, and video frame a can be temporarily not released.

[0251] Upon receiving vsync2, retrieve the latest decoded video frame c from the cache queue, begin rendering video frame c, and simultaneously release video frame c.

[0252] Upon receiving vsync3, the newly sent decoded video frame e is retrieved from the cache queue, and video frame e is rendered and released simultaneously. Video frames d preceding video frame e are not rendered and can be temporarily left unreleased.

[0253] The timing of the release of video frame a can be referred to the description in the following embodiments.

[0254] Figure 9 In the example shown, the video frames mapped at the touch point include b, c, and e, which are rendered and displayed. The video frames not rendered and displayed include a and d.

[0255] S229, The display manager sends rendering instructions for video frame b (for example only) to the renderer.

[0256] S230, the renderer renders video frame b.

[0257] according to Figure 9 As shown, when rendering video frames, the renderer does not render a video frame every time the decoded data of a video frame that needs to be rendered is sent to the cache queue. Instead, it retrieves the latest decoded video frame data from the cache queue according to a specific rendering frame rate. This may result in some video frames in the cache queue being successfully rendered and displayed, while others may be abandoned.

[0258] In this embodiment of the application, the video frames that may be cached are divided into two main categories: display video frames and non-display video frames;

[0259] The displayed video frames are the video frames mapped to the touch points in the above embodiments. These video frames are sent to the buffer queue after successful decoding. A first buffer queue can be set to buffer the displayed video frames.

[0260] Non-display video frames include two types: preset buffer video frames and temporary buffer video frames. Preset buffer video frames are the video frames at the end of the GOP group containing the display video frames, and a second buffer queue can be set up to buffer preset buffer video frames; temporary buffer video frames are the video frames adjacent to the display video frames, and a third buffer queue can be set up to buffer temporary buffer video frames.

[0261] For each possible cached video frame, there are marking conditions (i.e., conditions for determining that it is a cached video frame) and release conditions (i.e., conditions for releasing the video frame from the cache queue). The marking conditions and release conditions are described below for each type of cached video frame.

[0262] (a) Display the marking and release conditions of video frames.

[0263] Reference Figure 10 The diagram shows the marking and release conditions for displaying video frames provided in this embodiment of the application.

[0264] A1, successfully decoded and mapped video frame 1.

[0265] In this embodiment of the application, each touch point can be mapped to a video frame at any time. As mentioned above, the mapped video frame may be any frame in the GOP group.

[0266] A2 marks the mapped video frame 1 as the display video frame 1, and sends the decoded data of the display video frame 1 into the buffer queue 1 (first buffer queue).

[0267] In this embodiment, cache queue 1 is used to cache the decoded data of the displayed video frames. According to A1 and A2, in this embodiment, during the process of the user dragging the progress bar to display the frames, the touch moments detected by the electronic device are filtered, and each touch moment can be mapped to a corresponding video frame. These video frames can be determined as those that need to be rendered and displayed, and these video frames that need to be rendered and displayed are marked as display video frames.

[0268] A3, determine whether video frame 1 has been successfully rendered and displayed.

[0269] In this embodiment of the application, according to Figure 9 It is understandable that some video frames sent to buffer queue 1 can be successfully rendered and displayed, while others cannot.

[0270] A4, if video frame 1 is successfully rendered and displayed, set video frame 1 to enter an invalid state.

[0271] A5, if the display video frame 1 is not successfully rendered (e.g., rendering is abandoned), monitor the cache duration of the display video frame 1 in the cache queue 1.

[0272] A6. Determine whether the buffer duration of the displayed video frame 1 in the buffer queue 1 is greater than the threshold (first duration).

[0273] If the buffering time of displayed video frame 1 in buffer queue 1 is not greater than the threshold, continue to monitor the buffering time of displayed video frame 1 in buffer queue 1.

[0274] A7, if the buffer duration of display video frame 1 in buffer queue 1 exceeds the threshold, set display video frame 1 to enter an invalid state.

[0275] A8. After displaying video frame 1 enters the invalid state, check whether displaying video frame 1 is also marked as a non-displaying video frame.

[0276] As mentioned earlier, the video frames mapped to the touch points within the same Group of Pictures (GOP) are marked as display video frames. Simultaneously, some video frames at the end of the GOP can be marked as non-display video frames. Therefore, a video frame may be marked as both a display and a non-display video frame. Of course, the video frame may also be marked as either a display or a non-display video frame, or it may not be marked as either.

[0277] A9, if video frame 1 is displayed but also marked as a non-display video frame, then video frame 1 will be converted to a non-display video frame.

[0278] When display video frame 1 is sent to the buffer queue corresponding to the non-display video frame as a non-display video frame, display video frame 1 will be released from buffer queue 1.

[0279] A10, if display video frame 1 is not marked as a non-display video frame, release display video frame 1 from buffer queue 1.

[0280] according to Figure 10 Understandable. Figure 9 If video frame 'a' is not rendered (i.e. abandoned for rendering), it may be temporarily cached for a certain period of time. If it is marked as a non-display video frame in subsequent viewings, it will be converted into a non-display video frame and moved to another cache queue. If it is not marked as a non-display video frame in subsequent viewings, it will be released directly from this cache queue.

[0281] (ii) Marking and releasing conditions for pre-buffered video frames.

[0282] Marking criteria: At the end of the GOP group containing the currently being decoded mapped video frames, select a portion of video frames and mark them as preset buffered video frames according to the length of the buffer region Lpcf, the number of buffers Npcf, and the buffer interval Dpcf.

[0283] (1) First, calculate the number of intervals D between the selected preset buffer video frames. PCF .

[0284]

[0285] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0286] The video frame rate is fixed; for example, the video file itself may be 60fps (60 frames per second when playing the video in normal playback mode), 90fps, or 120fps. The rendering display frame rate represents the real-time rendering frame rate of the electronic device. Even if the refresh rate of the electronic device is 60Hz, the real-time rendering frame rate may not be stable at 60fps, but may fluctuate due to factors such as decoding speed. In this embodiment, the rendering frame rate is the frame rate detected in real time (e.g., it can be detected once per second, or once every 500ms, etc.), and the scrolling speed is the number of pixels scrolled per second.

[0287] As an example, during the forward scrolling process, if the video file has a frame rate of 120fps and the real-time rendering display frame rate is 60fps, then the buffer quantity interval is 2; when the video frame rate is 60fps and the real-time rendering display frame rate is 59fps, the buffer quantity interval is 1.

[0288] (2) Calculate the number and length L of the preset buffer video frame region. PCF .

[0289]

[0290] Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps This indicates the number of video frames decoded per second by the decoder. It is a real-time value of the current device's decoding rate, calculated from decoder statistics.

[0291] As an example, with a video frame rate of 60, a decoding rate of 120, and a GOP group containing 12 video frames, the total length is 4 frames.

[0292] (3) Calculate the cache size N based on the length and interval of the cache. PCF .

[0293]

[0294] Specifically, at least one video frame is marked at the end of the GOP as a pre-buffered video frame. The number of buffers is determined to be 2 based on the calculated length (e.g., 4 frames) and the number interval (e.g., 2).

[0295] Reference Figure 11 As shown, after determining the above parameters, during normal playback or frame playback, the video frames marked as preset cached video frames are determined according to the number of lengths, number of intervals, and number of buffers.

[0296] Reference Figure 12 In order to follow Figure 11 As an example of the method shown, continuing with a length of 4, an interval of 2, and a buffer size of 2, we take the selection of a portion of video frames at the end of the GOP as preset buffer video frames. Starting from the end of the GOP and counting 4 video frames from the length of the length, the fourth-to-last video frame (the circled P frame in the figure) is selected as the first preset buffer video frame. Then, according to the interval of 2, the second-to-last video frame (the circled B frame in the figure) is selected as the second preset buffer video frame.

[0297] In addition, in practical applications, one or more GOP groups can be decoded in advance. For example, in practical applications, the positions of the next n (e.g., 1, 2, 3, etc.) touch points can be predicted based on the current touch point position and sliding speed, and the touch point time can be calculated for each of the predicted next n touch point positions; or the touch point time can be directly predicted based on the current touch point time and sliding speed; the predicted next GOP group is obtained based on the predicted touch point time, and the predicted next GOP group can be decoded in advance. When decoding the predicted GOP group in advance, a portion of video frames can be selected as pre-buffered video frames for caching.

[0298] In practical applications, it is also possible to configure the pre-decoding of video frames of the GOP group to be performed during normal playback or forward frame playback.

[0299] Release condition: If the preset buffered video frames are no longer in the GOP group of the currently displayed video frame or the previous GOP group (based on the order of GOP groups in the video file), the preset buffered video frames in that group will be released.

[0300] In this embodiment of the application, there is also a limit to the caching duration of each group of preset cached video frames.

[0301] For example, if the determined display video frames are distributed in multiple consecutive video frame groups, and the video frame group containing the cached preset video frames is the i-th video frame group, when the currently determined display video frame is a video frame in the (i+1)-th video frame group, the preset cached video frames in the i-th video frame group can still continue to be cached. When the currently determined display video frame is a video frame in the (i+2)-th video frame group, the video frames in the i-th video frame group do not belong to the (i+2)-th video frame group, nor are they located in the previous video frame group of the (i+2)-th video frame group. Therefore, the preset cached video frames in the i-th video frame group can be deleted.

[0302] In addition, in practical applications, the video frame group to which the determined display video frame is located may not be continuous.

[0303] Reference Figure 13 Assume that the determined consecutive display video frames are located in the i-th video frame group, the (i+2)-th video frame group, and the (i+4)-th video frame, respectively.

[0304] The video frame group containing the cached preset video frame is the i-th video frame group. When the currently determined display video frame is a video frame in the (i+2)-th video frame group, the i-th video frame group is not the previous video frame group of the current (i+2)-th video frame group, and the preset cached video frame in the i-th video frame group can be released.

[0305] It can be understood that the cached preset video frames are always located near the currently displayed video frame or the currently decoded video frame (in the same video frame group or adjacent video frame groups). Preset cached video frames that are not located nearby will be released. This setting can make better use of the application's forward sliding and then reverse sliding. When sliding in reverse, the cached video frames at the end of the same video frame group or the previous video frame group can be used.

[0306] Of course, when applying this to a scenario where a reverse swipe is followed by a forward swipe, the release condition needs to be set as follows: release the pre-buffered video frames in a group if the pre-buffered video frames are no longer in the GOP group of the currently displayed video frame or in the next GOP group. Similarly, it can be set to pre-decode the video frames of the preceding GOP group when playing reverse frames.

[0307] (III) Marking and releasing conditions for temporary buffered video frames:

[0308] Marking criteria: In the vicinity of the currently being decoded mapped video frame, select a portion of video frames as temporary buffer video frames according to the buffer area length Ltcf, the number of buffers Ntcf, and the buffer interval Dtcf.

[0309] (1) First, calculate the number of intervals D between the selected preset buffer video frames. TCF .

[0310]

[0311] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0312] The video file has a fixed frame rate. For example, the video file itself may be 60fps (60 frames per second when playing the video in normal playback mode), 90fps, or 120fps. The rendering display frame rate represents the real-time rendering frame rate of the electronic device. Even if the refresh rate of the electronic device is 60Hz, the real-time rendering frame rate may not be stable at 60fps, but may fluctuate due to factors such as decoding speed. In this embodiment, the rendering frame rate is the frame rate detected in real time (e.g., it can be detected once per second, or once every 500ms, etc.), and the scrolling speed is the number of pixels scrolled per second.

[0313] As an example, during the forward scrolling process, if the video file has a frame rate of 120fps and the real-time rendering display frame rate is 60fps, then the buffer quantity interval is 2; when the video frame rate is 60fps and the real-time rendering display frame rate is 59fps, the buffer quantity interval is 1.

[0314] It is understandable that the interval between the number of temporary buffered video frames and the interval between the number of preset buffered video frames can be the same. Of course, in practical applications, they can also be set to be different.

[0315] When swiping forward, the cache size can be set to 2. When swiping backward, the cache area size can be calculated first, and then the cache size can be obtained based on the cache area size and the cache size interval.

[0316] (2) Calculate the cache area size L TCF .

[0317]

[0318] Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps The number of video frames the decoder decodes per second.

[0319] As an example, with a video file frame rate of 60, a decoding rate of 120, and a GOP group containing 12 video frames, the total length is 4 frames.

[0320] (3) Calculate the cache size N TCF .

[0321]

[0322] Specifically, during forward sliding, two video frames are selected as temporary buffer video frames in the region adjacent to the mapped video frame; during reverse sliding, at least one video frame is selected as a temporary buffer video frame in the region adjacent to the mapped video frame.

[0323] Reference Figure 14 As shown, as an example of forward swiping, the number of buffers is determined to be 2, and the two video frames before the mapped video frame are marked as temporary buffer video frames. In this way, in the scenario of switching from forward swiping to reverse swiping, the video frame mapped at the touch point is likely to be in the buffer, thereby reducing the time waiting for decoding data and improving the smoothness of the picture when playing the frame.

[0324] Reference Figure 15 As an example of reverse sliding, the determined buffer size is 4. The 4 video frames before the mapped video frame are marked as temporary buffer video frames, with an interval of 2. Also, a portion of video frames (circled in the diagram) are selected from the video frames before the mapped video frame as temporary buffer video frames.

[0325] It is understandable that, regardless of whether the current swipe is forward or reverse, one or more video frames preceding the mapped video frame will be used as temporary cached video frames. This ensures that the video frames mapped at subsequent touch points may be in the cache during the transition from forward to reverse swipe or reverse swipe scenarios, thus resolving the stuttering issue during the transition.

[0326] Of course, in practical applications, to broaden the application scenarios, a subset of video frames can be selected before mapping video frames, and another subset can be selected as temporary buffer video frames after mapping video frames. This application does not limit the specific method for selecting temporary buffer video frames.

[0327] Reference Figure 16 Release condition: Based on the calculated buffer length, temporary buffered video frames are cached in the third buffer queue according to the first-in, first-out principle. Correspondingly, after the third buffer queue is full of temporary buffered video frames, the video frame with the earliest cache time (or the largest difference between its timestamp and the timestamp of the currently mapped video frame) needs to be released. That is, the video frame at position 1 in the diagram is released.

[0328] Additionally, when the application primarily addresses screen stuttering during forward swipes to reverse swipes, or vice versa, to cache as many video frames as possible that might be hit by subsequent touch points, temporary cached video frames with timestamps greater than the currently displayed video frame can be released during forward or reverse swipes (i.e., the video frame at position 2 in the diagram is released). To maximize the number of video frames cached in the third cache queue, the release operation can be performed before the caching operation.

[0329] Therefore, regardless of whether the sliding is forward or backward, it can be done according to... Figure 16 The method shown is for caching temporary cached video frames and releasing temporary cached video frames.

[0330] It's understandable that during the user's dragging of the progress display frames, the third cache queue almost always contains a certain number of temporarily cached video frames. Furthermore, the most recently added temporarily cached video frame is close to and precedes the video frame mapped at the touch point. This allows for improved hit rates of subsequent touch point-mapped video frames in scenarios transitioning from forward to reverse swipes, and vice versa, while minimizing memory usage.

[0331] The order of I-frames, P-frames, and B-frames in a set of video frames in this embodiment, as well as the selected preset cached video frames and temporary cached video frames, are only examples and do not necessarily represent the order or type of video frames (I, P, B) in the examples.

[0332] Of course, in practical applications, there are other release conditions:

[0333] (1) When the video file playback interface is closed, i.e. Figure 2 (b) to Figure 2 Any of the interfaces shown in (e) is closed and returned to Figure 2 When the interface shown in (a) is displayed, it is necessary to release all three types of video frames in the three cache queues to free up memory resources.

[0334] (2) When the time elapsed since the user's release operation during the last frame playback exceeds a threshold (second duration), all three types of video frames in the three buffer queues are released. When releasing video frames from the three buffer queues, the interface displayed on the electronic device can be any interface; for example, the user can display a different interface when starting from... Figure 2 (b) to Figure 2 The threshold is reached only after any of the interfaces shown in (e) are returned to the background, requiring the release of the three types of video frames from the three cache queues. Figure 2 (b) to Figure 2In the case of any interface shown in (e), if the threshold is reached, it is also necessary to release the three types of video frames in the three buffer queues.

[0335] The following is through Figure 17 This describes a timing diagram for rendering and displaying video frames from a cache. Steps S201 to S214 in this timing diagram refer to... Figure 5 As shown.

[0336] S215, the decoder buffer sends rendering instructions for video frames to the renderer.

[0337] In this embodiment, if the cached decoded data includes the decoded data of video frame b, then the decoded data of video frame b is rendered; if the cached decoded data does not include the decoded data of video frame b (YUV data), but includes decoded data whose timestamp and the timestamp of video frame b are less than a threshold, then the decoded data whose timestamp and the timestamp of video frame b are less than the threshold are rendered. In this way, even if the decoded data of video frames relatively close to video frame b (e.g., the one, two, one, or two video frames before or after video frame b in the video file) are cached, there is no need to decode video frame b; instead, the decoded data adjacent to video frame b is rendered, reducing the waiting time for decoding and improving the smoothness of the playback. The threshold can be m times the period of the obtained frame rate of the video file (e.g., 1, 2, 3, 4, etc.), and the value of m can be set according to the situation.

[0338] S216, the renderer renders video frames.

[0339] Understandable, relative to Figure 5 It can directly retrieve video frame b from the cache queue without decoding, thus avoiding stuttering caused by the inability to render and display in time due to decoding time consumption.

[0340] This application provides a frame playback method, including:

[0341] The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file;

[0342] Decode each video frame in the first frame group containing the first video frame in the decoding order;

[0343] During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and the decoding data of the display video frame is cached; some video frames in the first frame group are marked as non-display video frames, and the decoding data of the non-display video frames is cached.

[0344] The second touch point moment is mapped to the second video frame. The second touch point moment and the first touch point moment are different touch point moments in the same sliding operation. The detection time of the second touch point moment is later than the detection time of the first touch point moment. The cached decoding data includes the decoding data of the second video frame or includes decoding data whose difference from the timestamp of the second video frame is less than a threshold.

[0345] The decoded data of the third video frame is obtained from the cache, and the decoded data of the third video frame is rendered and displayed. The third video frame is either the second video frame or a video frame whose timestamp difference with the second video frame is less than a threshold.

[0346] After marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes:

[0347] If the first video frame is successfully rendered and displayed, the decoded data of the first video frame in the first cache queue is marked as invalid. The first cache queue is used to cache the decoded data of the displayed video frame.

[0348] If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame;

[0349] If the first video frame marked as invalid is not marked as a non-display video frame, then the decoded data of the first video frame in the first buffer queue is released.

[0350] After marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes:

[0351] If the first video frame is not rendered and displayed (rendering and display are abandoned), the cache duration of the decoded data of the first video frame in the first cache queue is calculated.

[0352] If the buffer duration of the decoded data of the first video frame is longer than the first duration, the decoded data of the first video frame is marked as invalid.

[0353] Before caching the preset cached video frames from the first frame group in the second cache queue, the method further includes:

[0354] Release the preset cached video frames in the video frame group preceding the second frame group from the second cache queue. The second frame group is a video frame group in the video file that precedes the first frame group.

[0355] Before caching the temporary buffered video frames in the first frame group in the third buffer queue, the method further includes:

[0356] If there is a temporary cached video frame in the third cache queue with a timestamp greater than the timestamp of the first video frame, the temporary cached video frame with a timestamp greater than the timestamp of the first video frame will be released.

[0357] After releasing the temporary cached video frames whose timestamps are greater than the timestamp of the first video frame, the temporary cached video frames in the first frame group are cached in the third cache queue according to the first-in-first-out principle.

[0358] The second frame group is adjacent to the first frame group.

[0359] The step of selecting a portion of video frames from the tail of the first frame group and marking them as preset cached video frames includes:

[0360] Calculate the first number interval when caching the preset cached video frames in the first frame group, the first number interval being related to the frame rate and rendering frame rate of the video file;

[0361] Calculate the first quantity length when caching the preset cached video frames in the first frame group. The first quantity length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate.

[0362] Based on the first quantity interval and the first quantity length, the first cache number of the video frames marked as preset cache in the first frame group is obtained;

[0363] Starting from the end of the first frame group, a portion of video frames are selected and marked as preset cached video frames based on the first cache quantity and the first quantity interval.

[0364] The step of selecting a portion of video frames from adjacent positions of the first video frame in the first frame group and marking them as temporary buffer video frames includes:

[0365] Calculate a second quantity interval when caching temporary cached video frames in the first frame group, the second quantity interval being related to the frame rate and rendering frame rate of the video file;

[0366] During reverse sliding, the second number length of temporary cached video frames in the first frame group is calculated. The second number length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate. Based on the second number interval and the second number length, the second number of cached video frames marked as temporary cached in the first frame group is obtained. The third cache queue caches a maximum of the second number length of video frames. During forward sliding, the second number of cached frames is obtained.

[0367] Starting from the first video frame in the first frame group, based on the second cache quantity and the second quantity interval, a portion of video frames with timestamps less than the timestamp of the first video frame are selected and marked as the temporary cache video frames.

[0368] Additionally, if the video file's playback interface is closed, and the video file's cache queue contains the video file's video frame decoding data, then the video file's video frame decoding data is released.

[0369] After a second duration following the completion of the frame playback operation of the video file, if the video file's cache queue contains the decoded data of the video frame, then the decoded data of the video frame is released.

[0370] After decoding each video frame in the first frame group containing the first video frame in the decoding order, the process further includes:

[0371] In a forward playback scenario, the timing of the next touch point is predicted based on the sliding speed of the sliding operation during frame playback;

[0372] Decode the video frames in the third frame group containing the video frame mapped at the next touch moment according to the decoding order;

[0373] During the decoding of video frames in the third frame group, a portion of video frames are selected from the end of the third frame group and marked as preset cached video frames. The decoding data of the preset cached video frames in the third frame group is cached in the second cache queue.

[0374] This application uses a gallery application to implement video playback using a video playback engine as an example to describe the switching process between normal playback and frame playback. In practical applications, it can also be applied to other applications, such as video editing applications and video playback applications, to achieve the same function using the playback engine.

[0375] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0376] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0377] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0378] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0379] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0380] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0381] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0382] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A frame playback method, characterized in that, include: The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file; Decode each video frame in the first frame group containing the first video frame in the decoding order; During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and the decoding data of the display video frame is cached; some video frames in the first frame group are marked as non-display video frames, and the decoding data of the non-display video frames is cached. The second touch point moment is mapped to the second video frame. The second touch point moment and the first touch point moment are different touch point moments in the same sliding operation. The detection time of the second touch point moment is later than the detection time of the first touch point moment. The cached decoding data includes the decoding data of the second video frame or includes decoding data whose difference from the timestamp of the second video frame is less than a threshold. The decoded data of the third video frame is obtained from the cache, and the decoded data of the third video frame is rendered and displayed. The third video frame is either the second video frame or a video frame whose timestamp difference with the second video frame is less than a threshold.

2. The method as described in claim 1, characterized in that, After marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes: If the first video frame is successfully rendered and displayed, the decoded data of the first video frame in the first cache queue is marked as invalid. The first cache queue is used to cache the decoded data of the displayed video frame. If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame; If the first video frame marked as invalid is not marked as a non-display video frame, then the decoded data of the first video frame in the first buffer queue is released.

3. The method as described in claim 2, characterized in that, After marking the first video frame as a display video frame and caching the decoded data of the display video frame, the method further includes: If the first video frame is not rendered and displayed, calculate the caching duration of the decoded data of the first video frame in the first cache queue; If the buffer duration of the decoded data of the first video frame is longer than the first duration, the decoded data of the first video frame is marked as invalid. If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame; If the first video frame marked as invalid is not marked as a non-display video frame, then the first video frame is released from the first buffer queue.

4. The method according to any one of claims 1 to 3, characterized in that, The step of marking a portion of the video frames in the first frame group as non-display video frames and caching the decoded data of the non-display video frames includes: Select a portion of video frames from the end of the first frame group and mark them as preset cached video frames. Cache the preset cached video frames from the first frame group in the second cache queue. The non-display video frames include the preset cached video frames. A portion of video frames adjacent to the first video frame in the first frame group are selected and marked as temporary cache video frames. The temporary cache video frames in the first frame group are cached in the third cache queue. The non-display video frames include the temporary cache video frames.

5. The method as described in claim 4, characterized in that, Before caching the preset cached video frames from the first frame group in the second cache queue, the method further includes: Release the preset cached video frames in the video frame group preceding the second frame group from the second cache queue. The second frame group is a video frame group in the video file that precedes the first frame group. Before caching the temporary buffered video frames in the first frame group in the third buffer queue, the method further includes: If there is a temporary cached video frame in the third cache queue with a timestamp greater than the timestamp of the first video frame, the temporary cached video frame with a timestamp greater than the timestamp of the first video frame will be released. After releasing the temporary cached video frames whose timestamps are greater than the timestamp of the first video frame, the temporary cached video frames in the first frame group are cached in the third cache queue according to the first-in-first-out principle.

6. The method as described in claim 5, characterized in that, The second frame group is adjacent to the first frame group.

7. The method according to any one of claims 4 to 6, characterized in that, The step of selecting a portion of video frames from the tail of the first frame group and marking them as preset cached video frames includes: Calculate the first number interval when caching the preset cached video frames in the first frame group, the first number interval being related to the frame rate and rendering frame rate of the video file; Calculate the first quantity length when caching the preset cached video frames in the first frame group. The first quantity length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate. Based on the first quantity interval and the first quantity length, the first cache number of the video frames marked as preset cache in the first frame group is obtained; Starting from the end of the first frame group, a portion of video frames are selected and marked as preset cached video frames based on the first cache quantity and the first quantity interval.

8. The method as described in claim 7, characterized in that, The first quantity interval for calculating the preset cached video frames in the first frame group includes: Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

9. The method as described in claim 7 or 8, characterized in that, The first quantity length when calculating the preset cached video frames in the first frame group includes: Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps This indicates the number of video frames decoded per second by the decoder.

10. The method according to any one of claims 7 to 9, characterized in that, The step of obtaining the first buffer quantity of the video frames marked as preset buffers in the first frame group based on the first quantity interval and the first quantity length includes: Among them, D PCF L represents the first quantity interval. PCF The first quantity length is indicated by the `round()` function, which is used to round floating-point numbers to integers.

11. The method according to any one of claims 4 to 10, characterized in that, The step of selecting a portion of video frames from adjacent positions of the first video frame in the first frame group and marking them as temporary buffer video frames includes: Calculate a second quantity interval when caching temporary cached video frames in the first frame group, the second quantity interval being related to the frame rate and rendering frame rate of the video file; During reverse sliding, the second number length of temporary cached video frames in the first frame group is calculated. The second number length is related to the number of video frames in the first frame group, the frame rate of the video file, and the decoding frame rate. Based on the second number interval and the second number length, the second number of cached video frames marked as temporary cached in the first frame group is obtained. The third cache queue caches a maximum of the second number length of video frames. During forward sliding, the second number of cached frames is obtained. Starting from the first video frame in the first frame group, based on the second cache quantity and the second quantity interval, a portion of video frames with timestamps less than the timestamp of the first video frame are selected and marked as the temporary cache video frames.

12. The method as described in claim 11, characterized in that, The second quantity interval for calculating the temporary cached video frames in the first frame group includes: Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation when the frame is played, and the round() function is used to round the floating-point number to an integer. The second number length of the temporarily cached video frames in the first frame group includes: Where GOP is the number of video frames in the first frame group, Video fps Decode indicates the frame rate of a video file. fps The number of video frames decoded per second by the encoder; The step of obtaining the second buffer quantity of video frames marked as temporary buffers in the first frame group based on the second quantity interval and the second quantity length includes: Among them, D TCF L represents the first quantity interval. TCF The first quantity length is indicated by the `round()` function, which is used to round floating-point numbers to integers.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: During the first frame playback after the video file is opened this time, and during subsequent playback, the steps of marking and displaying video frames for caching and marking and not displaying video frames for caching are executed.

14. The method according to any one of claims 1 to 12, characterized in that, The method further includes: If the video file's playback interface is closed after detection, and the video file's cache queue contains the video file's video frame decoding data, then the video file's video frame decoding data is released. After a second duration following the completion of the frame playback operation of the video file, if the video file's cache queue contains the decoded data of the video frame, then the decoded data of the video frame is released.

15. The method according to any one of claims 4 to 12, characterized in that, After decoding each video frame in the first frame group containing the first video frame in the decoding order, the process further includes: In a forward playback scenario, the timing of the next touch point is predicted based on the sliding speed of the sliding operation during frame playback; Decode the video frames in the third frame group containing the video frame mapped at the next touch moment according to the decoding order; During the decoding of video frames in the third frame group, a portion of video frames are selected from the end of the third frame group and marked as preset cached video frames. The decoding data of the preset cached video frames in the third frame group is cached in the second cache queue.

16. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, which, when executed by the one or more processors, causes the electronic device to perform the method as described in any one of claims 1-15.

17. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-15.

18. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-15.

19. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-15.

Images