A frame playing method, an electronic device, a chip system and a storage medium

Abstract

The application provides a frame playing method, an electronic device, a chip system and a storage medium, and relates to the technical field of electronic devices. In the frame playing process, decoded data decoded by GPU hardware is still buffered in the GPU, and the index and timestamp of the decoded data buffered in the GPU are stored in the CPU. When the difference between the timestamp of the video frame to be played subsequently and the timestamp buffered in the CPU is within a threshold, the buffered decoded data can be found in the GPU through the index. The found decoded data is rendered by the GPU, and after the decoded data is rendered, the rendering picture of the decoded data can be displayed. In this way, the process of copying the GPU hardware-decoded data to the CPU buffer and then copying the data from the CPU buffer to the GPU for rendering can be avoided. The frame playing process is simplified, the time delay is lower, and the consumption of CPU memory is also reduced.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a frame playback method, electronic equipment, chip system, and storage medium. Background Technology

[0002] Gallery apps on electronic devices can provide image preview interfaces, allowing users to view various images stored on the device. Gallery apps also provide video playback interfaces, allowing users to play various videos stored on the device.

[0003] The video playback interface can also display the video playback progress, for example, through a progress bar, progress ring, or progress display frame. User drag gestures (press, drag, and release) on the progress bar, progress ring, or progress display frame can trigger the electronic device to play video frames in frame playback mode. However, currently, stuttering often occurs during frame playback due to user drag gestures. Summary of the Invention

[0004] This application provides a frame playback method, electronic device, chip system, and storage medium, which can reduce stuttering during frame playback.

[0005] To achieve the above objectives, the first aspect of this application provides a frame playback method, employing the following technical solution:

[0006] The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file;

[0007] The first video frame is hardware-decoded by the GPU to obtain the first decoded data of the first video frame.

[0008] The first index of the first decoded data in the GPU is stored in the first cache space of the CPU;

[0009] The second touch point moment is mapped to the second video frame. The second touch point moment is the playback moment of the video file where the second touch point position detected during frame playback is mapped to the playback moment of the video file. The detection time of the second touch point position is later than the detection time of the first touch point position.

[0010] If the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, the first index of the first decoded data of the first video frame in the GPU is obtained from the first cache space of the CPU.

[0011] Search for the first decoded data corresponding to the first index in the GPU;

[0012] The first decoded data is rendered by the GPU, and the rendered first decoded data is used for display.

[0013] In this application, the frame playback process is random, that is, the video frame to be displayed is related to the position of the user's finger touch. Even if the touch time is determined based on the position of the user's finger touch, and the touch time is mapped to the video frame, the video frame still needs to go through the decoding and rendering process before it is displayed on the screen. The decoding process doesn't involve immediately decoding any video frame specified. Video frames are decoded in groups, with each frame in its own decoding order, which isn't based on timestamps. Therefore, when a specific video frame (the one mapped to the touch point) needs to be displayed, its decoding order might be the last frame decoded in the group (or later in the decoding order). This could result in the frame not being decoded successfully after a certain time. To avoid stuttering due to long decoding times delaying subsequent rendering, a portion of the video frame's decoding data can be cached during playback. In subsequent frame playback, if the decoding data for the currently displayed video frame has already been cached (or if there is decoding data with a timestamp difference of less than a threshold), the data can be directly retrieved from the cache for rendering, thus avoiding stuttering caused by decoding delays.

[0014] Furthermore, in practical applications, decoding may be performed via the CPU or the GPU. If GPU decoding is used, the decoded data obtained by the GPU needs to be cached in the CPU. Then, when the same video frame needs to be rendered and displayed again, the decoded data cached in the CPU needs to be copied to the GPU for rendering. This method will also cause latency and consume CPU memory. The embodiments of this application can continue to cache the decoded data obtained by the GPU in the GPU, and cache the index of the decoded data cached in the GPU in the CPU. This avoids the process of copying the data hardware-decoded by the GPU to the CPU cache and then copying it from the CPU cache to the GPU. Instead, the cached decoded data is directly looked up in the GPU based on the index for rendering, which reduces latency, saves CPU memory space, and improves the smoothness of the picture when playing the frame.

[0015] As one implementation of the first aspect of this application, the first cache space of the CPU stores the first index and the first timestamp of the first decoded data;

[0016] When the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, obtaining the first index of the first decoded data of the first video frame in the GPU from the first cache space of the CPU includes:

[0017] The first cache space is searched to see if there exists a timestamp whose difference from the second timestamp is less than a threshold.

[0018] If a first timestamp is found whose difference from the second timestamp is less than a threshold, the first index corresponding to the first timestamp is obtained.

[0019] In this application, the CPU can cache the index and timestamp of the decoded data in the GPU. The CPU can determine whether the GPU has cached the corresponding decoded data by using the timestamp. After determining that the GPU has cached the corresponding decoded data by using the timestamp, the GPU can then search for the corresponding decoded data by using the index, thus avoiding invalid searches in the GPU directly.

[0020] As one implementation of the first aspect of this application, after checking whether a timestamp exists in the first cache space and whose difference with the second timestamp is less than a threshold, the method further includes:

[0021] If no first timestamp with a difference of less than a threshold is found, the second video frame is decoded by GPU or CPU to obtain the second decoded data of the second video frame.

[0022] The second decoded data is rendered by the GPU, and the rendered second decoded data is used for display.

[0023] In this application, if it is determined that the decoding data of the video frame mapped to the latest touch point is not cached, it is necessary to decode the mapped video frame to obtain the decoding data for subsequent rendering and display.

[0024] As one implementation of the first aspect of this application, before storing the first index of the first decoded data in the GPU in the first cache space of the CPU, the method further includes:

[0025] Determine whether to render and display the first decoded data;

[0026] If rendering and displaying the first decoded data is abandoned, determine whether to cache the first decoded data;

[0027] It is determined that the first decoded data will be cached.

[0028] In this application, after the video frame mapped at the touch point is decoded, the decoded data of the video frame may be rendered and displayed, or it may not be rendered and displayed. Usually, the GPU space is limited, and the amount of decoded data that can be cached is also limited. Therefore, it is possible to set the current video frame's decoded data to be cached only if the current video frame is not rendered and displayed. In addition, some caching conditions can be set so that the decoded data in the GPU is retained only when the current video frame is determined to need to be cached, and the index and timestamp of the decoded data in the GPU are cached in the CPU, thus saving GPU space.

[0029] As another implementation of the first aspect, after determining whether to render and display the first decoded data, the method further includes:

[0030] If the first decoded data is to be rendered and displayed, the first decoded data is rendered by the GPU, and the rendered first decoded data is used for display.

[0031] Release the first decoded data from the GPU.

[0032] In this application, GPU space is usually limited, and the amount of decoded data that can be cached is also limited. Therefore, it is possible to set the current video frame's decoded data to be released from the GPU when the current video frame is rendered and displayed, thereby saving storage resources in the GPU.

[0033] As one implementation of the first aspect, after determining whether to cache the first decoded data, the method further includes:

[0034] It is determined that the first decoded data will not be cached;

[0035] Release the first decoded data from the GPU.

[0036] As one implementation of the first aspect, determining whether to render and display the first decoded data includes:

[0037] Determine whether there is a third decoded data in the GPU that was decoded later than the first decoded data;

[0038] If, at the moment of rendering execution, there is third decoded data in the GPU that was decoded later than the first decoded data, the rendering and display of the first decoded data is abandoned.

[0039] When the time for rendering is reached, if there is no third decoded data in the GPU that was decoded later than the first decoded data, the first decoded data is rendered and displayed.

[0040] In this application, since the detection frequency of the touch point is higher than the rendering frequency, it is possible that some video frames will be rendered and displayed during the decoding process, while others will be abandoned. Typically, rendering is performed based on the vysnc signal. After receiving the vsync signal, there may be multiple video frames that have been successfully decoded and are waiting to be rendered and displayed. In this case, the decoded data of the latest decoded video frame in the video frames mapped at the touch point time is selected for rendering, and other decoded video frames will be abandoned for rendering and display. In this situation, some video frames will be abandoned for rendering and display, while others will be rendered and displayed.

[0041] As one implementation of the first aspect, determining whether to cache the first decoded data includes:

[0042] Determine whether the first video frame belongs to one of the M video frames at the end of the first frame group, where M is greater than or equal to 1.

[0043] If the first video frame belongs to one of the M video frames at the end of the first frame group, the first decoded data is buffered.

[0044] If the first video frame does not belong to one of the M video frames at the end of the first frame group, the first decoded data is not cached.

[0045] In this application, video frames located at the end of a Group of Frames (GOP) are typically decoded later, resulting in longer decoding times. These frames are most prone to stuttering due to the waiting time for decoding. Therefore, this application chooses to cache these video frames, retaining their decoded data in the GPU while storing the index and timestamp of the retained decoded data in the CPU. Decoding data for other video frames that do not need to be cached in the GPU can be released from the GPU.

[0046] As one implementation of the first aspect, the maximum number of decoded data cached in the GPU is N, where N is greater than or equal to 3; the method further includes:

[0047] When the number of decoded data cached in the GPU is N, if a new decoding instruction is received, the earliest decoded data in the currently cached decoded data in the GPU is deleted, and the first index of the earliest decoded data in the first cache space of the CPU is deleted.

[0048] In this application, the storage resources in the GPU are limited. Therefore, the decoded data cached in the GPU follows a first-in, first-out (FIFO) principle, ensuring that the maximum number of cached decoded data items in the GPU is N. To maintain data consistency, after releasing the decoded data from the GPU, it is also necessary to delete the index and timestamp of the released decoded data from the CPU.

[0049] As one implementation of the first aspect, the second touch point position and the first touch point position are touch point positions in the same swipe gesture.

[0050] As one implementation of the first aspect, the hardware decoding of the first video frame using a GPU includes:

[0051] The GPU decodes each video frame in the first frame group containing the first video frame in the decoding order.

[0052] During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and some video frames in the first frame group are marked as non-display video frames.

[0053] The step of determining whether to cache the first decoded data when rendering and displaying the first decoded data is abandoned includes:

[0054] If rendering and display of the first decoded data is abandoned, determine whether the storage time of the first decoded data in the GPU is greater than the first duration;

[0055] If the storage time of the first decoded data in the GPU exceeds the first duration, the first decoded data is marked as invalid.

[0056] If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame for caching.

[0057] In this application, since the video frames to be displayed are usually successfully decoded, the video frames to be displayed can be marked as display video frames, and the decoding data of the display video frames can be cached in the GPU first. In addition, since the decoding is done in groups, other video frames in the same frame group as the video frames to be displayed may also be decoded when decoding the video frames to be displayed. Therefore, it is also possible to select a portion of the video frames in the same video frame group to cache in the GPU.

[0058] As another implementation of the first aspect, after marking the first decoded data as invalid, it further includes:

[0059] If the first video frame marked as invalid is not marked as a non-display video frame, then the first video frame is released from the GPU.

[0060] As another implementation of the first aspect, after rendering the first decoded data using a GPU, it further includes:

[0061] The first decoded data is marked as invalid.

[0062] As another implementation of the first aspect, marking a portion of the video frames in the first frame group as non-display video frames includes:

[0063] M video frames are selected from the end of the first frame group and marked as preset cached video frames, and the non-display video frames include the preset cached video frames;

[0064] Q video frames are selected from the adjacent positions of the first video frame in the first frame group and marked as temporary buffer video frames. The non-display video frames include the temporary buffer video frames, and Q is greater than or equal to 1.

[0065] In this application, the displayed video frame may also be a non-display video frame that needs to be cached. Therefore, after the displayed video frame is rendered and displayed or after the cache time in the GPU exceeds a certain time, it is necessary to determine whether the displayed video frame is also marked as a non-display video frame. If it is not marked as a non-display video frame, it can be released from the GPU. If it is marked as a non-display video frame, it can continue to be cached in the GPU in the same way as non-display video frames.

[0066] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0067] Thirdly, a chip system is provided, including a processor coupled to a memory, wherein the processor executes a computer program stored in the memory to cause an electronic device to implement the method of any one of the first aspects of this application.

[0068] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0069] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0070] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0071] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0072] Figure 2 A schematic diagram of an interface for a normal playback mode and a frame playback mode provided in an embodiment of this application;

[0073] Figure 3 A schematic diagram illustrating the relationship between the playback order and decoding order of video frames in a video frame group, provided for an embodiment of this application;

[0074] Figure 4 A flowchart illustrating a frame buffering method provided in an embodiment of this application;

[0075] Figure 5 A timing diagram for caching YUV data provided in an embodiment of this application;

[0076] Figure 6 A flowchart illustrating another frame buffering method provided in an embodiment of this application;

[0077] Figure 7 A timing diagram of ID and timestamp for caching YUV data provided in an embodiment of this application;

[0078] Figure 8 This is a schematic diagram illustrating the rendering or abandonment of video frames in a cache queue, as provided in an embodiment of this application.

[0079] Figure 9 A schematic diagram illustrating the marking and failure conditions of a display video frame provided in an embodiment of this application;

[0080] Figure 10 A schematic diagram illustrating the determination of a preset buffered video frame, provided as an embodiment of this application;

[0081] Figure 11 A schematic diagram illustrating another method for determining preset buffered video frames, provided in an embodiment of this application;

[0082] Figure 12 A schematic diagram illustrating the release of a preset buffered video frame, provided as an embodiment of this application;

[0083] Figure 13 A schematic diagram illustrating the determination of temporary buffered video frames provided in an embodiment of this application;

[0084] Figure 14 A schematic diagram illustrating another method for determining temporary buffered video frames, provided as an embodiment of this application;

[0085] Figure 15A schematic diagram illustrating the release of a temporary buffered video frame, provided as an embodiment of this application;

[0086] Figure 16 This is a schematic diagram illustrating a scenario of caching YUV data and using the cached YUV data, provided as an embodiment of this application. Detailed Implementation

[0087] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0088] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0089] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0090] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0091] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0092] This application provides a frame playback method that can be applied to electronic devices. These electronic devices can be tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not limit the specific type of electronic device.

[0093] Figure 1 A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0096] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0097] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback). Touch sensor 180K, also called a "touch panel," can be disposed on display screen 194. Touch sensor 180K and display screen 194 together form a touch screen, also called a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor can transmit the detected touch operation to application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, in a different location than display screen 194.

[0098] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0099] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0100] This application does not specifically limit the structure of the execution subject of a frame playback method. As long as the code recording a frame playback method according to this application is executed, communication can be performed according to the frame playback method provided in this application. For example, the execution subject of a frame playback method provided in this application can be a functional module in an electronic device capable of calling and executing a program, or a communication device applied in an electronic device, such as a chip.

[0101] Users can use the gallery app on their electronic devices to view various photos stored on the device and play various videos stored on the device.

[0102] Reference Figure 2 This is a schematic diagram of an interface for playing videos stored on an electronic device through a gallery application, as provided in an embodiment of this application.

[0103] Reference Figure 2 (a) in the image gallery is a grid of images and videos stored on an electronic device, displayed by the image gallery application of the electronic device. The grid can display images and videos, where the cell containing a video displays the cover and duration of the video, and the cell containing an image displays a thumbnail of the image.

[0104] The grid image can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app, or it can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app and triggers more steps.

[0105] When displaying images and / or videos stored on an electronic device, this grid chart can be presented in various ways, for example, Figure 2 The method shown in (a) displays photos and videos in a photo group in reverse chronological order; in practical applications, gallery applications can also set up multiple object groups, for example, through... Figure 2 The control 20 shown in (a) can view multiple groups in the album: image group, video group, screenshot and screen recording group, etc. Users can also display pictures through the image group in the gallery application, videos through the video group in the gallery application, and screenshots and screen recordings through the screenshot and screen recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the grid diagram shown above.

[0106] This application does not limit the number of user operations required from the system desktop to the display of the grid chart in the gallery application, nor does it limit the way the grid chart displays images and videos.

[0107] Reference Figure 2 (b) in the text represents the user's click. Figure 2 Following the cover of video A in the grid diagram shown in (a), the electronic device displays the video playback interface of video A in response to the operation. The video playback interface includes a video playback area 21 and a progress display area 22.

[0108] The video playback area 21 is used to display the video frame during video playback. After entering the video playback interface from the grid interface, the control 211 is displayed in the video playback area. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the 0th second. When the video is paused, the control 211 is also displayed in the video playback area 21. When the user clicks the control 211, the electronic device responds to the click operation on the control 211 and starts playing the video frame in the video playback area 21 from the current paused moment.

[0109] The progress display area 22 is used to display the video playback progress. The progress display area 22 includes a progress display frame 221 and a progress indicator line 222. The position of the progress indicator line 222 can remain unchanged. The user can drag the progress display frame 221 located below the progress indicator line 222 so that the progress display frame 221 moves left or right without moving the progress indicator line 222. The position of the progress indicator line 222 above the progress display frame indicates the current playback progress of the video.

[0110] As an example, the total playback duration of the current video is 15 seconds, and the total length of the current video's progress display frames is represented by L pixels. Based on these two parameters, we can obtain the playback duration corresponding to a unit length (or the length corresponding to a unit of time). For example, divide the playback duration 15 by the total length L. Therefore, if the leftmost edge of the progress display frame (the position corresponding to the first pixel) corresponds to second 0 of the video, then different positions of the progress indicator line on the progress display frame will correspond to different playback times. For example, if the distance between the position of the progress indicator line on the progress display frame and the leftmost edge of the progress display frame is X, then the playback time corresponding to the progress indicator line is: X*15 / L.

[0111] For ease of subsequent description, the embodiments in this application are described in the manner of "the time corresponding to the progress indicator line" or "the contact time".

[0112] Due to space limitations in the progress display area 22, only a portion of the video frames or a portion of the video frames from the progress display frame 221 can be displayed in the progress display area 22. For example... Figure 2 As shown in (b), the progress display frame 221 displayed in the progress display area 22 includes: thumbnails of three complete video frames and a partial area of ​​a thumbnail of a video frame.

[0113] The number of thumbnails displayed in the progress bar is usually related to the video length. Generally, the longer the video, the more thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames), and the shorter the video, the fewer thumbnails are used to display the video's playback progress (thumbs up in the progress bar frames).

[0114] Typically, the video playback area 21 displays the video frame corresponding to the time indicated by the progress indicator line 222 in the progress display area 22.

[0115] In a specific implementation, there is a situation where the time corresponding to the progress indicator line 222 is the time between the playback times of two adjacent video frames, that is, there is no video frame in the video file corresponding to the time of the progress indicator line 222.

[0116] As an example, in a video frame sorted by playback time (denoted as timestamp) from smallest to largest, the playback time of the i-th video frame is ti, and the playback time of the (i+1)-th video frame is t(i+1); the time t corresponding to the progress indicator line 222 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval to the time t corresponding to the progress indicator line 221 can be taken as the video frame corresponding to the time of the progress indicator line 222.

[0117] Touch point time: The time when the progress indicator line points to is determined based on the touch point position, indicating the moment when the user drags their finger to the current touch point position and expects to display the image from the video file at that moment; for example, if the touch point time is 1015ms, it means that the user expects to display the image from the 1015ms position in the video file.

[0118] System time at the moment of contact: The system time at which the contact moment is acquired. For example, if the contact moment (at 1015ms) is acquired at 13:54:24 on the electronic device, then the system time at the contact moment (at 1015ms) is 13:54:24.

[0119] The video frame mapped to the touch point: The video frame in the video file whose timestamp is the smallest difference from 1015ms, which is the 1016ms video frame. For example, the video frame mapped to the touch point (1015ms) is the video frame with timestamp 1016.67ms.

[0120] It is understandable that if there is no video frame in the video file with the same timestamp and the time corresponding to the progress indicator line, the video frame preceding time t (the video frame corresponding to ti) corresponding to time t of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 221; or the video frame following time t (the video frame corresponding to t(i+1)) corresponding to time t of progress indicator line 222 can be used as the video frame corresponding to time t of progress indicator line 222. In practical applications, there are many ways to display the playback progress in the progress display area 22. For example, the video playback progress can be displayed as a progress bar or as a progress ring.

[0121] This application provides two playback modes for video playback: normal playback and frame-by-frame playback. The embodiments in this application only illustrate... Figure 2 This serves as an example to illustrate the difference between normal playback and frame playback.

[0122] Reference Figure 2 (c) in the text represents the user's click. Figure 2 After the control 211 (which can also be referred to as the playback control) in the interface shown in (b) is displayed, the electronic device responds to the operation by displaying an interface diagram during the playback of video A. This interface diagram indicates that the video A has been played to the 5th second, and the video frame corresponding to the 5th second is displayed in the video playback area 21. At the same time, the progress display frame 221 in the progress display area 22 moves its position so that the time corresponding to the progress indicator line 222 is the 5th second. Among them, from Figure 2 (b) in Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0123] A video file consists of multiple video frames, each with a timestamp. The timestamp records the playback time of each video frame during the video file's playback process. For example, arranged by timestamp from smallest to largest, the video frames in the file are: Video Frame 0 (timestamp t0), Video Frame 1 (timestamp t1), Video Frame 2 (timestamp t2), Video Frame 3 (timestamp t3), Video Frame 4 (timestamp t4), Video Frame 5 (timestamp t5), Video Frame 6 (timestamp t6), Video Frame 7 (timestamp t7)... Normal playback involves displaying each video frame sequentially in the playback area according to its corresponding timestamp, from smallest to largest.

[0124] As an example of a normal playback process, video frame 0 is displayed in video playback area 21 at time t0 after playback begins; video frame 1 is displayed in video playback area 21 at time t1 after playback begins; video frame 2 is displayed in video playback area 21 at time t2 after playback begins; video frame 3 is displayed in video playback area 21 at time t3 after playback begins; video frame 4 is displayed in video playback area 21 at time t4 after playback begins; video frame 5 is displayed in video playback area 21 at time t5 after playback begins; video frame 6 is displayed in video playback area 21 at time t6 after playback begins; video frame 7 is displayed in video playback area 21 at time t7 after playback begins…

[0125] from Figure 2 (b) in Figure 2 (c) shows the normal playback process of video A: starting from the 0th second of video A, each video frame is displayed sequentially according to the playback time indicated by the timestamp; Figure 2 (c) in the diagram is a schematic of the interface displaying the video frame up to the 5th second.

[0126] Reference Figure 2 In (d), the user drags... Figure 2 The progress display frame of the interface shown in (c) is in response to... Figure 2 As shown in (d) above, the electronic device displays the corresponding screen in the video playback area for the drag operation.

[0127] As an example, refer to Figure 2 In step (e), when the user drags the progress bar to display a frame such that the progress indicator line corresponds to the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be a video frame in video A with a timestamp of 14 seconds; it can also be a video frame with a timestamp closest to 14 seconds; it can also be a video frame before 14 seconds whose time interval between timestamp and 14 seconds is closest; or it can be a video frame after 14 seconds whose time interval between timestamp and 14 seconds is closest.

[0128] from Figure 2 (d) in Figure 2 The playback process shown in (e) is the frame playback process of the video.

[0129] As the user drags the progress display frame 221, a relative position is formed between the progress indicator line 222 and the progress display frame 221. The position of the progress indicator line 222 on the progress display frame 221 is related to the time corresponding to the progress indicator line 222.

[0130] In the specific implementation, assuming that the time indicated by the progress indicator line 222 is t1 before the user moves in the area where the progress display frame 221 is located on the screen. When the user moves left or right in the area where the progress display frame is located, the electronic device can detect the user's real-time touch position at certain time intervals. The electronic device determines the user's movement direction and distance based on the continuous real-time touch positions. The electronic device determines the time change Δt based on the user's movement distance; and determines the calculation method for the touch time t2 based on the user's movement direction. For example, if the movement direction is left, the touch time t2 is obtained by subtracting the time change Δt from the starting time t1; if the movement direction is right, the touch time t2 is obtained by adding the time change Δt to the starting time t1. After obtaining the touch time t2, the electronic device searches for the video frame corresponding to the touch time t2 and displays the video frame corresponding to the touch time t2 in the video playback area.

[0131] It's understandable that the time or touch point time corresponding to the progress indicator line is not the system time, but rather the playback time in the video file where the touch point position is mapped. The video frame corresponding to touch point time t2 can be found in the detailed explanation of "the video frame corresponding to the 14th second". The video frame corresponding to touch point time t2 can also be understood as the video frame mapped to touch point time t2.

[0132] On the other hand, the electronic device obtains the movement direction of the progress display frame based on the movement direction determined by the real-time touch point position, and determines the movement distance of the progress display frame based on the movement distance of the real-time touch point position; the electronic device moves the progress display frame in the progress display area according to the movement direction and movement distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0133] As another example, the electronic device also determines the moving direction (left if t2 is greater than t1, right if t2 is less than t1) and moving distance of the progress display frame based on the start time t1 and the touch time t2; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0134] It should be noted that the above process is only an example. In actual applications, there are multiple ways to determine the touch point time corresponding to the progress indicator line 222, the position of the progress display frame 221, and the video frame displayed in the video playback area 21 when the user drags the progress display frame 121.

[0135] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. It may move to the left, it may move to the right, it may be dragged quickly, or it may be dragged slowly. Accordingly, the video frame images displayed in the video playback area 21 have randomness.

[0136] As an example, before the user moves within the area of ​​the progress display frame on the screen, the starting time indicated by the progress indicator is t1. During the user's dragging of the progress display frame, the electronic device detects the user's real-time touch position at regular time intervals, thus obtaining multiple consecutive time periods corresponding to the real-time times: t2, t3, t7, t4, and t3. In the video playback area, video frames are played in the following order: starting with the video frame corresponding to time t1, the video frames corresponding to time t2, t3, t7, t4, and t3 are played sequentially.

[0137] It can be understood that the process of a user clicking control 21 to trigger the electronic device to play video in the video playback area is normal playback; the process of a user dragging the progress display frame to trigger the electronic device to display the video frame corresponding to the progress indicator line in the video playback area is frame playback.

[0138] Figure 2 The example shown is intended to illustrate the difference between normal playback and frame playback, and does not imply that the interface during normal playback and frame playback must be identical. Figure 2 Consistent with the above.

[0139] In the above embodiments, whether it is normal playback or frame playback, the video frames need to be decoded before they are displayed in the video playback area. Usually, the video frames are decoded in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a keyframe (I-frame), at least one forward reference frame (P-frame), and at least two bidirectional reference frames (B-frames).

[0140] Reference Figure 3 This is a schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application. A set of video frames can also be referred to as a group of pictures (GOP).

[0141] Each group of video frames, in playback order, includes: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. Among these, the I-frame, as a keyframe, can be decoded to obtain a complete image, but decoding is slower. P-frames record changes relative to the previous frame (not a B-frame); information from the previous frame is required to decode the P-frame. Similarly, information from previous P-frames (which may also be I-frames) and subsequent P-frames is needed to decode the B-frame. P-frames and B-frames improve video smoothness while reducing the file size.

[0142] Whether it's an I-frame, P-frame, or B-frame, they are all video frames. When decoding a video frame at a specific moment, you don't directly decode the video frame at that moment. Instead, you decode the group of video frames that the video frame at that moment belongs to. When decoding video frames within a group of video frames, you decode them one by one in the decoding order.

[0143] As an example, if we need to decode the video frame at time t1, and the video frame at time t1 is the 6th video frame in the playback order of a group of video frames: frame B; then the decoding order is:

[0144] First decoding: the video frame played in the first playback order: I-frame;

[0145] The second decoding: the 4th video frame in playback order: P-frame;

[0146] The third decoding step: the second video frame played in the playback order: frame B;

[0147] The fourth decoding: the third video frame in playback order: frame B;

[0148] The fifth decoding: the 7th video frame in playback order: P-frame;

[0149] The sixth decoding: the 5th video frame in playback order: frame B;

[0150] The seventh decoding: the 6th video frame in playback order: frame B;

[0151] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding operation to decode the video frame at time t1.

[0152] Currently, in normal playback scenarios, the video frames in each video frame group are parsed, decoded, rendered, and displayed sequentially according to the playback order. Of course, when decoding the video frames in each video frame group, the process is as follows: Figure 3The decoding order shown decodes each video frame in a video frame group sequentially. Normally, in a continuous playback scenario, when displaying frames from the previous video frame group, all frames in the next video frame group have already been decoded. However, when switching from frame playback mode to normal playback mode, it's necessary to determine the video frame group containing the latest frame based on its playback time, and then decode the video frames sequentially according to the above decoding order.

[0153] In a frame playback scenario, the process of a user dragging the progress bar to display a frame is uncertain. The user may drag the progress bar to the left, or to the right, or drag it quickly or slowly. Consequently, the video frame images displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display a frame, the electronic device needs to determine the moment the progress indicator line points based on the real-time detected touch point position, and then decode the video frames sequentially according to the above decoding order.

[0154] However, since the user's dragging process may be fast or slow, if the above embodiment is followed exactly: each time a touch point is detected, the touch point is mapped to a video frame, the video frame is decoded (in groups and in the order of the group decoding), and then rendered and displayed, the decoding needs to be done in groups, so the waiting time for decoding may be long, so screen stuttering often occurs.

[0155] Currently, in order to improve the smoothness of the picture in frame playback scenarios, the YUV data decoded during previous frame playback or normal playback can be stored in the cache queue in the CPU memory for caching. During subsequent frame playback, the relevant YUV data can be retrieved from the cache and rendered directly. Since there is no need to wait for the decoding process, the phenomenon of picture stuttering is reduced.

[0156] In practical applications, video decoding can be implemented by either the CPU or the GPU. For example, video frame encoding and decoding tasks can be performed using software on the CPU (FFmpeg, libx264, etc.), offering greater flexibility and supporting a wider range of decoding formats. Alternatively, video decoding can be performed using dedicated hardware on the GPU (NVIDIA's NVDEC, AMD's UVD or UVC, Intel's Quick Sync Video, etc.), which improves decoding efficiency and reduces the CPU's workload.

[0157] Reference Figure 4 The diagram illustrates a process for caching decoded data according to an embodiment of this application. In this process, the decoded data is cached in the CPU.

[0158] OnOutputBufferAvailable() and dequeueOutputBuffer() are two key functions in the MediaCodec API used to process decoded data streams.

[0159] `OnOutputBufferAvailable()` is one of the MediaCodec callback functions. It is called when the decoder finishes decoding and is ready to output data. Its main purpose is to notify the application layer that new decoded data is available for processing. This notification mechanism is primarily used for status notifications.

[0160] `DequeueOutputBuffer()` is the primary method used by the application layer to retrieve decoded data. When `onOutputBufferAvailable()` is called, the application layer can obtain the decoded data by calling `dequeueOutputBuffer()`. This is the method called by the application layer to actually retrieve the decoded data.

[0161] After decoding by either the CPU or GPU, the decoded data (YUV data) can be sent to the output queue. The cache pointer of the decoded data in the GPU can be obtained based on the outputid (or id) of the decoded data in the output queue. The YUV data can be retrieved based on the cache pointer and cached in the cache space of the CPU memory. After being cached in the CPU memory, the YUV data in the output queue is released.

[0162] Subsequently, the YUV data in the CPU memory can be used to perform rendering or to abandon rendering. Of course, the rendering work is performed in the GPU, so it is also necessary to copy data from the CPU memory cache to the GPU renderer.

[0163] Of course, YUV data can also be cached for a longer period of time. When the difference between the timestamp of the video frame mapped to the subsequent touch point position and the timestamp of the decoded data in the cache is less than a threshold, the decoded data with a timestamp less than the threshold mapped to the current touch point position can be retrieved from the cache and rendered directly.

[0164] Reference Figure 5As shown, if decoding is performed using the GPU, the decoded YUV data needs to be copied to the CPU for caching. During rendering, it also needs to be copied from the CPU to the GPU, resulting in low efficiency. Furthermore, the YUV data of video frames cached in the CPU also consumes CPU memory, leading to CPU memory resource constraints. Moreover, even when the decoded data of subsequent touchpoint positions mapped to video frames is already cached in the CPU, it still needs to be copied from the CPU to the GPU for rendering, complicating the process.

[0165] Reference Figure 6 This application provides a caching method that, after decoding by the GPU, no longer performs the caching of YUV data from the GPU to the CPU. Instead, the ID and timestamp of the YUV data decoded and output by the GPU are cached in the CPU. When rendering after decoding, the CPU sends the ID of the YUV data to the renderer. The renderer retrieves the YUV data from the GPU's output queue based on the ID for rendering. This method is more efficient in transmitting YUV data and has lower latency. Furthermore, it eliminates the need to cache YUV data in the CPU, thus saving CPU memory. On the other hand, when the decoded data of the video frame mapped to the subsequent touch position during frame playback is already cached in the GPU, it is no longer necessary to copy it from the CPU to the GPU for rendering. Instead, the corresponding decoded data is retrieved directly from the GPU for rendering, simplifying the process.

[0166] Of course, GPU cache space is also limited, so there are some other limitations to caching YUV data in GPU. For example, a maximum of 12 frames can be cached (for example only), and it is best to keep it between 3 and 5 frames (also for example only). Since the number of video frames cached is relatively small, its application is limited to scenarios with short caching times.

[0167] Reference Figure 7 This is a caching method for YUV data implemented in this application embodiment based on a scenario with a short caching time.

[0168] After obtaining the decoded data through OnOutputBufferAvailable() or dequeueOutputBuffer(), determine whether the currently decoded YUV data needs to be rendered and displayed. If it needs to be rendered and displayed, render and display the decoded data and release the YUV data cached in the GPU. If it does not need to be rendered and displayed, determine whether the video frame needs to be cached. If it needs to be cached, cache the ID and timestamp of the YUV data in the GPU in the CPU. If it does not need to be cached, release the YUV data in the GPU.

[0169] The following describes in detail whether the content is rendered and displayed, and whether it is cached.

[0170] In this embodiment, with a screen refresh rate of 60Hz, the electronic device displays 16.6ms per second. Since the detection frequency for touch moments is typically at least 120Hz, not every video frame mapped to a touch moment is rendered. Typically, if multiple decoded video frames exist in the first buffer queue (also known as the rendering queue), the most recently added frame may be selected for rendering. This results in some video frames in the first buffer queue being rendered while others are not.

[0171] This application provides an example of a situation that causes video frames in the first buffer queue to not be rendered and displayed.

[0172] Reference Figure 8 As shown, after receiving vsync1, the latest decoded video frame b is retrieved from the first buffer queue, and video frame b is rendered. Video frame a is abandoned and rendered.

[0173] After receiving vsync2, retrieve the latest decoded video frame c from the first buffer queue and start rendering video frame c.

[0174] Upon receiving vsync3, the newly sent decoded video frame e is retrieved from the first buffer queue, and video frame e is rendered. Video frame d is discarded for rendering and display.

[0175] When rendering video frames, the renderer does not render a video frame every time the decoded data of a video frame that needs to be rendered is sent to the first buffer queue. Instead, it retrieves the latest decoded video frame data from the first buffer queue according to a specific rendering frame rate. This may result in some video frames in the first buffer queue being successfully rendered and displayed, while others may be abandoned and not rendered.

[0176] Of course, the first buffer queue in this embodiment can be a queue formed after the video frames mapped at the touch point are successfully decoded. This first buffer queue can be maintained in the GPU or in the CPU. If it is maintained in the CPU, the first buffer queue contains the ID and timestamp of the decoded video frame data.

[0177] The following describes in detail whether the decoded data of the video frame is cached.

[0178] In this embodiment of the application, the video frames that may be cached are divided into two main categories: display video frames and non-display video frames;

[0179] The displayed video frames are the video frames mapped to the touch points in the above embodiments. After these video frames are successfully decoded, their IDs and timestamps are sent to the first buffer queue. The first buffer queue can be set to cache the displayed video frames.

[0180] Non-display video frames include two types: preset buffer video frames and temporary buffer video frames. Preset buffer video frames are those at the end of the GOP (Group of Pictures) containing the display video frames; a second buffer queue can be set up to cache their IDs and timestamps. Temporary buffer video frames are those adjacent to the display video frames; a third buffer queue can be set up to cache their IDs and timestamps. All three types of buffer queues can be configured within the CPU.

[0181] For each possible cached video frame, there are marking conditions (i.e. conditions for determining it to be a cached video frame) and release conditions. The marking conditions and release conditions are described below for each type of cached video frame.

[0182] (a) Display the marking and release conditions of video frames.

[0183] Reference Figure 9 The diagram shows the marking and release conditions for displaying video frames provided in this embodiment of the application.

[0184] A1, the GPU successfully decoded and mapped video frame 1.

[0185] In this embodiment of the application, each touch point can be mapped to a video frame at any time. As mentioned above, the mapped video frame may be any frame in the GOP group.

[0186] In addition, when the GPU decodes, it decodes each video frame in the video frame containing the mapped video frame 1.

[0187] A2, the CPU marks the mapped video frame 1 as the display video frame 1, and sends the ID and timestamp of the decoded data of the display video frame 1 into the cache queue 1 (first cache queue).

[0188] In this embodiment, cache queue 1 is used to cache the timestamps and IDs of the decoded data of the displayed video frames. According to A1 and A2, in this embodiment, during the user's dragging of the progress bar to display the frames, the touch moments detected by the electronic device are filtered, and each touch moment can be mapped to a corresponding video frame. After these video frames are successfully decoded, they can be determined as video frames that need to be rendered and displayed, and these video frames that need to be rendered and displayed are marked as display video frames.

[0189] A3, determine whether video frame 1 has been successfully rendered and displayed.

[0190] In this embodiment of the application, according to Figure 8It is understandable that some video frames sent to buffer queue 1 can be successfully rendered and displayed, while others cannot.

[0191] A4, if video frame 1 is successfully rendered and displayed, set video frame 1 to enter an invalid state.

[0192] A5, if the display video frame 1 is not successfully rendered (e.g., rendering is abandoned), monitor the cache duration of the timestamp and id of the display video frame 1 in the cache queue 1.

[0193] A6. Determine whether the cache duration of the timestamp and ID of the displayed video frame 1 in the cache queue 1 is greater than the threshold (first duration).

[0194] If the cache duration of the timestamp and ID of the displayed video frame 1 in the cache queue 1 is not greater than the threshold, continue to monitor the cache duration of the timestamp and ID of the displayed video frame 1 in the cache queue 1.

[0195] A7, if the cache duration of the ID and timestamp of video frame 1 in cache queue 1 exceeds the threshold, set the display of video frame 1 to enter an invalid state.

[0196] A8. After displaying video frame 1 enters the invalid state, check whether displaying video frame 1 is also marked as a non-displaying video frame.

[0197] As mentioned earlier, the video frames mapped to the touch points within the same Group of Pictures (GOP) are marked as display video frames. Simultaneously, some video frames at the end of the GOP can be marked as non-display video frames. Therefore, a video frame may be marked as both a display and a non-display video frame. Of course, the video frame may also be marked as either a display or a non-display video frame, or it may not be marked as either.

[0198] A9, if video frame 1 is displayed but also marked as a non-display video frame, then video frame 1 will be converted to a non-display video frame.

[0199] When video frame 1 is displayed as a non-display video frame, its corresponding ID and timestamp are sent to the cache queue corresponding to the non-display video frame. Then, the ID and timestamp of video frame 1 are released from cache queue 1, and the YUV data of video frame 1 in the GPU are also released.

[0200] A10 will release video frame 1 from cache queue 1 and the GPU if video frame 1 is not marked as a non-display video frame.

[0201] (ii) Marking and releasing conditions for pre-buffered video frames.

[0202] Marking criteria: At the end of the GOP group containing the currently being decoded mapped video frames, select a portion of video frames and mark them as preset buffered video frames according to the length of the buffer region Lpcf, the number of buffers Npcf, and the buffer interval Dpcf.

[0203] As an example, you can directly set the cache quantity and cache quantity interval. For example, you can set the cache quantity to 1 or 2, and set the cache quantity interval to 1, 2, 3 or 4, etc.

[0204] Of course, it can also be calculated based on other information. The process of calculating based on other information is described in detail below.

[0205] (1) First, calculate the number of intervals D between the selected preset buffer video frames. PCF .

[0206]

[0207] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0208] The video frame rate is fixed; for example, the video file itself may be 60fps (60 frames per second when playing the video in normal playback mode), 90fps, or 120fps. The rendering display frame rate represents the real-time rendering frame rate of the electronic device. Even if the refresh rate of the electronic device is 60Hz, the real-time rendering frame rate may not be stable at 60fps, but may fluctuate due to factors such as decoding speed. In this embodiment, the rendering frame rate is the frame rate detected in real time (e.g., it can be detected once per second, or once every 500ms, etc.), and the scrolling speed is the number of pixels scrolled per second.

[0209] As an example, during the forward scrolling process, if the video file has a frame rate of 120fps and the real-time rendering display frame rate is 60fps, then the buffer quantity interval is 2; when the video frame rate is 60fps and the real-time rendering display frame rate is 59fps, the buffer quantity interval is 1.

[0210] (2) Calculate the number and length L of the preset buffer video frame region. PCF .

[0211]

[0212] Where GOP represents the number of video frames in a group of video frames, Video fpsDecode indicates the frame rate of a video file. fps This indicates the number of video frames decoded per second by the decoder. It is a real-time value of the current device's decoding rate, calculated from decoder statistics.

[0213] As an example, with a video frame rate of 60fps, a decoding rate of 120fps, and a GOP group containing 12 video frames, the length is 4 frames.

[0214] (3) Calculate the cache size N based on the length and interval of the cache. PCF .

[0215]

[0216] Specifically, at least one video frame is marked at the end of the GOP as a pre-buffered video frame. The number of buffers is determined to be 2 based on the calculated length (e.g., 4 frames) and the number interval (e.g., 2).

[0217] Reference Figure 10 As shown, after determining the above parameters, during normal playback or frame playback, the video frames marked as preset cached video frames are determined according to the number of lengths, number of intervals, and number of buffers.

[0218] Reference Figure 11 In order to follow Figure 11 As an example of the method shown, continuing with a length of 4, an interval of 2, and a buffer size of 2, we take the selection of a portion of video frames at the end of the GOP as preset buffer video frames. Starting from the end of the GOP and counting 4 video frames from the length of the length, the fourth-to-last video frame (the circled P frame in the figure) is selected as the first preset buffer video frame. Then, according to the interval of 2, the second-to-last video frame (the circled B frame in the figure) is selected as the second preset buffer video frame.

[0219] In addition, in practical applications, one or more GOP groups can be decoded in advance. For example, in practical applications, the positions of the next n (e.g., 1, 2, 3, etc.) touch points can be predicted based on the current touch point position and sliding speed, and the touch point time can be calculated for each of the predicted next n touch point positions; or the touch point time can be directly predicted based on the current touch point time and sliding speed; the predicted next GOP group is obtained based on the predicted touch point time, and the predicted next GOP group can be decoded in advance. When decoding the predicted GOP group in advance, a portion of video frames can be selected as pre-buffered video frames for caching.

[0220] In practical applications, it is also possible to configure the pre-decoding of video frames of the GOP group to be performed during normal playback or forward frame playback.

[0221] Release condition: If the pre-buffered video frames are no longer in the GOP group of the currently displayed video frames or the previous GOP group (based on the order of GOP groups in the video file), the pre-buffered video frames in that group will be released from the GPU, and the corresponding IDs and timestamps will be released from the CPU.

[0222] In this embodiment of the application, there is also a limit to the caching duration of each group of preset cached video frames.

[0223] For example, if the determined display video frames are distributed in multiple consecutive video frame groups, and the video frame group containing the cached preset video frames is the i-th video frame group, when the currently determined display video frame is a video frame in the (i+1)-th video frame group, the preset cached video frames in the i-th video frame group can still continue to be cached. When the currently determined display video frame is a video frame in the (i+2)-th video frame group, the video frames in the i-th video frame group do not belong to the (i+2)-th video frame group, nor are they located in the previous video frame group of the (i+2)-th video frame group. Therefore, the preset cached video frames in the i-th video frame group can be deleted.

[0224] In addition, in practical applications, the video frame group to which the determined display video frame is located may not be continuous.

[0225] Reference Figure 12 Assume that the determined consecutive display video frames are located in the i-th video frame group, the (i+2)-th video frame group, and the (i+4)-th video frame, respectively.

[0226] The video frame group containing the cached preset video frame is the i-th video frame group. When the currently determined display video frame is a video frame in the (i+2)-th video frame group, the i-th video frame group is not the previous video frame group of the current (i+2)-th video frame group, and the preset cached video frame in the i-th video frame group can be released.

[0227] It can be understood that the cached preset video frames are always located near the currently displayed video frame or the currently decoded video frame (in the same video frame group or adjacent video frame groups). Preset cached video frames that are not located nearby will be released. This setting can make better use of the application's forward sliding and then reverse sliding. When sliding in reverse, the cached video frames at the end of the same video frame group or the previous video frame group can be used.

[0228] Of course, when applying this to a scenario where a reverse swipe is followed by a forward swipe, the release condition needs to be set as follows: release the pre-buffered video frames in a group if the pre-buffered video frames are no longer in the GOP group of the currently displayed video frame or in the next GOP group. Similarly, it can be set to pre-decode the video frames of the preceding GOP group when playing reverse frames.

[0229] (III) Marking and releasing conditions for temporary buffered video frames:

[0230] Marking criteria: In the vicinity of the currently being decoded mapped video frame, select a portion of video frames as temporary buffer video frames according to the buffer area length Ltcf, the number of buffers Ntcf, and the buffer interval Dtcf.

[0231] As an example, you can directly set the cache quantity and cache quantity interval. For example, you can set the cache quantity to 1 or 2, and set the cache quantity interval to 1, 2, 3 or 4, etc.

[0232] Of course, it can also be calculated based on other information. The process of calculating based on other information is described in detail below.

[0233] (1) First, calculate the number of intervals D between the selected preset buffer video frames. TCF .

[0234]

[0235] Among them, Video fps Indicates the frame rate of the video file, Render fps The rendering frame rate, speed represents the sliding speed of the sliding operation during frame playback, and the round() function is used to round floating-point numbers to integers.

[0236] The video file has a fixed frame rate. For example, the video file itself may be 60fps (60 frames per second when playing the video in normal playback mode), 90fps, or 120fps. The rendering display frame rate represents the real-time rendering frame rate of the electronic device. Even if the refresh rate of the electronic device is 60Hz, the real-time rendering frame rate may not be stable at 60fps, but may fluctuate due to factors such as decoding speed. In this embodiment, the rendering frame rate is the frame rate detected in real time (e.g., it can be detected once per second, or once every 500ms, etc.), and the scrolling speed is the number of pixels scrolled per second.

[0237] As an example, during the forward scrolling process, if the video file has a frame rate of 120fps and the real-time rendering display frame rate is 60fps, then the buffer quantity interval is 2; when the video frame rate is 60fps and the real-time rendering display frame rate is 59fps, the buffer quantity interval is 1.

[0238] It is understandable that the interval between the number of temporary buffered video frames and the interval between the number of preset buffered video frames can be the same. Of course, in practical applications, they can also be set to be different.

[0239] When swiping forward, the cache size can be set to 2. When swiping backward, the cache area size can be calculated first, and then the cache size can be obtained based on the cache area size and the cache size interval.

[0240] (2) Calculate the cache area size L TCF .

[0241]

[0242] Where GOP represents the number of video frames in a group of video frames, Video fps Decode indicates the frame rate of a video file. fps The number of video frames the decoder decodes per second.

[0243] As an example, with a video file frame rate of 60fps, a decoding rate of 120fps, and a GOP group containing 12 video frames, the length is 4 frames.

[0244] (3) Calculate the cache size N TCF .

[0245]

[0246] Specifically, during forward sliding, two video frames are selected as temporary buffer video frames in the region adjacent to the mapped video frame; during reverse sliding, at least one video frame is selected as a temporary buffer video frame in the region adjacent to the mapped video frame.

[0247] Reference Figure 13 As shown, as an example of forward swiping, the number of buffers is determined to be 2, and the two video frames before the mapped video frame are marked as temporary buffer video frames. In this way, in the scenario of switching from forward swiping to reverse swiping, the video frame mapped at the touch point is likely to be in the buffer, thereby reducing the time waiting for decoding data and improving the smoothness of the picture when playing the frame.

[0248] Reference Figure 14 As an example of reverse sliding, the determined buffer size is 4. The 4 video frames before the mapped video frame are marked as temporary buffer video frames, with an interval of 2. Also, a portion of video frames (circled in the diagram) are selected from the video frames before the mapped video frame as temporary buffer video frames.

[0249] It is understandable that, regardless of whether the current swipe is forward or reverse, one or more video frames preceding the mapped video frame will be used as temporary cached video frames. This ensures that the video frames mapped at subsequent touch points may be in the cache during the transition from forward to reverse swipe or reverse swipe scenarios, thus resolving the stuttering issue during the transition.

[0250] Of course, in practical applications, to broaden the application scenarios, a subset of video frames can be selected before mapping video frames, and another subset can be selected as temporary buffer video frames after mapping video frames. This application does not limit the specific method for selecting temporary buffer video frames.

[0251] Reference Figure 15 Release condition: Based on the calculated buffer length, temporary buffered video frames are cached in the third buffer queue according to the first-in, first-out principle. Correspondingly, after the third buffer queue is full of temporary buffered video frames, the video frame with the earliest cache time (or the largest difference between its timestamp and the timestamp of the currently mapped video frame) needs to be released. That is, the video frame at position 1 in the diagram is released.

[0252] Additionally, when the application primarily addresses screen stuttering during forward swipes to reverse swipes, or vice versa, to cache as many video frames as possible that might be hit by subsequent touch points, temporary cached video frames with timestamps greater than the currently displayed video frame can be released during forward or reverse swipes (i.e., the video frame at position 2 in the diagram is released). To maximize the number of video frames cached in the third cache queue, the release operation can be performed before the caching operation.

[0253] Therefore, regardless of whether the sliding is forward or backward, it can be done according to... Figure 16 The method shown is for caching temporary cached video frames and releasing temporary cached video frames.

[0254] It's understandable that during the user's dragging of the progress display frames, the third cache queue almost always contains a certain number of temporarily cached video frames. Furthermore, the most recently added temporarily cached video frame is close to and precedes the video frame mapped at the touch point. This allows for improved hit rates of subsequent touch point-mapped video frames in scenarios transitioning from forward to reverse swipes, and vice versa, while minimizing memory usage.

[0255] In practical applications, it is also possible to configure the system to cache only one type or two types of video frames. This application does not impose any restrictions on which specific video frames are cached.

[0256] Due to the limitation on the amount of YUV data cached in the GPU, the maximum number of cached data in each cache queue of the CPU for caching IDs and timestamps can be preset, and each cache queue can follow the first-in-first-out principle. This application embodiment does not limit the specific caching method.

[0257] Of course, you can also set the total length of cached data in the three cache queues (i.e., the total length of cached YUV data in the GPU). When the total length of the three cache queues is fixed, the principle of first-in-first-out (FIFO) follows the order of the time when the cached data is cached in the three cache queues.

[0258] The order of I-frames, P-frames, and B-frames in a set of video frames in this embodiment, as well as the selected preset cached video frames and temporary cached video frames, are only examples and do not necessarily represent the order or type of video frames (I, P, B) in the examples.

[0259] Reference Figure 16 This example illustrates a scenario for implementing frame playback using the frame buffering method provided in this application. In this example, when the user slides forward (the touch time corresponding to the next touch position is greater than the touch time corresponding to the previous touch position) and then immediately slides backward (the touch time corresponding to the next touch position is less than the touch time corresponding to the previous touch position), the scenario is as follows:

[0260] During the forward sliding process:

[0261] Touch point position 0 is detected and mapped to frame 27; frame 27 is rendered and displayed. At the same time, when decoding the video frames in the video frame group where frame 27 is located, frames 28, 29 and 30 are cached in the GPU, and the IDs and timestamps of frames 28 to 30 are cached in the CPU.

[0262] Touch point position 1 is detected and mapped to frame 33. Frame 33 is rendered and displayed. At the same time, when decoding the video frames in the video frame group where frame 33 is located, frames 32 and 34 are cached in the GPU. The IDs and timestamps of frames 32 and 34 are also cached in the CPU. If the maximum number of video frames that can be cached in the GPU is limited to 4, then when caching the YUV data of frame 34, the YUV data of frame 28 is released.

[0263] During the forward slide followed immediately by the reverse slide:

[0264] Touch point position 2 is detected and mapped to frame 32. At this time, frame 32 is still cached in the GPU. In this case, the CPU cache data is checked to see if the timestamp of frame 32 exists. If it exists, the YUV data of frame 32 is found in the GPU by ID. Finally, the GPU renders the YUV data of frame 32 cached in the GPU, saving the process of copying YUV data to the CPU and then from the CPU to the GPU.

[0265] Contact point 3 is detected and mapped to frame 28. At this point, frame 28 has been released from the GPU. In this case, the CPU cache data is searched for the timestamp of frame 28, which does not exist. Then, frame 28 is decoded by the hardware decoder in the GPU. Of course, when decoding frame 28, it is necessary to decode each video frame in the video frame group to which frame 28 belongs. After decoding each video frame in the video frame group to which frame 28 belongs, it is also necessary to determine whether to cache each video frame in the above manner. If caching is required, the ID and timestamp of the cached video frame are cached in the CPU.

[0266] In this embodiment of the application, in order to improve the utilization rate of cached data, the following can also be set:

[0267] If the cached decoding data includes the decoding data of the video frame mapped to the current touch position (e.g., frame 28 mapped to touch position 3), then the decoding data of that video frame (frame 28) is rendered. If the cached decoding data does not include the decoding data of the video frame mapped to the current touch position (e.g., frame 28 mapped to touch position 3), but includes decoding data whose timestamp and the timestamp of frame 28 are less than a certain value (e.g., including YUV data of frame 29), then the decoding data whose timestamp and the timestamp of frame 28 are less than the threshold (YUV data of frame 29) is rendered. In this way, even if the decoding data of video frames relatively close to frame 28 (e.g., the video frames in the video file before, two before, one after, or two after frame 28) are cached, there is no need to decode frame 28. Instead, the decoding data adjacent to frame 28 is rendered, which reduces the waiting time for decoding and improves the smoothness of the picture when playing frames. The threshold can be m times the period of the obtained frame rate of the video file (e.g., 1, 2, 3, 4, etc.), and the value of m can be set according to the situation.

[0268] In practical applications, there are other conditions for releasing cached YUV data in the GPU:

[0269] (1) When the video file playback interface is closed, i.e. Figure 2 (b) to Figure 2 Any interface shown in (e) is closed and returned to Figure 2 When the interface shown in (a) is displayed, it is necessary to release all three types of video frames in the three cache queues to free up memory resources.

[0270] (2) When the time elapsed since the user's release operation during the last frame playback exceeds a threshold (second duration), all three types of video frames in the three buffer queues are released. When releasing video frames from the three buffer queues, the interface displayed on the electronic device can be any interface; for example, the user can display a different interface when starting from... Figure 2 (b) to Figure 2The threshold is reached only after any of the interfaces shown in (e) are returned to the background, requiring the release of the three types of video frames from the three cache queues. Figure 2 (b) to Figure 2 In the case of any interface shown in (e), if the threshold is reached, it is also necessary to release the three types of video frames in the three buffer queues.

[0271] When releasing the YUV data cached in the GPU, it is also necessary to release the ID and timestamp of the released YUV data in the CPU at the same time.

[0272] This application uses data decoded in a GPU as an example. In practical applications, if YUV data is decoded and output in a CPU, it is also necessary to send it from the decoded output queue to different cache queues. This application can also be applied to scenarios where the data is decoded by the CPU. For example, after the CPU decodes and outputs YUV data, it is not necessary to copy the YUV data output by the CPU to different cache queues. Instead, the YUV data output by the CPU is kept in the original storage space, and the ID and timestamp of the YUV data are cached in different cache queues. That is, the ID and timestamp of the YUV data are used instead of the YUV data itself to be cached in different cache queues. In this way, the process of copying the YUV data from the decoded output queue to different cache queues can be avoided, and the migration process of YUV data can also be saved.

[0273] This application uses a gallery application to implement YUV data caching during frame playback and the use of caching during frame playback as examples to describe the switching process between normal playback and frame playback. In practical applications, it can also be applied to other applications, such as video editing applications and video playback applications.

[0274] This application provides a frame playback method, including:

[0275] The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file;

[0276] The first video frame is hardware-decoded by the GPU to obtain the first decoded data of the first video frame.

[0277] The first index of the first decoded data in the GPU is stored in the first cache space of the CPU;

[0278] The second touch point moment is mapped to the second video frame. The second touch point moment is the playback moment of the video file where the second touch point position detected during frame playback is mapped to the playback position. The detection time of the second touch point position is later than the detection time of the first touch point position.

[0279] If the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, the first index of the first decoded data of the first video frame in the GPU is obtained from the first cache space of the CPU.

[0280] Search for the first decoded data corresponding to the first index in the GPU;

[0281] The first decoded data is rendered by the GPU, and the rendered first decoded data is used for display.

[0282] In this application, if the first video frame is not rendered and displayed, the decoded data of the video frame may be cached in the GPU. The timestamp of the decoded data and the ID in the GPU are cached in the CPU. When the timestamp of the video frame mapped to the subsequent touch point is less than the threshold, the ID can be looked up in the CPU, and the decoded data can be looked up in the GPU by the ID. The found decoded data is then rendered and displayed, which improves the smoothness of frame playback and reduces the consumption of CPU memory.

[0283] The first cache space of the CPU (e.g., one or more of three cache queues) stores the first index (e.g., id) and the first timestamp of the first decoded data;

[0284] When the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, obtaining the first index of the first decoded data of the first video frame in the GPU from the first cache space of the CPU includes:

[0285] The first cache space is searched to see if there exists a timestamp whose difference from the second timestamp is less than a threshold.

[0286] If a first timestamp is found whose difference from the second timestamp is less than a threshold, the first index corresponding to the first timestamp is obtained.

[0287] After checking if a timestamp exists in the first cache space whose difference from the second timestamp is less than a threshold, the process further includes:

[0288] If no first timestamp with a difference of less than a threshold is found, the second video frame is decoded by GPU or CPU to obtain the second decoded data of the second video frame.

[0289] The second decoded data is rendered by the GPU, and the rendered second decoded data is used for display.

[0290] Before storing the first index of the first decoded data in the GPU in the first cache space of the CPU, the process further includes:

[0291] Determine whether to render and display the first decoded data;

[0292] If rendering and displaying the first decoded data is abandoned, determine whether to cache the first decoded data;

[0293] It is determined that the first decoded data will be cached.

[0294] After determining whether to render and display the first decoded data, the process also includes:

[0295] If the first decoded data is to be rendered and displayed, the first decoded data is rendered by the GPU, and the rendered first decoded data is used for display.

[0296] Release the first decoded data from the GPU.

[0297] After determining whether to cache the first decoded data, the process also includes:

[0298] It is determined that the first decoded data will not be cached;

[0299] Release the first decoded data from the GPU.

[0300] The determination of whether to render and display the first decoded data includes:

[0301] Determine whether there is a third decoded data in the GPU that was decoded later than the first decoded data;

[0302] If, at the moment of rendering execution, there is third decoded data in the GPU that was decoded later than the first decoded data, then the rendering and display of the first decoded data is abandoned.

[0303] When the time for rendering is reached, if there is no third decoded data in the GPU that was decoded later than the first decoded data, the first decoded data is rendered and displayed.

[0304] The determination of whether to cache the first decoded data includes:

[0305] Determine whether the first video frame belongs to one of the M video frames at the end of the first frame group, where M is greater than or equal to 1.

[0306] If the first video frame belongs to one of the M video frames at the end of the first frame group, the first decoded data is buffered.

[0307] If the first video frame does not belong to one of the M video frames at the end of the first frame group, the first decoded data is not cached.

[0308] The maximum number of decoded data cached in the GPU is N, where N is greater than or equal to 3; the method also includes:

[0309] When the number of decoded data cached in the GPU is N, if a new decoding instruction is received, the earliest decoded data in the currently cached decoded data in the GPU is deleted, and the first index of the earliest decoded data in the first cache space of the CPU is deleted.

[0310] The second touch point position and the first touch point position are touch point positions in the same swipe gesture.

[0311] The hardware decoding of the first video frame via GPU includes:

[0312] The GPU decodes each video frame in the first frame group containing the first video frame in the decoding order.

[0313] During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and some video frames in the first frame group are marked as non-display video frames.

[0314] The step of determining whether to cache the first decoded data when rendering and displaying the first decoded data is abandoned includes:

[0315] If rendering and display of the first decoded data is abandoned, determine whether the storage time of the first decoded data in the GPU is greater than the first duration;

[0316] If the storage time of the first decoded data in the GPU exceeds the first duration, the first decoded data is marked as invalid.

[0317] If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame for caching.

[0318] After marking the first decoded data as invalid, the method further includes:

[0319] If the first video frame marked as invalid is not marked as a non-display video frame, then the first video frame is released from the GPU.

[0320] After rendering the first decoded data using the GPU, the process also includes:

[0321] The first decoded data is marked as invalid.

[0322] The step of marking a portion of the video frames in the first frame group as non-display video frames includes:

[0323] M video frames are selected from the end of the first frame group and marked as preset cached video frames, and the non-display video frames include the preset cached video frames;

[0324] Q video frames are selected from the adjacent positions of the first video frame in the first frame group and marked as temporary buffer video frames. The non-display video frames include the temporary buffer video frames, and Q is greater than or equal to 1.

[0325] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0326] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0327] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0328] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0329] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0330] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0331] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0332] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A frame playback method, characterized in that, include: The first touch point moment is mapped to the first video frame, where the first touch point moment is the first touch point position detected during frame playback mapped to the playback moment in the video file; The first video frame is hardware-decoded by the GPU to obtain the first decoded data of the first video frame. The first index of the first decoded data in the GPU is stored in the first cache space of the CPU; The second touch point moment is mapped to the second video frame. The second touch point moment is the playback moment of the video file where the second touch point position detected during frame playback is mapped to the playback moment of the video file. The detection time of the second touch point position is later than the detection time of the first touch point position. If the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, the first index of the first decoded data of the first video frame in the GPU is obtained from the first cache space of the CPU. Search for the first decoded data corresponding to the first index in the GPU; The first decoded data is rendered by the GPU, and the rendered first decoded data is used for display.

2. The method as described in claim 1, characterized in that, The CPU's first cache space stores the first index and the first timestamp of the first decoded data; When the difference between the second timestamp of the second video frame and the first timestamp of the first video frame is less than a threshold, obtaining the first index of the first decoded data of the first video frame in the GPU from the first cache space of the CPU includes: The first cache space is searched to see if there exists a timestamp whose difference from the second timestamp is less than a threshold. If a first timestamp is found whose difference from the second timestamp is less than a threshold, the first index corresponding to the first timestamp is obtained.

3. The method as described in claim 2, characterized in that, After checking if a timestamp exists in the first cache space whose difference from the second timestamp is less than a threshold, the process further includes: If no first timestamp with a difference of less than a threshold is found, the second video frame is decoded by GPU or CPU to obtain the second decoded data of the second video frame. The second decoded data is rendered by the GPU, and the rendered second decoded data is used for display.

4. The method according to any one of claims 1 to 3, characterized in that, Before storing the first index of the first decoded data in the GPU in the first cache space of the CPU, the process further includes: Determine whether to render and display the first decoded data; If rendering and displaying the first decoded data is abandoned, determine whether to cache the first decoded data; It is determined that the first decoded data will be cached.

5. The method as described in claim 4, characterized in that, After determining whether to render and display the first decoded data, the process also includes: If the first decoded data is to be rendered and displayed, the first decoded data is rendered by the GPU, and the rendered first decoded data is used for display. Release the first decoded data from the GPU.

6. The method as described in claim 4 or 5, characterized in that, After determining whether to cache the first decoded data, the process also includes: It is determined that the first decoded data will not be cached; Release the first decoded data from the GPU.

7. The method as described in claim 4, characterized in that, The determination of whether to render and display the first decoded data includes: Determine whether there is a third decoded data in the GPU that was decoded later than the first decoded data; If, at the moment of rendering execution, there is third decoded data in the GPU that was decoded later than the first decoded data, then the rendering and display of the first decoded data is abandoned. When the time for rendering is reached, if there is no third decoded data in the GPU that was decoded later than the first decoded data, the first decoded data is rendered and displayed.

8. The method according to any one of claims 4 to 7, characterized in that, The determination of whether to cache the first decoded data includes: Determine whether the first video frame belongs to one of the M video frames at the end of the first frame group, where M is greater than or equal to 1. If the first video frame belongs to one of the M video frames at the end of the first frame group, the first decoded data is buffered. If the first video frame does not belong to one of the M video frames at the end of the first frame group, the first decoded data is not cached.

9. The method according to any one of claims 1 to 8, characterized in that, The maximum number of decoded data cached in the GPU is N, where N is greater than or equal to 3; the method also includes: When the number of decoded data cached in the GPU is N, if a new decoding instruction is received, the earliest decoded data in the currently cached decoded data in the GPU is deleted, and the first index of the earliest decoded data in the first cache space of the CPU is deleted.

10. The method according to any one of claims 1 to 9, characterized in that, The second touch point position and the first touch point position are touch point positions in the same swipe gesture.

11. The method as described in claim 5, characterized in that, The hardware decoding of the first video frame via GPU includes: The GPU decodes each video frame in the first frame group containing the first video frame in the decoding order. During the decoding of each video frame in the first frame group: after the first video frame is successfully decoded, the first video frame is marked as a display video frame, and some video frames in the first frame group are marked as non-display video frames. The step of determining whether to cache the first decoded data when rendering and displaying the first decoded data is abandoned includes: If rendering and display of the first decoded data is abandoned, determine whether the storage time of the first decoded data in the GPU is greater than the first duration; If the storage time of the first decoded data in the GPU exceeds the first duration, the first decoded data is marked as invalid. If the first video frame marked as invalid is also marked as a non-display video frame, then the first video frame is converted into the non-display video frame for caching.

12. The method as described in claim 11, characterized in that, After marking the first decoded data as invalid, the method further includes: If the first video frame marked as invalid is not marked as a non-display video frame, then the first video frame is released from the GPU.

13. The method as described in claim 12, characterized in that, After rendering the first decoded data using the GPU, the process also includes: The first decoded data is marked as invalid.

14. The method as described in claim 11, characterized in that, The step of marking a portion of the video frames in the first frame group as non-display video frames includes: M video frames are selected from the end of the first frame group and marked as preset cached video frames, and the non-display video frames include the preset cached video frames; Q video frames are selected from the adjacent positions of the first video frame in the first frame group and marked as temporary buffer video frames. The non-display video frames include the temporary buffer video frames, and Q is greater than or equal to 1.

15. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when the one or more processors execute the computer program, causes the electronic device to perform the method as described in any one of claims 1-14.

16. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-14.

17. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-14.

18. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-14.

Images