A video playing method, an electronic device, a storage medium and a chip

Abstract

This application provides a video playback method, electronic device, storage medium, and chip, relating to the field of terminal device technology. The method involves decoding a video file to obtain I-frames, P-frames, and B-frames during normal video playback, and then re-encoding the P-frames and B-frames into I-frames. The original I-frames of the video file and the re-encoded I-frames are then cached. In a frame playback scenario, based on the detected frame playback time, the corresponding I-frame is retrieved from the cache, and then the retrieved I-frame is decoded, avoiding the decoding of groups of video frames and improving the smoothness of frame playback. Furthermore, even when the I-frame corresponding to the frame playback time is not cached, the video frame group containing the I-frame corresponding to the frame playback time can be decoded using an idle decoder from at least two configured decoders. Decoding via an idle decoder improves decoding efficiency and also enhances the smoothness of frame playback.

Description

Technical Field

[0001] This application relates to the field of terminal equipment technology, and in particular to a video playback method, electronic device, storage medium and chip. Background Technology

[0002] Gallery apps on electronic devices can provide image preview interfaces, allowing users to view various images stored on the device. Gallery apps also provide video playback interfaces, allowing users to play various videos stored on the device.

[0003] The video playback interface can also display the video playback progress, for example, through a progress bar, progress ring, or progress display frame. The user's dragging operation on the progress bar, progress ring, or progress display frame can trigger the electronic device to play the video frame by frame. However, in the frame playback scenario, the display process of the video is relatively choppy and not smooth. Summary of the Invention

[0004] This application provides a video playback method, electronic device, storage medium, and chip, which can reduce video stuttering in frame-by-frame playback scenarios and improve the smoothness of video playback in frame-by-frame playback scenarios.

[0005] To achieve the above objectives, the first aspect of this application adopts the following technical solution:

[0006] The first aspect of this application provides a video playback method, including:

[0007] An electronic device displays a first interface of a first video, the first interface including a first area and a first control, the first area including a second control;

[0008] After receiving the first operation on the second control, the electronic device displays the first video frame obtained by decoding the first encoded data of the first video in the first area.

[0009] During the process of displaying the first video frame in the first area, the electronic device controls the first control to move from a first position to a second position along a first trajectory;

[0010] When the first control is moved to the second position, the electronic device receives a first drag operation on the first control;

[0011] During the process of moving the first control from the second position to the third position through the first drag operation, the electronic device displays a video frame obtained by decoding the second encoded data of the first video in the first area. The third position is on the first trajectory and the third position is located between the first position and the second position. The electronic device's efficiency in decoding the second encoded data is higher than its efficiency in decoding the first encoded data.

[0012] In this application, during the process of displaying video frames of the first video triggered by the first operation of the second control, the video frame to be displayed can be obtained by decoding the first encoded data of the first video; during the process of displaying video frames of the first video triggered by the drag operation (denoted as frame playback), the video frame to be displayed can be obtained by decoding the second encoded data; since the electronic device is more efficient at decoding the second encoded data than at decoding the first encoded data, the decoding efficiency in the frame playback scenario can be improved, thereby improving the smoothness of the video picture in the frame playback scenario.

[0013] As another implementation of the first aspect, during the process of displaying the first video frame in the first region, the method further includes:

[0014] The electronic device caches the second encoded data of the first video frame.

[0015] During the process of displaying the first video frame obtained by decoding the first encoded data of the first video in the playback order triggered by the second control, the second encoded data of the first video frame can be cached. The decoding efficiency of the second encoded data is higher than that of the first encoded data. In the subsequent process of displaying video frames through the first drag operation on the first control, if the second encoded data of the video frame to be displayed has been cached, the second encoded data can be directly decoded and the decoded video frame can be displayed. Since the decoding efficiency of the second encoded data is higher than that of the first, the display of video frames during subsequent playback is faster, improving the smoothness of the video picture in the frame playback scene.

[0016] As an example, some encoded data in the first encoded data needs to refer to other encoded data when decoding, resulting in low decoding efficiency; while each encoded data in the second encoded data does not need to refer to other encoded data when decoding, resulting in high decoding efficiency.

[0017] As another example, some encoded data in the first encoded data need to be referenced when decoding other encoded data. For example, the decoding of P-frames and I-frames needs to refer to the encoded data of I-frames; the second encoded data can also be the various YUV data obtained by decoding the first encoded data.

[0018] As one implementation of the first aspect, the frame types in the first encoded data include keyframes and non-keyframes, and the frame types in the second encoded data are all keyframes.

[0019] In this application, keyframes are I-frames, and non-keyframes are P-frames and B-frames. Since the I-frames, P-frames, and B-frames in the first encoded data are all re-encoded into I-frames, the I-frames corresponding to each timestamp can be stored. Therefore, during decoding, each I-frame can be decoded individually, without the need to decode video frames in groups. This improves decoding speed and the smoothness of the video playback scene.

[0020] As another implementation of the first aspect, the method further includes:

[0021] When the first control is dragged to the fourth position, the electronic device acquires the third encoded data of the first video. When the first control is at the fourth position, the playback progress of the first video is at the first moment. The encoded data corresponding to the first moment is the fourth encoded data in the third encoded data. The frame type in the third encoded data is the same as the frame type in the first encoded data. The fourth position is not on the first trajectory.

[0022] The electronic device decodes the third encoded data using a first decoder;

[0023] After the electronic device decodes the fourth encoded data through the first decoder, it displays the video frame obtained by decoding the fourth encoded data in the first area.

[0024] In this application, when the fourth position is not on the first track and the first control is in the fourth position, the playback progress is at the first moment, which means that the video frame corresponding to the first moment is not cached. Therefore, it is necessary to obtain the third encoded data corresponding to the video frame group where the first moment is located, and then decode the third encoded data to obtain the fourth encoded data corresponding to the first moment. The frame types in the third encoded data and the first encoded data are the same. Compared with the process of obtaining the second encoded data from the cache and decoding and displaying it, the process of decoding the third encoded data and displaying the fourth decoded data is also less efficient.

[0025] As another implementation of the first aspect, the method further includes:

[0026] When the first control is dragged from the fourth position to the fifth position, the electronic device acquires the fifth encoded data of the first video. When the first control is in the fifth position, the playback progress of the first video is at the second moment. The encoded data corresponding to the second moment is the sixth encoded data in the fifth encoded data. The frame type in the fifth encoded data is the same as the frame type in the first encoded data. The first video frame in the fifth encoded data and the last video frame in the third encoded data are adjacent video frames.

[0027] The electronic device decodes the fifth encoded data using a second decoder;

[0028] After the electronic device completes the decoding of the fifth encoded data through the second decoder, the electronic device displays the video frame obtained by decoding the sixth encoded data in the first area.

[0029] In this application, in order to improve decoding efficiency, at least two decoders are set for frame playback. By decoding two consecutive sets of video frames with different decoders, even if the first decoder has not finished decoding the first set of video frames (e.g., the third encoded data), it does not affect the second decoder to decode and display the second set of video frames (e.g., the fifth encoded data) in a timely manner, thus improving the smoothness of the video picture in the frame playback scene.

[0030] As another implementation of the first aspect, the process of displaying the first video frame obtained by decoding the first encoded data of the first video in the first area according to the playback order includes:

[0031] For each piece of first encoded data, the electronic device decodes the first encoded data using a third decoder to obtain YUV data;

[0032] The electronic device displays video frames corresponding to the YUV data obtained by decoding the first encoded data in the first area.

[0033] As another implementation of the first aspect, the process of the electronic device caching the second encoded data of the first video frame includes:

[0034] For each first encoded data, the electronic device encodes the YUV data obtained from the first encoded data to obtain the second encoded data corresponding to the first encoded data;

[0035] The electronic device caches the second encoded data corresponding to the first encoded data and the timestamp of the first encoded data in the first storage space.

[0036] In this application, since the YUV data already needs to be decoded during the previous normal playback, the decoded YUV data is re-encoded and each re-encoded data and timestamp is cached; no additional decoding operation is required, thus improving efficiency.

[0037] As another implementation of the first aspect, the first encoded data includes encoded data with a timestamp of a third moment. Before the electronic device encodes the YUV data obtained from the first encoded data at the third moment, the method further includes:

[0038] The electronic device determines that the second encoded data of the third moment of the first video does not exist in the first storage space.

[0039] In this application, to avoid duplicate caching, the re-encoding and caching operation can be performed only if the second encoded data of the third moment of the first video does not exist in the first storage space.

[0040] As another implementation of the first aspect, during the process of dragging the first control to the third position via the first drag operation, the electronic device displays the video frame obtained by decoding the second encoded data in the first area, including:

[0041] When the first control is dragged to the sixth position by the first drag operation, the electronic device displays a video frame obtained by decoding the seventh encoded data in the second encoded data in the first area. When the first control is at the sixth position, the playback progress of the first video is at the fourth moment, the seventh encoded data is the encoded data corresponding to the fourth moment, and the sixth position is on the first trajectory.

[0042] As another implementation of the first aspect, when the first control is dragged to the sixth position via the first drag operation, the electronic device displays a video frame obtained by decoding the seventh encoded data in the second encoded data in the first area, including:

[0043] When the first control is dragged to the sixth position via the first drag operation, the electronic device determines that the playback progress of the first video is the fourth moment.

[0044] The electronic device queried that the seventh encoded data corresponding to the fourth moment was stored in the first storage space;

[0045] The electronic device retrieves the seventh encoded data from the first storage space;

[0046] The electronic device obtains the seventh encoded data from the first storage space;

[0047] The electronic device decodes the seventh encoded data;

[0048] The electronic device displays the video frame obtained by decoding the seventh encoded data in the first area.

[0049] In this application, before decoding and displaying the cached encoded data, it is first necessary to determine whether the first storage space stores the video frame to be displayed, so that the corresponding video frame can be successfully obtained from the first storage space.

[0050] As another implementation of the first aspect, when the first control is dragged to the fourth position via the first drag operation, the electronic device acquires the third encoded data of the first video, including:

[0051] When the first control is dragged to the fourth position via the first drag operation, the electronic device determines that the playback progress of the first video is the first moment.

[0052] The electronic device did not find the encoded data corresponding to the first moment in the first storage space;

[0053] The electronic device acquires third encoded data of the first moment of the first video.

[0054] In this application, when the encoded data corresponding to the frame playback time is not found in the first storage space during frame playback, the first encoder or the second encoder is used in groups to decode the frames, thereby improving the smoothness of frame playback through concurrent frame decoding.

[0055] As another implementation of the first aspect, the encoded data corresponding to the first moment is encoded data in which the difference between the timestamp and the first moment is less than or equal to a first value, where the first value is half the difference between the timestamps of two adjacent video frames.

[0056] In this application, after determining a certain moment based on the control position, this moment may be the time between the playback moments of two adjacent video frames, that is, there is no video frame with a timestamp of this moment. Therefore, a video frame corresponding to two timestamps adjacent to this moment can be used as the video frame corresponding to this moment. Of course, a first value (half the difference between the timestamps of two adjacent video frames) can also be used as a reference, and the encoded data whose difference between the timestamp and the moment is less than the first value can be used as the encoded data of this moment.

[0057] As another implementation of the first aspect, the first interface further includes a second area, in which the first control is located and can move within the second area;

[0058] The first region and the second region do not overlap, or the second region is within the first region.

[0059] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0060] Thirdly, a chip is provided, including a processor coupled to a memory, the processor executing a computer program stored in the memory to cause an electronic device to implement the method of any of the first aspects of this application.

[0061] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when the computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0062] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0063] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0064] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0065] Figure 2 A schematic diagram of a video playback process provided in an embodiment of this application;

[0066] Figure 3 A schematic diagram of another video playback process provided in an embodiment of this application;

[0067] Figure 4 A schematic diagram of a video playback interface provided in an embodiment of this application;

[0068] Figure 5 A schematic diagram of another video playback process provided in an embodiment of this application;

[0069] Figure 6 A schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application;

[0070] Figure 7 This is a schematic diagram illustrating the playback and decoding order of video frames after being re-encoded into I-frames, as provided in an embodiment of this application.

[0071] Figure 8A schematic diagram illustrating the frame playback process of a video provided in this application embodiment;

[0072] Figure 9 This is a schematic diagram of the technical architecture for video playback provided in an embodiment of this application;

[0073] Figure 10 A timing diagram showing the preparatory work before normal playback and frame playback provided in the embodiments of this application;

[0074] Figure 11 A timing diagram illustrating a normal video playback process provided in an embodiment of this application;

[0075] Figure 12 A timing diagram of a video frame playback process provided in an embodiment of this application;

[0076] Figure 13 A timing diagram of another video frame playback process provided in an embodiment of this application;

[0077] Figure 14 A timing diagram for continuing normal playback after the end frame is played, as provided in the embodiments of this application;

[0078] Figure 15 This is a timing diagram of exiting the video playback interface provided in an embodiment of this application. Detailed Implementation

[0079] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0080] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0081] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0082] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0083] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0084] This application provides a video playback method that can be applied to electronic devices, such as tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not limit the specific type of electronic device.

[0085] Figure 1 A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 1211, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0086] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0087] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the video playback method in the embodiments of this application.

[0088] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] Internal memory 1211 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 1211. Internal memory 1211 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback function). Touch sensor 180K, also called a "touch panel". Touch sensor 180K can be set on display screen 194, and touch sensor 180K and display screen 194 form a touch screen, also called a "touch screen". Touch sensor 180K is used to detect touch operations on or near it. Touch sensor can transmit the detected touch operation to application processor to determine the touch event type. Visual output related to touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be set on the surface of electronic device 100, in a different position than display screen 194. For example, click operations of controls and drag operations of progress display frames provided in this application embodiment can be recognized by touch sensor.

[0090] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. For example, in this embodiment, the process of rendering YUV data can be implemented using a GPU.

[0091] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1. For example, in the embodiments of this application... Figure 2 or Figure 3All interfaces shown are displayed on the monitor.

[0092] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats. For example, the video codec can decode H.264 format video frames as provided in the embodiments of this application, or re-encode them into H.264 format video frames that are all keyframes.

[0093] This application does not specifically limit the structure of the execution entity of a video playback method. As long as communication can be performed according to the video playback method provided by this application by running code containing the code of this application's video playback method, it is acceptable. For example, the execution entity of the video playback method provided by this application can be a functional module in an electronic device capable of calling and executing programs, or a communication device applied in an electronic device, such as a chip.

[0094] Users can use the gallery app on their electronic devices to view various photos stored on the device and play various videos stored on the device.

[0095] Reference Figure 2 This is a schematic diagram of an interface for playing videos stored on an electronic device through a gallery application, as provided in an embodiment of this application.

[0096] Reference Figure 2 (a) in the image gallery is a grid of images and videos stored on an electronic device, displayed by the image gallery application of the electronic device. The grid can display images and videos, where the cell containing a video displays the cover and duration of the video, and the cell containing an image displays a thumbnail of the image.

[0097] The grid image can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app, or it can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app and triggers more steps.

[0098] When displaying images and / or videos stored on an electronic device, this grid chart can be presented in various ways, for example, Figure 2 The method shown in (a) displays photos and videos in a photo group in reverse chronological order; in practical applications, gallery applications can also set up multiple object groups, for example, through... Figure 2The control 10 shown in (a) can view multiple groups in the album: image group, video group, screenshot and screen recording group, etc. Users can also display pictures through the image group in the gallery application, videos through the video group in the gallery application, and screenshots and screen recordings through the screenshot and screen recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the grid diagram shown above.

[0099] This application does not limit the number of user operations required from the system desktop to the display of the grid chart in the gallery application, nor does it limit the way the grid chart displays images and videos.

[0100] Reference Figure 2 (b) in the text represents the user's click. Figure 2 Following the cover of video A in the grid diagram shown in (a), the electronic device displays the video playback interface of video A in response to the operation. The video playback interface includes a video playback area 11 and a progress display area 12.

[0101] The video playback area 11 is used to display the video frame during video playback. After entering the video playback interface from the grid interface, the control 111 is displayed in the video playback area. When the user clicks the control 111, the electronic device responds to the click operation on the control 111 and starts playing the video frame in the video playback area 11 from the 0th second. When the video is paused, the control 111 is also displayed in the video playback area 11. When the user clicks the control 111, the electronic device responds to the click operation on the control 111 and starts playing the video frame in the video playback area 11 from the current paused moment.

[0102] The progress display area 12 is used to display the video playback progress. The progress display area 12 includes a progress display frame 121 and a progress indicator line 122. The position of the progress indicator line 122 can remain unchanged. The user can drag the progress display frame 121 located below the progress indicator line 122 so that the progress display frame 121 moves left and right without moving the progress indicator line 122. The position of the progress indicator line 122 above the progress display frame indicates the current video playback progress.

[0103] As an example, the total playback duration of the current video is 15 seconds, and the total length of the current video's progress display frame is represented by L pixels. Based on these two parameters, we can obtain the playback duration corresponding to a unit length (or the length corresponding to a unit of time). Therefore, if the leftmost edge of the progress display frame (the position corresponding to the first pixel) is set to correspond to the 0th second of the video, then different positions of the progress indicator line on the progress display frame will correspond to different playback times. For ease of subsequent description, this application embodiment will be described using the terms "the time corresponding to the progress indicator line" or "the time the progress indicator line points to".

[0104] Due to space limitations in the progress display area 12, only a portion of the video frames or a portion of the video frames from the progress display frame 121 can be displayed in the progress display area 12. For example... Figure 2 As shown in (b), the progress display frame 121 displayed in the progress display area 12 includes: thumbnails of three complete video frames and a partial area of ​​a thumbnail of a video frame.

[0105] Typically, the video playback area 11 displays the video frame at the moment indicated by the progress indicator line 122 in the progress display area 12.

[0106] In a specific implementation, there is a situation where the time pointed to by the progress indicator line 122 is the time between the playback times of two adjacent video frames, that is, there is no video frame in the video file at the time pointed to by the progress indicator line 122.

[0107] As an example, in a video frame sorted by playback time (denoted as timestamp) from smallest to largest, the playback time of the i-th video frame is ti, and the playback time of the (i+1)-th video frame is t(i+1); the time t pointed to by the progress indicator line 122 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval to the time t pointed to by the progress indicator line 122 can be taken as the video frame at the time pointed to by the progress indicator line 122.

[0108] Of course, in practical applications, if there is no video frame in the video file with the same timestamp and the time pointed to by the progress indicator line, the video frame before the time t pointed to by the progress indicator line 122 (the video frame corresponding to ti) can be used as the video frame of the time pointed to by the progress indicator line 122; or the video frame after the time t pointed to by the progress indicator line 122 (the video frame corresponding to t(i+1)) can be used as the video frame of the time pointed to by the progress indicator line 122.

[0109] In practical applications, the progress display area 12 can display the playback progress in many ways. For example, it can display the video playback progress using a progress bar or a progress ring. This application provides two playback modes for video playback: normal playback and frame-by-frame playback. This application's embodiment only uses... Figure 2 The progress display area shown in (b) serves as an example to illustrate the difference between normal playback and frame playback.

[0110] Reference Figure 2 (c) in the text represents the user's click. Figure 2 Following control 111 in the interface shown in (b), the electronic device responds to the operation by displaying an interface diagram during the playback of video A. This interface diagram indicates that the video A has reached the 5th second, and displays the video frame corresponding to the 5th second in the video playback area 11. Simultaneously, the progress display frame 121 in the progress display area 12 moves its position so that the time corresponding to the progress indicator line 122 is the 5th second. From... Figure 2 (b) in Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0111] A video file consists of multiple video frames, each with a timestamp. The timestamp records the playback time of each video frame during the video file's playback process. For example, arranged by timestamp from smallest to largest, the video frames in the file are: Video Frame 0 (timestamp t0), Video Frame 1 (timestamp t1), Video Frame 2 (timestamp t2), Video Frame 3 (timestamp t3), Video Frame 4 (timestamp t4), Video Frame 5 (timestamp t5), Video Frame 6 (timestamp t6), Video Frame 7 (timestamp t7)... Normal playback involves displaying each video frame sequentially in the playback area according to its corresponding timestamp, from smallest to largest.

[0112] As an example of a normal playback process, video frame 0 is displayed in video playback area 11 at time t0 after playback begins; video frame 1 is displayed in video playback area 11 at time t1 after playback begins; video frame 2 is displayed in video playback area 11 at time t2 after playback begins; video frame 3 is displayed in video playback area 11 at time t3 after playback begins; video frame 4 is displayed in video playback area 11 at time t4 after playback begins; video frame 5 is displayed in video playback area 11 at time t5 after playback begins; video frame 6 is displayed in video playback area 11 at time t6 after playback begins; video frame 7 is displayed in video playback area 11 at time t7 after playback begins...

[0113] Combination Figure 2 (b) in Figure 2(c) shows the normal playback process of video A: starting from the 0th second of video A, each video frame is displayed sequentially according to the playback time indicated by the timestamp; Figure 2 (c) in the diagram is a schematic of the interface displaying the video frame up to the 5th second. (See reference...) Figure 2 In (d), the user drags... Figure 2 The progress display frame of the interface shown in (c) is shown in the image.

[0114] In response to Figure 2 As shown in (d) above, the electronic device displays the corresponding screen in the video playback area for the drag operation.

[0115] As an example, refer to Figure 2 In (e), when the user drags the progress bar to display frames so that the progress indicator points to the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be the video frame in video A with the timestamp of 14 seconds; it can also be the video frame with the timestamp closest to 14 seconds; it can also be the video frame before 14 seconds whose time interval between the timestamp and 14 seconds is closest; or it can be the video frame after 14 seconds whose time interval between the timestamp and 14 seconds is closest.

[0116] It should be noted that, in subsequent embodiments, the specific meaning of the description method of "video frame corresponding to a certain moment" can be referred to the detailed explanation of "video frame corresponding to the 14th second," and will not be repeated hereafter. Figure 2 (d) in Figure 2 The playback process shown in (e) is the frame playback process of the video. Typically, the video frame corresponding to a certain moment is the video frame whose timestamp differs from that moment by less than or equal to a first value (half the difference between the timestamps of two adjacent video frames).

[0117] As the user drags the progress display frame 121, a relative position is formed between the progress indicator line 122 and the progress display frame 121, and the position of the progress indicator line 122 on the progress display frame 121 is related to the time that the progress indicator line 122 points to.

[0118] In the specific implementation, assuming that the time indicated by the progress indicator line 122 is t1 before the user moves in the area where the progress display frame 121 is located on the screen. When the user moves left or right in the area where the progress display frame is located, the electronic device can detect the user's real-time touch position at certain time intervals. The electronic device determines the user's movement direction and distance based on the continuous real-time touch positions. The electronic device determines the time change Δt based on the user's movement distance; it determines the calculation method for real-time time t2 based on the user's movement direction. For example, if the movement direction is left, the starting time t1 is subtracted from the time change Δt to obtain the real-time time t2; if the movement direction is right, the starting time t1 is added to the time change Δt to obtain the real-time time t2. After obtaining the real-time time t2, the electronic device finds the video frame corresponding to the real-time time t2 and displays the video frame corresponding to t2 in the video playback area.

[0119] On the other hand, the electronic device determines the movement direction of the progress display frame based on the movement direction determined by the real-time touch point position, and determines the movement distance of the progress display frame based on the movement distance of the real-time touch point position. The electronic device then moves the progress display frame in the progress display area according to the movement direction and distance, so that the time indicated by the progress indicator line is t2. As another example, the electronic device also determines the movement direction (left if t2 is greater than t1, right if t2 is less than t1) and movement distance of the progress display frame based on the start time t1 and the real-time time t2. The electronic device then moves the progress display frame in the progress display area according to the movement direction and distance, so that the time indicated by the progress indicator line is t2.

[0120] It should be noted that the above process is only an example. In actual applications, there are multiple ways for the user to determine the time indicated by the progress indicator line 122, the position of the progress display frame 121, and the video frame displayed in the video playback area 11 while dragging the progress display frame 121.

[0121] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. It may move to the left, it may move to the right, it may be dragged quickly, or it may be dragged slowly. Accordingly, the video frame images displayed in the video playback area 11 have randomness.

[0122] As an example, before the user moves the area where the progress display frame is located on the screen, the starting time indicated by the progress indicator line is t1. During the user's dragging of the progress display frame, the electronic device detects the user's real-time touch position at certain time intervals, thus obtaining the real-time times corresponding to multiple consecutive time periods: t2, t3, t7, t4, and t3. In the video playback area, video frames are played in the following order: starting with the video frame corresponding to time t1, the video frames corresponding to time t2, t3, t7, t4, and t3 are played sequentially. It can be understood that the process of the electronic device playing video in the video playback area triggered by the user clicking control 111 is normal playback; the process of the electronic device displaying the video frame at the time indicated by the progress indicator line in the video playback area triggered by the user dragging the progress display frame is frame playback; furthermore, during the normal playback of video A, if the user drags the progress display frame and then releases it from the area where the progress display frame is located, the electronic device continues to play video A in normal playback mode.

[0123] Figure 2 (d) to Figure 2 As shown in (e), this is in Figure 2 Based on the interface shown in (c), the user drags the progress bar to the left to display the frames, triggering the electronic device to display the video frames after the 5th second of video A in a frame playback mode.

[0124] As another embodiment of this application, in Figure 2 Based on the interface shown in (c), the user can also drag the progress bar to the right to display the frames, triggering the electronic device to display the video frames between the 0th and 5th seconds of video A in frame playback mode.

[0125] Reference Figure 3 In (a), the illustration is similar to Figure 2 (c) corresponds to playing video A in normal mode up to the 5th second.

[0126] Reference Figure 3 As shown in (b), when the video is played normally up to the 5th second of video A, if the user drags the progress bar to the right to display the frames, the electronic device will be triggered to display the video frames before the 5th second of video A in frame playback mode.

[0127] Reference Figure 3 As shown in (c), when the user drags the progress display frame so that the progress indicator line points to the 2nd second, the electronic device displays the 2nd second video frame of video A in the video playback area.

[0128] Reference Figure 3As shown in (d), when the user drags the progress display frame to the 2nd second, the user lifts their finger to release the progress display frame, and the electronic device will start playing video A in normal playback mode from the 2nd second.

[0129] In a specific implementation, this application also provides another schematic diagram of an interface for playing videos stored in an electronic device through a gallery application.

[0130] This application provides a video recording mode: one-to-many. That is, while recording a video, photos can be generated simultaneously (e.g., photos automatically captured by the electronic device based on the screen content, and photos manually taken by the user using the camera control on the video recording interface), so that photos taken during the video recording process can be obtained. The recorded video can be associated with the photos taken during the video recording process; similarly, when viewing the video through a gallery application, the photos taken during the video recording process can also be displayed in the progress display area of ​​the video playback interface.

[0131] Reference Figure 4 (a) in the text refers to this interface. Figure 2 The description of (a) in the text.

[0132] Reference Figure 4 (b) in the text represents the user's click. Figure 4 After the cover of video A in the grid diagram shown in (a) appears, the electronic device displays the video playback interface of video A in response to this operation. This video playback interface and... Figure 2 The video playback interface shown is similar, the difference being the content displayed in the progress display area 12.

[0133] The progress display frame 121 in progress display area 12 is in compressed state. A compressed progress display frame can display two video frames, three video frames, etc. (Refer to...) Figure 2 As shown in (b), the progress display frame 121 in the compressed state includes thumbnails of the first frame and intermediate frames of the video. Of course, in practical applications, thumbnails of other frames of the video can also be displayed, such as thumbnails of the first frame and the last frame. Thumbnails 123 of photos taken during the video recording process are displayed to the right of the progress display frame 121.

[0134] Users can interact with the progress bar (e.g., click or double-click) to trigger it to switch from compressed to expanded mode. An expanded progress bar can include more video frame thumbnails than a compressed one. The number of video frames displayed in the expanded state is related to the video's duration; for example, the longer the video, the more video frames the progress bar will include.

[0135] Reference Figure 4As shown in (c), this is the user's click. Figure 4 The diagram shows an expanded progress display frame displayed by an electronic device in response to the operation following the compressed progress display frame 121 shown in (b). Due to space limitations, the photo obtained during the video recording process is hidden after the progress display frame is expanded.

[0136] Figure 4 The interface shown in (c) and Figure 2 The interface shown in (b) is the same, therefore, based on Figure 4 The normal playback process and frame playback process of the interface shown in (c) can be referred to Figure 2 (b) to Figure 2 The description in (e) is not repeated here.

[0137] Reference Figure 5 This is a schematic diagram of another interface for playing videos stored in an electronic device through a gallery application, as described in an embodiment of this application.

[0138] Figure 5 (a) and Figure 5 (b) in the text can be referred to. Figure 4 (a) and Figure 4 The description in (b) of the document.

[0139] exist Figure 5 In the interface shown in (b), the progress display frame is compressed. When the user clicks... Figure 5 Control 111 in the interface shown in (b) triggers the electronic device to play video A in normal playback mode.

[0140] Reference Figure 5 (c) in the text represents the user's click. Figure 5 The interface shown in (b) is an interface diagram during the playback of video A after control 111 is used. This interface diagram corresponds to an interface during normal playback when the progress display frame is in compressed state. This interface diagram indicates that the video A has been played to the 5th second, and the video frame corresponding to the 5th second is displayed in the video playback area; at the same time, the progress display frame in the progress display area moves so that the time corresponding to the progress indicator line is the 5th second. Among them, from Figure 5 (b) in Figure 5 The playback process shown in (c) is the normal playback process of the video.

[0141] Reference Figure 5 In (d), the user is Figure 5 The progress bar is displayed by dragging the frame based on the interface shown in (c).

[0142] Reference Figure 5 In (e), the electronic device responds to Figure 4The drag operation shown in (d) displays the video frame at the 14th second, indicated by the progress indicator line in the video playback area. Specifically, the progress indicator line in the progress display area corresponds to the 14th second of playback, and the video playback area displays the video frame corresponding to that 14th second.

[0143] It should be noted that if the progress display frame is in compressed mode, dragging the progress display frame will maintain the compressed state; of course, the user can also drag it... Figure 4 As shown, the progress display frame is first triggered to change from a compressed state to an expanded state by an operation, and then the expanded progress display frame is dragged, so the progress display frame remains in the expanded state.

[0144] Of course, in practical applications, Figure 4 and Figure 5 In the scenario shown, the user can drag the progress display frame to the left or to the right, which will not be elaborated upon in this application.

[0145] Furthermore, the operations shown in the embodiments of this application are merely examples. In practical applications, other operations can be set according to specific circumstances. For example, some single-click operations can also be double-click operations or other air gestures. Some drag operations can also be air gestures. Similarly, some examples of operations in subsequent embodiments can also be other operations.

[0146] In the above embodiments, whether it is normal playback or frame playback, the video frames need to be decoded before they are displayed in the video playback area. Usually, the video frames are decoded in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a keyframe (I-frame), at least one forward reference frame (P-frame), and at least two bidirectional reference frames (B-frames).

[0147] Reference Figure 6 This is a schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application. A set of video frames can also be referred to as a group of pictures (GOP).

[0148] Each group of video frames, in playback order, includes: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. Among these, the I-frame, as a keyframe, can be decoded to obtain a complete image, but decoding is slower. The P-frame records the changes relative to the previous frame (not a B-frame); information from the previous frame is required to decode the P-frame. Similarly, information from previous P-frames (which may also be I-frames) and subsequent P-frames is needed to decode the B-frame. P-frames and B-frames improve video smoothness while reducing the file size.

[0149] Whether it's an I-frame, P-frame, or B-frame, they are all video frames. When decoding a video frame at a specific moment, you don't directly decode the video frame at that moment. Instead, you decode the group of video frames that the video frame at that moment belongs to. When decoding video frames within a group of video frames, you decode them one by one in the decoding order.

[0150] As an example, if we need to decode the video frame at time t1, and the video frame at time t1 is the 6th video frame in the playback order of a group of video frames: frame B; then the decoding order is:

[0151] First decoding: the video frame played in the first playback order: I-frame;

[0152] The second decoding: the 4th video frame in playback order: P-frame;

[0153] The third decoding step: the second video frame played in the playback order: frame B;

[0154] The fourth decoding: the third video frame in playback order: frame B;

[0155] The fifth decoding: the 7th video frame in playback order: P-frame;

[0156] The sixth decoding: the 5th video frame in playback order: frame B;

[0157] The seventh decoding: the 6th video frame in playback order: frame B;

[0158] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding operation to decode the video frame at time t1.

[0159] Currently, in normal playback scenarios, the video frames in each video frame group are parsed, decoded, rendered, and displayed sequentially according to the playback order. Of course, when decoding the video frames in each video frame group, the process is as follows: Figure 6 The decoding order shown decodes each video frame in the video frame group sequentially. Typically, when displaying video frames from the previous video frame group, all video frames in the next video frame group have already been decoded. Therefore, in normal playback scenarios, each video frame can be displayed according to its timestamp, resulting in smooth video playback.

[0160] In frame-based playback scenarios, as mentioned earlier, the process of a user dragging the progress bar to display frames is unpredictable. The user may drag the progress bar to the left, or to the right, or drag it quickly or slowly. Consequently, the video frames displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display frames, the electronic device needs to determine the moment the progress indicator line points based on the real-time detected touch point position. Then, it needs to parse, decode, render, and display the video frame at the moment the progress indicator line points. As mentioned earlier, before decoding the video frame at that moment, it may be necessary to decode other video frames in the same video frame group. Therefore, frame-based playback scenarios are prone to stuttering and unsmoothness.

[0161] In other frame-based playback scenarios, to quickly decode and render the video frame at the time indicated by the progress indicator line (e.g., t1), only the first video frame in the decoding order within the video frame group containing the video frame at time t1 can be decoded, and this first decoded frame can be used as the video frame at time t1. Of course, other frames in the same video frame group (e.g., P-frames and B-frames) are not decoded during this process. However, while this method improves decoding speed, sacrificing non-critical frames used to improve smoothness will result in discontinuous and stuttering video playback.

[0162] In view of this, this application provides a method for playing video frames. Since normal video playback requires decoding each video frame (I-frame, P-frame, and B-frame) in each group of video frames, the P-frames and B-frames decoded during normal playback can be re-encoded into I-frames. This is equivalent to re-encoding the I-frames, P-frames, and B-frames decoded during normal playback into a video frame consisting entirely of I-frames, which is then cached. During subsequent frame playback, the I-frames corresponding to the timestamps are directly retrieved from the cache, and then decoded and rendered. This eliminates the need for additional decoding of other video frames, thereby improving the smoothness of the video during frame playback.

[0163] Of course, in practical applications, if the cache space has already cached some key frames of the video file, these key frames of the video file can also be obtained from the cache space for decoding and playback during normal playback.

[0164] Reference Figure 7 As shown, to... Figure 6The diagram shows the playback and decoding order of a set of video frames encoded as I-frames. The playback order remains unchanged. Regarding the decoding order, since each video frame is an I-frame, the keyframe corresponding to time t1 can be directly decoded, effectively making each video frame the first one in the decoding order. This improves the decoding speed during frame playback, resulting in smoother video playback.

[0165] In this application embodiment, this frame playback method is referred to as frame buffer playback; it can be understood that the implementation of frame buffer playback requires normal playback in advance in order to cache the re-encoded I-frames of the same video file in the buffer space.

[0166] However, in practical applications, when a video frame at a certain moment is played, the buffer space may not have cached the I-frame at that moment; for example, normal playback may not have been performed before the I-frame is played; or, although normal playback was performed before the video frame at a certain moment is played, the normal playback process only played a segment of video and did not play the video frame at that moment in normal playback mode. For example, in Figure 2 (b) to Figure 2 In step (c), video frames from second 0 to second 5 were played using normal playback, therefore, the keyframes from second 0 to second 5 were cached in the buffer space. However... Figure 2 (d) to Figure 2 During the frame playback process shown in (e), it is necessary to display certain video frames between the 5th and 14th seconds in a frame playback manner, but the cache space does not cache the key frames between the 5th and 14th seconds of the video file.

[0167] In view of this, this application also provides another frame playback method. The electronic device sets up at least two decoders independent of the normal playback process for the frame playback process. If the buffer space does not cache the video frame at a certain moment pointed to by the frame playback scene, the idle decoder among the at least two set decoders is used to decode the video frame group containing that moment, avoiding insufficient decoding capability affecting the smoothness of the picture. This application refers to this frame playback method as concurrent frame decoding playback.

[0168] Reference Figure 8 This is a flowchart illustrating a video frame playback method provided in an embodiment of this application. The flowchart covers normal playback, frame buffer playback, and concurrent frame decoding playback.

[0169] During normal playback, the video file is parsed to obtain video and audio streams; the audio stream is decoded to obtain audio frames; the video stream is decoded to obtain video frames; the decoded video and audio frames are synchronized in time; the decoded video frames are rendered; and the decoded audio frames are rendered. Finally, audio playback and video display are also performed.

[0170] During normal playback, after the video is decoded to obtain video frames (including I-frames, P-frames, and B-frames), the P-frames and B-frames are re-encoded into I-frames; the original I-frames and the re-encoded I-frames are buffered.

[0171] During frame playback, the time indicated by the progress indicator line is first determined based on the dragging operation; then, the cache space is checked to see if there is a video frame corresponding to that time. The specific meaning of the video frame corresponding to a certain time can be found in the explanation of "the video frame corresponding to the 14th second" above.

[0172] If the video frame corresponding to that moment exists in the cache space, the frame cache playback process is executed: retrieve the video frame corresponding to that moment from the cache space; decode the video frame corresponding to that moment; render the video frame after decoding and display it subsequently.

[0173] If the cache does not contain the video frame corresponding to that moment, the concurrent decoding and playback process is executed: the video file is parsed, and the idle decoder in GOP decoder one or GOP decoder two is determined through scheduling; then the determined decoder is used for decoding; the video frame corresponding to that moment is obtained from the corresponding decoder through scheduling; finally, the video frame is rendered and subsequently displayed.

[0174] For a detailed description of the above-described normal playback, frame buffer playback, and concurrent frame decoding playback, please refer to the following embodiments; they will not be repeated here. (Refer to...) Figure 9 This is a technical architecture diagram of the playback engine corresponding to the video frame playback method provided in the embodiments of this application.

[0175] The application layer of an electronic device includes multiple applications, such as gallery applications, file management applications, video playback applications, and audio playback applications. This application's embodiments only list some applications related to audio and video playback. Of course, in practical applications, only some of the applications listed in the application layer may be used, and other applications not listed in the application layer may also be used to implement the video playback process.

[0176] The application framework layer has a basic platform, which contains a unified playback engine. This unified playback engine provides the foundation for upper-layer applications to implement audio and video playback.

[0177] The unified playback engine has an interface layer that is used to transfer data between upper-layer applications and other modules in the unified playback engine.

[0178] A standard media player is used to implement the standard playback process described in the above embodiments. The standard media player includes: a basic interface module, a standard playback module, an audio rendering module, and a video rendering module.

[0179] The basic interface module is used to provide a basic interface for the video playback process. This basic interface can be used by video playback applications that do not have their own interface. In this embodiment, the gallery application may not use the basic interface, but instead use the video playback interface provided by the gallery application itself.

[0180] The audio rendering module is used to call the audio rendering engine to render the audio stream in the video file during normal playback;

[0181] The video rendering module is used to call the graphics rendering engine to render the video stream in the video file during normal playback;

[0182] The general playback module is used to play audio and video files and call other modules (such as the codec module) to implement the normal playback process. In practical applications, the audio rendering module and the video rendering module can also be set up inside the general playback module.

[0183] The frame player is used to implement the frame playback process. The frame player provides two playback modes: frame buffering playback mode and frame concurrent decoding playback mode.

[0184] The frame buffer playback mode involves a frame extraction module and a buffer cleanup module. The frame extraction module is used to extract non-critical frames for re-encoding and buffering. The buffer cleanup module is used to clean up relevant information in the buffer (e.g., linked lists) at appropriate times.

[0185] The concurrent frame decoding playback mode involves a GOP parser and a GOP decoder, which are used to parse video files and decode video frames.

[0186] The decision module can be a module independent of the ordinary player and the frame player, or it can be a module included in the frame player. It is used to decide whether to perform concurrent frame decoding playback or frame buffer playback. In this embodiment, the decision module is used in the frame player as an example.

[0187] The media parsing module includes a metadata module and a thumbnail generation module. The metadata module is used to parse the video duration. The thumbnail generation module is used to generate multiple thumbnails as progress display frames based on the video duration in a multi-channel scenario.

[0188] The data extraction module includes: a subtitle extraction module, a video data extraction module, and an audio data extraction module. The subtitle extraction module extracts subtitle data from video files; the video data extraction module extracts video stream data from video files; and the audio data extraction module extracts audio stream data from video files.

[0189] The encoding and decoding module includes: a video encoding module, a video decoding module, an audio decoding module, and a hardware / software adaptive module.

[0190] The video encoding module, video decoding module, and audio decoding module can all be called by ordinary players and frame players, and at the same time call the encoding and decoding engine below to achieve the corresponding functions.

[0191] The hardware and software automatic adaptation module is used to implement corresponding functions using software modules when hardware resources are limited.

[0192] The application framework layer also includes an audio rendering engine, a graphics rendering engine, and a codec engine;

[0193] The audio rendering engine is invoked to provide audio rendering; the graphics rendering engine is invoked to provide video rendering; and the codec engine is invoked to decode and encode video frames.

[0194] Reference Figure 10 This is a timing diagram illustrating the preparation process before normal playback and frame playback of the electronic device provided in this application embodiment. This timing diagram can be compared with... Figure 2 (a) and Figure 2 (b) corresponds to this. Of course, video A in this example can be a video that corresponds to multiple recordings, or it can be a video that corresponds to multiple recordings.

[0195] S101, the gallery application displays a grid image, which includes the cover of video A.

[0196] The gallery app can display grid images that include not only the cover art for video A, but also the cover art for other videos, and of course, thumbnails of images. Figure 2 The grid diagram shown in (a) can serve as an example of the grid diagram used in this step.

[0197] S102, the gallery application receives a click operation on the cover of video A.

[0198] In this embodiment of the application, the electronic device displays... Figure 2 When the user clicks on the interface shown in (a), Figure 2 In the interface shown in (a), the grid display shows the cover art of any video. In response to this action, the electronic device displays the video playback interface for that video via the gallery application. This click action can be referenced... Figure 2 The user's finger tapping action in the interface shown in (a).

[0199] It should be noted that in practical applications, other modules of the electronic device may receive the click operation and then transmit the click operation to the gallery application. For the sake of simplicity, the intermediate steps are omitted in the embodiments of this application. Taking the gallery application as an example, it can receive the click operation on the cover of video A.

[0200] S103, after receiving a click on the cover of video A, the gallery application displays the video playback interface of video A.

[0201] The video playback interface of video A can be referenced. Figure 2 The interface shown in (b) of the video A interface includes a video playback area and a progress display area for displaying the playback progress.

[0202] It should be noted that in response to the click operation, the electronic device is simultaneously executing the process of displaying the video playback interface of video A, and also executing the corresponding processes in the subsequent steps. Therefore, there is no strict sequential order between steps S103 and S104 (and subsequent steps).

[0203] S104, after receiving the click operation on the cover of video A, the gallery application sends a request to the decision module to create the concurrent frame decoding playback module and the frame buffer playback module.

[0204] After receiving the creation requests for the concurrent frame decoding and playback module and the frame buffer playback module, the decision module creates the frame buffer playback module in the frame player and the concurrent frame decoding and playback module in the frame player.

[0205] In practical applications, S104 can also be broken down into two steps: one is sending a request to the decision module to create a concurrent frame decoding playback module, and the other is sending a request to the decision module to create a frame buffer playback module. This application does not limit the specific implementation process.

[0206] S105, after receiving the creation requests for the concurrent frame decoding playback module and the frame buffer playback module, the decision module creates the frame buffer playback module in the frame player.

[0207] S106, After the frame buffer playback module is successfully created, the frame buffer playback module creates a cache module.

[0208] S107 After receiving the creation requests for the concurrent frame decoding and playback module and the frame buffer playback module, the decision module creates the concurrent frame decoding and playback module in the frame player.

[0209] S108, after receiving the click operation on the cover of video A, the gallery application sends the file path of video A to the decision module.

[0210] In this embodiment of the application, the gallery application of the electronic device stores the file paths of various pictures and video files in the electronic device. The gallery application can display pictures and play videos based on the file paths of various pictures and video files.

[0211] In this embodiment of the application, steps S104 and S108 can be executed as two independent steps, or they can be combined into one step.

[0212] After receiving the file path of video A, the decision module sends the file path of video A to the normal playback module to prepare for normal playback of video A; it also sends the file path of video A to the frame concurrent decoding playback module to prepare for frame concurrent decoding playback of video A; and it sends the file path of video A to the frame buffer playback module to prepare for frame buffer playback of video A.

[0213] First, the file path of video A is sent to the normal playback module, and the subsequent preparation work of the normal playback module is described.

[0214] S109, after receiving the file path of video A, the decision module sends the file path of video A to the normal playback module.

[0215] S110, after receiving the file path of video A, the ordinary playback module obtains video A according to the file path and parses it to obtain the encoding format of video A. This application takes the encoding format of video A as H.264 as an example.

[0216] S111, the normal playback module creates an H.264 decoder for normal playback based on the encoding format of video A. This decoder is used for normal playback, so it can be referred to as the normal decoder.

[0217] The following describes how to send the file path of video A to the frame concurrent decoding and playback module, as well as the subsequent preparation work for the frame concurrent decoding and playback module.

[0218] S112, after receiving the file path of video A, the decision module sends the file path of video A to the created frame concurrent decoding and playback module.

[0219] S113, after receiving the file path of video A, the concurrent frame decoding and playback module obtains video A based on the file path and parses it to obtain the encoding format and file format of video A. This embodiment of the application uses H.264 encoding format and MP4 file format as an example.

[0220] S114, the concurrent frame decoding and playback module creates an MP4 parser based on the parsed file format, and simultaneously sends the file path of video A to the created parser. This parser can be referred to as the GOP parser.

[0221] S115, the concurrent frame decoding and playback module creates the first H.264 decoder based on the parsed encoding format.

[0222] S116, the concurrent frame decoding and playback module creates a second H.264 decoder based on the parsed encoding format.

[0223] For ease of description, the created decoders can be referred to as GOP decoder one and GOP decoder two, respectively. Of course, in practical applications, more decoders can be created to improve the decoding capabilities of electronic devices. If more decoders are created for frame playback, they can also be referred to in sequence as GOP decoder three, and so on.

[0224] The embodiments of this application can obtain a decoder through a one-step creation operation, and perform multiple creation operations as needed for the number of decoders to be created; or multiple decoders can be created simultaneously through a one-step creation operation.

[0225] In this embodiment, the parser is used to parse an MP4 file into a video stream; the decoder is used to decode the video stream into video pixel information.

[0226] The following describes sending the file path of video A to the frame buffer playback module and the subsequent preparation work of the frame buffer playback module.

[0227] S117, after receiving the file path of video A, the decision module sends the file path of video A to the created frame buffer playback module.

[0228] S118, after receiving the file path of video A, the frame buffer playback module obtains video A according to the file path and parses it to obtain the encoding format of video A. In this embodiment of the application, the encoding format is H.264.

[0229] S119, the frame buffer playback module creates an H.264 encoding module based on the parsed encoding format. This encoding module is used to re-encode P-frames and B-frames into I-frames.

[0230] S120, Frame buffer playback creates an H.264 decoding module based on the parsed encoding format. This decoding module is used to decode the I-frames in the buffer space to obtain YUV data during the frame buffer playback process.

[0231] After this step, the preparations for normal playback, concurrent frame decoding playback, and frame buffer playback are all completed. Upon receiving an operation for normal playback, the electronic device will execute normal video playback. Upon receiving an operation for frame playback, it can determine whether to execute concurrent frame decoding playback or frame buffer playback based on whether the buffer space stores the corresponding video frame at that moment.

[0232] This application's embodiments and Figure 2 The playback process in the illustrated scenario corresponds to, for example, receiving a normal playback operation first. (Refer to...) Figure 11 This is a timing diagram of a common video playback method provided in an embodiment of this application. This timing diagram can be compared with... Figure 2 The operation shown in (b) is displayed. Figure 2 (c) in the text corresponds to the process on the interface.

[0233] S201, the gallery application receives a click operation on a regular playback control.

[0234] In this embodiment of the application, the electronic device displays... Figure 2 When the user clicks on the video playback interface of video A shown in (b) above... Figure 2 The normal playback control in the video playback area shown in (b) can trigger the electronic device to execute the normal playback process of video A.

[0235] S202, after receiving a click operation on the normal playback control, the gallery application sends a normal playback request for video A to the decision module. The request carries the normal playback start time as second 0.

[0236] S203, after receiving the normal playback request of video A, the decision module sends a playback instruction for video A to the normal playback module. The playback instruction carries the start time of the playback as the 0th second.

[0237] S204: After receiving the playback command for video A, the normal playback module parses video A and obtains video frames in H.264 format.

[0238] S205, the normal playback module sends a video frame decoding request to the normal decoder.

[0239] In practical applications, a regular playback module can cache H.264 format video frames, and the decoding request carries the cache address. The regular decoder retrieves the H.264 format video frames from the cache address for decoding. Alternatively, the regular playback module can also carry H.264 format video frames in the decoding request, and the regular decoder decodes the received H.264 format video frames. This application does not limit the specific implementation method.

[0240] S206, a standard decoder decodes video frames to obtain decoded video data: YUV data.

[0241] The decoded video data is divided into I-frames, P-frames, and B-frames. After decoding the YUV data, it is rendered and displayed, and simultaneously re-encoded and cached.

[0242] First, the process of rendering and displaying YUV data is described.

[0243] S207: The ordinary decoder sends the decoded YUV data to the ordinary player.

[0244] Similarly, a regular decoder can also cache YUV data and then send the address of the cached YUV data to a regular player. This application does not limit the specific transmission method.

[0245] S208: After receiving the decoded YUV data, a regular media player renders the YUV data to obtain the video image.

[0246] S209: A regular media player sends the rendered video frame to the gallery application, along with the timestamp of the video frame.

[0247] S210, after receiving the video frame, the gallery application displays the received video frame in the video playback area, and at the same time updates the position of the progress display frame in the progress display area so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video frame.

[0248] The following describes the process of caching YUV data after re-encoding. After S206:

[0249] S211, the ordinary decoder sends the decoded YUV data to the decision module. At this time, the YUV data is an I-frame, P-frame, or B-frame.

[0250] Of course, in practical applications, ordinary decoders can also send YUV data in groups. For example, after decoding a group of YUV data, the decoder sends the group of YUV data to the decision module. In this case, the YUV data includes I-frames, P-frames, and B-frames. This application does not limit the specific implementation method.

[0251] S212, after receiving the YUV data, the decision module sends the YUV data to the frame buffer playback module.

[0252] S213, after receiving the YUV data, the frame buffer playback module sends the YUV data to the buffer module.

[0253] S214, after receiving YUV data, the cache module checks the cache space and finds that the video frame corresponding to the received YUV data does not exist.

[0254] In practical applications, YUV data itself carries a timestamp. In step S118, it has been determined that the currently displayed video playback interface is for video A. Therefore, the caching module can search the cache space to see if a video frame corresponding to the timestamp carried by the YUV data of video A exists. Of course, the decision module can also carry the identification information of video A when sending YUV data to the frame buffer playback module.

[0255] S215, if the buffer module does not have a video frame corresponding to the received YUV data in the buffer space, it sends YUV data to the encoding module.

[0256] S216, the encoding module encodes the received YUV data into an H.264 format I-frame.

[0257] In practical implementation, the YUV data received by the encoding module is the decoded data of I-frames, P-frames, and B-frames. Therefore, the encoding module also needs to encode the YUV data of I-frames into I-frame data in H.264 format; re-encode the YUV data of P-frames into I-frame data in H.264 format; and re-encode the YUV data of B-frames into I-frame data in H.264 format.

[0258] In practice, any encoder that can re-encode YUV data into H.264 format video frames can be selected, such as VJEncoder2.

[0259] S217, the encoding module sends the encoded I-frame to the buffer module.

[0260] S218, after receiving the I-frame sent by the encoding module, the buffer module buffers the received I-frame.

[0261] In practical applications, electronic devices need to continuously execute S206 to S218 to display each video frame on the video playback interface according to the playback time represented by the timestamp of each video frame in the video file; at the same time, the decoded video frames are re-encoded into I-frames and then buffered.

[0262] Reference Figure 2 As shown in (c), this is the interface displayed in the gallery application at the 5-second mark after the user clicks the normal playback control. The normal playback module performs parsing and rendering operations. Therefore, the normal playback module can also include a normal parser and the rendering thread used by a normal player.

[0263] In this embodiment of the application, when video A is played from second 0 to second 5 in normal playback mode, the caching module has already cached all the I-frame video frames of video A from second 0 to second 5. Some of these cached I-frames were I-frames in the previous encoding, some were P-frames in the previous encoding, and some were B-frames in the previous encoding.

[0264] As mentioned earlier, the electronic device also performs preparatory work for concurrent frame decoding and playback and frame buffering playback, so video A can also be played frame by frame through the gallery application.

[0265] This application embodiment is passed through Figure 12 Describe the concurrent frame decoding and playback process of video A. Figure 12 The timing diagram shown corresponds to Figure 2 (d) in Figure 2 The process of (e) in the middle.

[0266] Users can drag Figure 2 The progress display frame shown in (d) of the interface can be used to switch video A, which is playing in normal mode, to frame-by-frame playback, or to pause the video. Figure 2 In the interface shown in (c), video A is playing in normal mode. Video A is paused, and then the paused video A is resumed in frame-by-frame playback mode by dragging the progress bar to display the frames. This embodiment of the application uses the example of the user dragging the progress bar to the left to switch to frame-by-frame playback mode while video A is playing in normal mode.

[0267] S301, the gallery application receives a drag operation on the progress display frame.

[0268] In this embodiment, the dragging operation is a leftward dragging operation. (Refer to...) Figure 2 As shown in (d) in the figure.

[0269] For electronic devices including touchscreens, the touch driver can sample user touch operations at intervals T. This interval, also known as the deployment cycle or reporting cycle, is the period during which the touch driver collects user touch operations.

[0270] Taking a touch driver that collects data 240 times per second as an example, the period T = 1 / 240 = 4.16 ms, meaning the touch driver collects touch data once every 4.16 ms. If, during a certain period, the user presses the touchscreen with one or two fingers, the touch driver can determine the touch position of the single or two fingers based on voltage changes, etc.

[0271] For example, when a user wants to change the position of the progress display frame by dragging it to change the video frame displayed in the video playback area, the touch driver receives the user's click operation corresponding to the progress display frame and passes a down event to the application processor. The application processor passes the down event to the gallery application. After receiving the click event, the touch driver continues to receive the user's continuous drag operation corresponding to the progress display frame and passes a move event to the application processor. The application processor passes the move event to the gallery application.

[0272] S302, the gallery application determines the frame playback timestamp t1 of video A based on the drag position of the drag operation.

[0273] In this embodiment of the application, a continuous drag operation includes: the user's finger touching the progress display frame (down event), the user's finger moving (move event), and the user's finger releasing from the progress display frame (up event). Figure 7 The illustrated embodiment uses the drag operation corresponding to the first two touch points detected by the electronic device after the user's finger touches the progress display frame as an example to describe the frame playback process.

[0274] S303, when video A is playing in normal playback mode, the gallery application sends a normal playback pause command and a frame playback request to the decision module. The frame playback request carries the timestamp t1 of video A.

[0275] After receiving the normal playback pause command and frame playback request for video A, the decision module needs to pause the normal playback of video A and start the frame playback of video A.

[0276] S304, after receiving the normal playback pause command, the decision module sends a command to the normal playback module to pause the playback of video A.

[0277] S305: After receiving the instruction to pause playback of video A, the normal playback module sends a pause decoding instruction to the normal decoder.

[0278] S306: After receiving a temporary decoding instruction, the ordinary decoder pauses the decoding of video A.

[0279] S307, after receiving the instruction to pause playback of video A, the player pauses the rendering of video A.

[0280] When playing a frame of video A, the system first checks if the video frame corresponding to time t1 is cached. If it is, the frame caching playback process is executed; otherwise, the concurrent frame decoding playback process is executed. However, in practice, the concurrent frame decoding playback process is only one possible approach when the video frame corresponding to time t1 is not cached. In other applications, it can also be used for other frame playback scenarios where caching is not required.

[0281] S308 After the decision module receives the frame playback request, it first sends a video frame query request to the frame buffer playback module. The video frame query request carries the timestamp t1 of video A.

[0282] S309, after receiving the query request, the frame buffer playback module sends a query request for the video frame corresponding to time t1 of video A to the buffer module.

[0283] S310, the cache module queries the video frame corresponding to time t1 and the result is: no video frame at time t1.

[0284] In practical applications, when the caching module queries the video frame corresponding to time t1, it doesn't necessarily need to find the video frame at time t1 itself. Instead, it searches for video frames whose timestamp differs from time t1 by a certain threshold. This is because time t1 may not perfectly correspond to a single playback time (timestamp), but rather lie between two playback timestamps. Therefore, half the difference between the timestamps of two adjacent video frames can be set as this threshold.

[0285] S311, the buffer module sends information about the video frame at time t1 when there is no video A to the frame buffer playback module.

[0286] S312, after receiving information about a video frame at time t1 without video A, the frame buffer playback module sends a query result for a video frame at time t1 to the decision module.

[0287] S313, after receiving the query result of the video frame without time t1 sent by the frame buffer playback module, the decision module sends a frame concurrent decoding playback instruction to the frame concurrent decoding playback module, which carries the timestamp t1 of video A.

[0288] S314 After receiving the frame playback instruction, the concurrent frame decoding and playback module sends a parsing request to the GOP parser. The parsing request carries the timestamp t1 of video A.

[0289] S315 After receiving the frame playback instruction, the concurrent frame decoding and playback module sends a rendering request to the rendering thread. The rendering request carries the timestamp t1 of video A.

[0290] S316, after receiving the parsing request, the GOP parser parses video A and obtains the H.264 format video frames in the GOP group where the video frame with the current timestamp t1 in video A belongs. It records the timestamp ts of the first video frame and the timestamp te of the last video frame in the GOP group.

[0291] S317, the GOP parser sends a decoding request to the GOP decoding thread.

[0292] In practice, the GOP parser can buffer the decoded H.264 format video frames and send the buffer address with the decoding request, or it can directly send the parsed H.264 format video frames.

[0293] S318, the decoding thread sends a status acquisition request to GOP decoder 1.

[0294] S319, the decoding thread receives the status of GOP decoder one as idle.

[0295] In this embodiment, when GOP decoder one is in an idle state, decoding is performed by GOP decoder one; if GOP decoder one is not in an idle state, it is necessary to check whether other GOP decoders are in an idle state so as to decode by the decoder in an idle state, thereby decoding the video frame at a faster speed.

[0296] This example uses decoder one as an example, where decoder one is in an idle state.

[0297] S320, the decoding thread sends an H.264 format video frame or the buffer address of an H.264 format video frame to the idle GOP decoder.

[0298] The S321 GOP decoder decodes a pair of H.264 format video frames to obtain YUV data and buffers the YUV data.

[0299] The H.264 format video frames obtained by the GOP decoder are all video frames in the GOP group where the t1 video frame is located. They need to be decoded sequentially according to the decoding order until all video frames in the group are decoded. The decoded data is YUV data.

[0300] In practical applications, GOP decoder one can decode the YUV data of a video frame and store the YUV data of the video frame in buffer space one. Correspondingly, GOP decoder two has a corresponding buffer space two to store the YUV data of the video frames decoded by GOP decoder two. When GOP decoder two needs to decode, it can decode the YUV data of each video frame and store the YUV data of the video frame in buffer space two.

[0301] S322, the rendering thread finds the GOP decoder corresponding to the video frame with timestamp t1 in the received rendering request: GOP decoder one.

[0302] In practical applications, the rendering thread sends query requests to each GOP decoder in a polling manner. The query request carries a timestamp t1. The GOP decoder corresponding to the video frame with timestamp t1 is determined based on the query results sent by the decoder based on the query request.

[0303] The rendering thread can also search for the video frame at time t1 in the cache space corresponding to each GOP decoder in a polling manner until the video frame at time t1 is found in one of the cache spaces, thereby determining the GOP decoder corresponding to the video frame with timestamp t1.

[0304] The following example illustrates how the rendering thread sends query requests to the GOP decoder in a polling manner. In this embodiment, after receiving the rendering request:

[0305] The rendering thread first sends a query request (carrying time t1) to GOP decoder 1. GOP decoder 1 then checks the cache space 1 to confirm that there is no video frame at time t1, and sends this information back to the rendering thread. In practical applications, if GOP decoder 1 is not the decoder for the video frame with timestamp t1, then the cache space 1 will not contain the video frame with timestamp t1, and GOP decoder 1 will send this information back to the rendering thread. If GOP decoder 1 is the decoder for the video frame with timestamp t1, but has not yet decoded the video frame with timestamp t1 according to the decoding order, then the cache space 1 will also not contain the video frame with timestamp t1, and GOP decoder 1 will send this information back to the rendering thread. This is because when GOP decoder decodes video frames in a video frame group, it decodes them according to the decoding order. When GOP decoder 1 receives the query request, the decoder may not have yet decoded the video frame with timestamp t1. Therefore, if the cache space 1 also does not contain the video frame with timestamp t1, it will also need to send this information back to the rendering thread. If the rendering thread receives information about a video frame without timestamp t1 from GOP decoder 1, it sends a query request (carrying t1) to GOP decoder 2. GOP decoder 2 queries buffer space 2 to determine that there is no video frame at time t1, and then sends information about the absence of a video frame at time t1 to the rendering thread. Similarly, if GOP decoder 2 is not a decoder for video frames with timestamp t1, then buffer space 2 will not have a video frame at time t1, and GOP decoder 2 will send information about a video frame without timestamp t1 to the rendering thread. If GOP decoder 2 is a decoder for video frames with timestamp t1, but has not yet decoded the video frame at t1, then buffer space 2 will also not have a video frame at time t1, and GOP decoder 2 will send information about a video frame without timestamp t1 to the rendering thread.

[0306] If the rendering thread receives information about a video frame without timestamp t1 sent by GOP decoder 2, it continues to send a query request (carrying t1) to GOP decoder 1...;

[0307] Following this polling method, the process continues until the rendering thread receives information about a video frame with time t1 from any decoder. In this embodiment, taking the rendering thread receiving information about a video frame with time t1 from GOP decoder one as an example, the rendering thread can determine that the decoder that decodes the video frame with timestamp t1 is GOP decoder one.

[0308] S323, the rendering thread retrieves the YUV data at time t1 from the cache (cache space one) of the determined GOP decoder one.

[0309] S324, the rendering thread renders the acquired YUV data to obtain the video frame image after rendering at time t1.

[0310] S325, the rendering thread sends the video frame corresponding to time t1 to the gallery application.

[0311] S326, the gallery application displays the video frame of video A at time t1 in the video playback area, and at the same time updates the position of the progress display frame so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video frame.

[0312] As the user drags the progress display frame, the position of the user's finger touch detected by the electronic device may change. Consequently, the positional relationship between the progress indicator line and the progress display frame also changes, and the corresponding time of the progress indicator line also changes. Therefore, during the user's dragging of the progress display frame, it may be executing in a loop. Figure 12 The frame concurrent decoding and playback process shown continues until the corresponding video frame is found in the cache space.

[0313] In the specific implementation process, as the user drags the progress display frame, the position of the user's finger touch detected by the electronic device may change. Correspondingly, the positional relationship between the progress indicator line and the progress display frame also changes, and the time corresponding to the progress indicator line also changes. If the times corresponding to two consecutive detected touch positions are t1 and t2, and the GOP group of the video frame at time t2 is the same as the GOP group of the video frame at time t1, then during the frame playback to display the video frame at time t2, there is no need to repeat the decoding.

[0314] As an example, after receiving a parsing request, the GOP parser determines whether the video frame corresponding to the timestamp t2 in the parsing request is in the newly determined GOP group.

[0315] As mentioned earlier, in S316, the timestamps ts of the first video frame and the timestamp te of the last video frame in the GOP group to which the video frame at time t1 belongs have been recorded. It can be determined whether t2 is greater than or equal to ts and less than or equal to te; if t2 is greater than or equal to ts and less than or equal to te, it means that the video frame corresponding to t2 is in the latest GOP group; if t2 is less than ts or greater than te, it means that the video frame corresponding to t2 is not in the latest GOP group.

[0316] In this case, the GOP parser no longer parses video A. The rendering thread can find the GOP decoder corresponding to timestamp t2, which is GOP decoder one. Then, it continues to execute the subsequent processes.

[0317] As mentioned earlier, another scenario is where the cache space stores the video frame corresponding to a certain moment, and the video frame corresponding to that moment is displayed through the frame cache playback process. The following will demonstrate... Figure 13Describes the process of frame buffer playback, which corresponds to... Figure 3 (b) to Figure 3 The frame playback process is shown in (c). The user drags the progress bar to the right to display certain video frames before the 5th second. This application embodiment provides a process for implementing frame buffered playback at time t3, where time t3 is located at... Figure 3 Take the time between the 2nd and 5th seconds as an example.

[0318] S401, the gallery application receives a drag operation on the progress display frame.

[0319] This drag operation can be performed in Figure 3 The drag operation is based on the interface shown in (b). Before the drag operation, the user has already played the video from second 0 to second 5 of video A in normal playback mode, and correspondingly, the cache space has cached the keyframes between second 0 and second 5 of video A.

[0320] S402, the gallery application determines the timestamp t3 of the video A frame playback based on the drag position.

[0321] The process of detecting timestamp t3 can be referred to the above description.

[0322] S403, the gallery application sends a frame playback request to the decision module, which carries the timestamp t3 of video A.

[0323] S404 After receiving the frame playback request, the decision module first sends a video frame query request to the frame buffer playback module. The query request carries the timestamp t3 of video A.

[0324] S405, after receiving the query request, the frame buffer playback module sends a query request for the video frame at time t3 of video A to the buffer module.

[0325] S406, the query result of the cache module for video frames at time t3 of video A is: there are video frames for video A at time t3.

[0326] S407, the cache module sends the query result to the frame buffer playback module.

[0327] S408, the frame buffer playback module sends the query results to the decision module.

[0328] S409, after receiving the query result of the video frame at time t3 of video A, the decision module sends a frame buffer playback instruction to the frame buffer playback module, which carries the timestamp t3 of video A.

[0329] S410, after receiving the frame buffer playback request, the frame buffer playback module sends a request to the buffer module to obtain the video frame at time t3 of video A.

[0330] S411, after receiving the acquisition request, the cache module sends the video frame at time t3 to the decoding module.

[0331] In this application, the caching module can first obtain the video frame at time t3 from the cache space. Since the caching module caches keyframes corresponding to each time, the obtained video frame at time t3 is a keyframe at time t3, and there are no other video frames.

[0332] S412, after receiving the video frame at time t3, the decoding module decodes the H.264 format video frame at time t3 to obtain the YUV data at time t3. This step only requires decoding the video frame at time t3.

[0333] S413, the decoding module sends the YUV data at time t3 to the buffer module.

[0334] S414, after receiving the YUV data at time t3, the buffer module sends the YUV data at time t3 to the frame buffer playback module.

[0335] S415, after the frame buffer playback module receives the YUV data at time t3, it renders the YUV data at time t3 to obtain the video frame at time t3.

[0336] S416, the frame buffer playback module sends the video frame at time t3 to the gallery application.

[0337] S417 After receiving the video frame at time t3, the gallery application displays the video frame of video A at time t3 in the video playback area, and updates the position of the progress display frame so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video frame.

[0338] pass Figure 13 As shown in the timing diagram, by using frame buffering playback, only one video frame at time t3 needs to be decoded during decoding, which reduces the time required to obtain the video frame at time t3, thereby improving the smoothness of frame playback.

[0339] Of course, as the user drags the progress bar to display the frames, the electronic device may be looping. Figure 13 The frames shown are buffered and played until no corresponding video frame can be found in the buffer space.

[0340] In addition, after the user finishes dragging the progress display frame, that is, after the user lifts their finger from the area of ​​the progress display frame, the playback process of video A will continue to return to the state before switching to frame playback: normal playback of video A or paused playback of video A.

[0341] In the specific implementation, the first case is as follows: if the user drags the progress display frame while the video A is playing in normal mode, then after the user lifts their finger from the progress display frame area, the playback process of video A will switch from frame playback to normal playback mode, and will continue to play in normal mode from the moment indicated by the progress indicator line when the user lifts their finger from the progress display frame.

[0342] The second scenario: If, while playing video A in normal mode, the user first pauses the normal playback of video A, and then drags the progress display frame while video A is paused, then after the user lifts their finger from the progress display frame area, the playback of video A will switch from frame playback to a paused state. After the user clicks the normal playback control in the video playback area of ​​the interface, playback will resume in normal mode from the moment indicated by the progress indicator line (the moment indicated by the progress indicator line when the user lifts their finger from the progress display frame area).

[0343] This application takes the first case as an example. At time t4, after the user lifts their finger from the progress display frame area to release the progress display frame, video A switches from frame playback to normal playback.

[0344] Reference Figure 14 The time indicated by the progress indicator line on the progress display frame provided in this application embodiment during the release operation is time t4 (e.g., Figure 3 The timing diagram shown is for the transition from frame playback to normal playback in the case of the 2nd second (as illustrated). This diagram corresponds to... Figure 3 The normal playback process after (d) in the middle.

[0345] S501, the gallery application receives a release operation on the progress display frame.

[0346] In this embodiment, the release operation in S501 (after dragging, the user's finger is lifted off the screen) and the preceding... Figure 13 The drag operation is a continuous gesture operation, that is, the user drags the progress display frame with their finger until the time indicated by the progress indicator line reaches the 2nd second, and then lifts their hand. This lifting of the hand is the release operation on the progress display frame.

[0347] S502, in response to this operation, the gallery application sends the frame playback end information of video A and the end timestamp t3 to the decision module.

[0348] In practical implementation, a flag can be set in the gallery application or decision module to indicate whether video A is in normal playback or paused playback state before frame playback. If it is in paused playback state, the subsequent process of resuming normal playback will not be executed. If it is in playback state, since the decision module pauses normal playback when switching to frame playback (the flag is not modified when normal playback is paused while switching to frame playback). Therefore, it is necessary to resend the playback command to the normal player, that is, to execute the subsequent operations.

[0349] S503, after receiving the frame playback end information and timestamp t4, the decision module sends a video A start playback command to the normal playback module. This command carries the start playback time t4.

[0350] S504: After receiving the start playback command and timestamp t4 from video A, the normal playback module parses the file of video A based on time t4 to obtain video frames in H.264 format.

[0351] After S604, the normal playback process is performed and Figure 11 The steps following S204 are similar, except... Figure 11 Decoding starts from the 0th second of the video frame; Figure 14 After S504, playback begins from the video frame at time t4.

[0352] Of course, in practical applications, it may be necessary to start decoding from the keyframe in the GOP group where time t4 is located. Therefore, it may be necessary to start decoding from the keyframe preceding the video frame at time t4.

[0353] S505: The standard playback module sends a video frame decoding request to the standard decoder. A description of this decoding request can be found in [reference needed]. Figure 11 The description in S205.

[0354] The S506 is a standard decoder that decodes video frames to obtain decoded video data: YUV data.

[0355] The decoded video data is divided into I-frames, P-frames, and B-frames. After decoding the YUV data, it is rendered and displayed, and simultaneously re-encoded and cached.

[0356] First, the process of rendering and displaying YUV data is described.

[0357] S507: The ordinary decoder sends the decoded YUV data to the ordinary player.

[0358] Similarly, a regular decoder can also cache YUV data and then send the address of the cached YUV data to a regular player. This application does not limit the specific transmission method.

[0359] The S508, when a regular media player receives the decoded YUV data, renders the YUV data to obtain the video image.

[0360] S509: A regular media player sends rendered video frames to a gallery application, along with the timestamp of the video frames.

[0361] S510: After receiving a video frame, the gallery application displays the received video frame in the video playback area and updates the position of the progress display frame in the progress display area so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video frame.

[0362] In the specific implementation process, according to Figure 2 (b) and Figure 2 (c) and Figure 11 As shown, video frames from second 0 to 5 have been cached; if the drag operation is to the left (refer to...), the video will be cached. Figure 2 (d) and Figure 2 (e) in the middle, then according to Figure 12 Execute the concurrent frame decoding and playback process; if the drag operation is to the right (refer to...) Figure 3 (b) and Figure 3 (c) in the middle, then according to Figure 13 The execution frame buffer playback process is shown; if in Figure 3 If the user releases their finger from the progress display frame as shown in (c) (where the progress indicator line points to the 2nd second), then in the subsequent normal playback process, the video frames from the 2nd to the 5th second will no longer be re-encoded and cached; the video frames after the 5th second will continue to be re-encoded and cached.

[0363] The following description is detailed in Figure 3 The process of re-encoding in the normal playback flow after the user's finger is released from the progress display frame, based on the interface shown in (c). After S506:

[0364] S511, the ordinary decoder sends the decoded YUV data to the decision module. At this time, the YUV data is an I-frame, P-frame, or B-frame.

[0365] S512, after receiving the YUV data, the decision module sends the YUV data to the frame buffer playback module.

[0366] S513, after receiving YUV data, the frame buffer playback module sends YUV data to the buffer module.

[0367] S514 After receiving YUV data, the cache module checks whether the cache space contains the video frame corresponding to the received YUV data.

[0368] If the cache already contains the video frame corresponding to the received YUV data, the cache module discards the received YUV data.

[0369] S515: If the video frame corresponding to the received YUV data does not exist in the buffer space, send the YUV data to the encoding module.

[0370] S516, the encoding module encodes the received YUV data into an H.264 format I-frame.

[0371] S517, the encoding module sends the encoded I-frame to the buffer module.

[0372] S518, after the buffer module receives the I-frame sent by the encoding module, it buffers the received I-frame.

[0373] In practical applications, as normal playback proceeds, the electronic device needs to continuously execute S506 to S518 to display each video frame on the video playback interface according to the playback time represented by the timestamp of each video frame in the video file; at the same time, the decoded video frames are re-encoded and cached.

[0374] The above describes the normal playback process and frame playback process of video A. As mentioned earlier, when opening the video playback interface of video A, it is necessary to create a parser, decoder, etc.; similarly, when exiting the video playback interface of video A, it is also necessary to destroy the created parser, decoder, etc.

[0375] Reference Figure 15 This is a timing diagram showing the destruction of the parser and decoder when exiting the video playback interface of video A, as provided in this embodiment of the application.

[0376] S601, while displaying the video playback interface of video A, the gallery application receives a return operation.

[0377] In this embodiment, the return operation can be a swipe from the far right of the screen to the left on the interface, which can trigger the gallery application to return. Figure 2 The grid diagram interface shown in (a) is shown in the image.

[0378] S602, after receiving a return operation, the gallery application displays a grid chart, which is located on the interface for entering... Figure 2 The grid diagram shown in (b) is a previous screen before the video playback interface of video A, and includes the cover of video A.

[0379] S603, after receiving the return operation, the gallery application sends a destruction request to the decision module. This destruction request is used to destroy the various modules created when entering the video playback interface of video A.

[0380] It should be noted that S602 and S603 are not strictly sequential; they can be two parallel steps, or one step can be executed before the other.

[0381] Upon receiving a destruction request, the decision module destroys three modules within the frame player: those used for concurrent frame decoding and playback (frame concurrent decoding and playback module, GOP parser, and GOP decoder), those used for frame buffer playback (frame buffer playback module, buffer module, encoding module, and decoding module), and those in the general player (e.g., the general decoder). These three destruction operations can be performed simultaneously, or one operation can be performed first, followed by the others.

[0382] The modules related to destroying the frame buffer for playback (e.g., frame buffer playback module, buffer module, encoding module, and decoding module) are described in S604 to S609.

[0383] S604 After receiving the destruction request, the decision module sends the destruction requests for the buffer module, encoding module and decoding module to the frame buffer playback module.

[0384] S605, after receiving the destruction requests from the cache module, encoding module and decoding module, the frame buffer playback module destroys the cache module.

[0385] S606: After receiving the destruction requests from the buffer module, encoding module, and decoding module, the frame buffer playback module destroys the encoding module.

[0386] S607: After receiving the destruction requests from the buffer module, encoding module, and decoding module, the frame buffer playback module destroys the decoding module.

[0387] S608: After destroying the buffer module, encoding module, and decoding module, the frame buffer playback module sends a message to the decision module indicating that the buffer module, encoding module, and decoding module have been successfully destroyed.

[0388] S609, after receiving the information that the cache module, encoding module and decoding module have been successfully destroyed, the decision module destroys the frame buffer playback module.

[0389] The process of destroying the relevant modules for concurrent frame decoding and playback in the frame player (e.g., concurrent frame decoding and playback module, GOP parser, and GOP decoder) is described in S610 to S615.

[0390] S610, after receiving the destruction request, the decision module sends the destruction request for the parser and decoder to the frame concurrent decoding and playback module.

[0391] S611, after receiving the destruction requests of the parser and decoder, the concurrent frame decoding and playback module destroys the GOP parser.

[0392] S612, after receiving the destruction requests from the parser and decoder, the concurrent frame decoding playback module destroys GOP decoder one.

[0393] S613, after receiving the destruction requests from the parser and decoder, the concurrent frame decoding playback module destroys GOP decoder two.

[0394] There is no strict order in which the steps of destroying the GOP parser and GOP decoder are performed.

[0395] S614, after destroying the GOP parser, GOP decoder one, and GOP decoder two, the concurrent frame decoding and playback module sends a message to the decision module indicating that the parser and decoder have been successfully destroyed.

[0396] S615, after receiving the message from the concurrent frame decoder playback module that the parser and decoder have been successfully destroyed, the decision module destroys the concurrent frame decoder playback module.

[0397] The process of destroying relevant modules (e.g., common decoders) in a common player is described in S616 to S618.

[0398] S616 After receiving the destruction request, the decision module sends a decoder destruction request to the ordinary playback module.

[0399] S617: After receiving a request to destroy the decoder, the normal playback module destroys the normal decoder.

[0400] S618: After the normal playback module destroys the normal decoder, it sends a message to the decision module that the decoder has been successfully destroyed.

[0401] The above describes the destruction of modules related to frame buffer playback, concurrent frame decoding playback, and normal playback. After the modules related to the above three playback methods are destroyed, a message indicating successful destruction is sent to the gallery application.

[0402] S619, after destroying the concurrent frame decoding playback module, destroying the frame buffer playback module, and receiving the decoder destruction success message from the normal playback module, the decision module sends a destruction success message to the gallery application.

[0403] The video playback method provided in this application includes a normal playback process, a frame buffer playback process, and a frame concurrent decoding playback process. Specifically, it may include: an electronic device displaying a first interface of a first video, the first interface including a first area and a first control, the first area including a second control;

[0404] Normal playback process: After the electronic device receives the first operation on the second control, it displays the first video frame obtained by decoding the first encoded data of the first video in the first area according to the playback order. It moves the first control from the first position to the second position along the first trajectory and buffers the second encoded data of the first video frame. The decoding efficiency of the second encoded data is higher than that of the first encoded data.

[0405] Frame buffer playback process: The electronic device receives the first drag operation on the first control; during the process of dragging the first control to the third position through the first drag operation, the electronic device displays the second encoded data in the first area to decode the video frame, and the third position is on the first track.

[0406] Frame concurrent decoding and playback process: When the first control is dragged to the fourth position by the first drag operation, the electronic device obtains the third encoded data of the first video. When the first control is in the fourth position, the playback progress of the first video is the first moment. The encoded data corresponding to the first moment is the fourth encoded data in the third encoded data. The frame type in the third encoded data is the same as the frame type in the first encoded data.

[0407] The electronic device decodes the third encoded data using a first decoder;

[0408] After the electronic device decodes the fourth encoded data through the first decoder, it displays the video frame obtained from the decoding of the fourth encoded data in the first area.

[0409] When the first control is dragged from the fourth position to the fifth position through the first drag operation, the electronic device obtains the fifth encoded data of the first video. When the first control is in the fifth position, the playback progress of the first video is the second moment. The encoded data corresponding to the second moment is the sixth encoded data in the fifth encoded data. The frame type in the fifth encoded data is the same as the frame type in the first encoded data. The first video frame in the fifth encoded data and the last video frame in the third encoded data are adjacent video frames.

[0410] The electronic device decodes the fifth encoded data using a second decoder;

[0411] After the electronic device decodes the fifth encoded data through the second decoder, it displays the video frame obtained by decoding the sixth encoded data in the first area.

[0412] During the video playback process described above, the first video can be video A in the above embodiment, the first interface can be the video playback interface in the above embodiment, the first area can be the video playback area in the above embodiment, and the first control can be a progress display frame; the second control can be a control in the video playback area that triggers normal playback. The position of the first control is the position of a reference point on the progress display frame. The reference point can be any point on the progress display frame, or it can be a point corresponding to the touch point of the user's finger during dragging.

[0413] The first video frame in the example above can be Figure 2 The video frames shown are from second 0 to second 5. When the first control is in the first position, refer to... Figure 2 The position of the progress display frame shown in (b) is referenced; when the first control is in the second position, refer to Figure 2 The position of the progress display frame is shown in (c). The first encoded data consists of the I-frame, P-frame, and B-frame data from second 0 to second 5. The second encoded data consists of the I-frame, P-frame, and B-frame data from second 0 to second 5, all re-encoded into the corresponding encoded data after the I-frame.

[0414] The first drag operation can be Figure 3 (b) to Figure 3 The drag operation corresponding to (c) in the diagram is therefore, the third position can be... Figure 3 The position of the progress display frame is shown in (c). When displaying the video frame obtained by decoding the second encoded data, the video frame obtained by decoding the encoded data between the 2nd and 5th seconds of the second encoded data can be displayed.

[0415] In addition, the position passed during the dragging process includes the sixth position. For example, when the first control is in the sixth position, the playback progress is at the fourth moment. The fourth moment is between the 2nd and 5th seconds. Therefore, the video frame obtained by decoding the encoded data corresponding to the fourth moment after re-encoding can be displayed.

[0416] Of course, in accordance with Figure 2 When dragging as shown, it's possible to drag from the 5th second to the 14th second. The position of the first control is... Figure 2 (d) and Figure 2 The positions shown in (e) are in the diagram. Figure 2 (d) in Figure 2During process (e), the position of the first control passes through the fourth and fifth positions. When the first control is at the fourth position, the playback progress of the first video is at the first moment, which is a moment between the 5th and 14th seconds. When the first control is at the fifth position, the playback progress of the first video is at the second moment, which is also a moment between the 5th and 14th seconds. The video frame group containing the first moment and the video frame group containing the second moment are two adjacent video frame groups, that is, the last video frame in the video frame group containing the first moment and the first video frame in the video frame group containing the second moment are adjacent video frames. The two adjacent video frame groups may be decoded using different decoders. For example, if the previous video frame group uses the first decoder, then the next video frame group uses the second decoder; of course, an idle decoder can also be used for decoding.

[0417] In addition, during the process of displaying the video frame of the first video triggered by the first operation of the second control, the video frame to be displayed can be obtained by decoding the first encoded data of the first video; during the process of displaying the video frame of the first video triggered by the drag operation (referred to as frame playback), the video frame to be displayed can be obtained by decoding the second encoded data; since the electronic device is more efficient at decoding the second encoded data than at decoding the first encoded data, the decoding efficiency in the frame playback scenario can be improved, thereby improving the smoothness of the video in the frame playback scenario.

[0418] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0419] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0420] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0421] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0422] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0423] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0424] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0425] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video playback method, characterized in that, include: An electronic device displays a first interface of a first video, the first interface including a first area and a first control, the first area including a second control; After receiving the first operation on the second control, the electronic device displays a first video frame in the first area, which is obtained by decoding the first encoded data of the first video. The frame type in the first encoded data of the first video frame includes keyframes and non-keyframes. During the process of displaying the first video frame in the first area, the electronic device controls the first control to move from a first position to a second position along a first trajectory; When the first control is moved to the second position, the electronic device receives a first drag operation on the first control; During the process of moving the first control from the second position to the third position through the first drag operation, the electronic device displays a video frame obtained by decoding the second encoded data of the first video in the first area. The third position is on the first trajectory and the third position is located between the first position and the second position. The frame type in the second encoded data of the first video frame is a keyframe.

2. The method as described in claim 1, characterized in that, During the process of displaying the first video frame in the first area, the method further includes: The electronic device caches the second encoded data of the first video frame.

3. The method as described in claim 2, characterized in that, The method further includes: When the first control is dragged from the third position to the fourth position, the electronic device obtains the third encoded data of the first video. The fourth position is not on the first trajectory. When the first control is at the fourth position, the playback progress of the first video is the first moment. The encoded data corresponding to the first moment is the fourth encoded data in the third encoded data. The frame type in the third encoded data is the same as the frame type in the first encoded data. The electronic device decodes the third encoded data using a first decoder; After the electronic device decodes the fourth encoded data through the first decoder, it displays the video frame obtained by decoding the fourth encoded data in the first area.

4. The method as described in claim 3, characterized in that, The method further includes: When the first control is dragged from the fourth position to the fifth position, the electronic device acquires the fifth encoded data of the first video. The fifth position is not on the first trajectory. When the first control is at the fifth position, the playback progress of the first video is the second moment. The encoded data corresponding to the second moment is the sixth encoded data in the fifth encoded data. The frame type in the fifth encoded data is the same as the frame type in the first encoded data. The first video frame in the fifth encoded data and the last video frame in the third encoded data are adjacent video frames. The electronic device decodes the fifth encoded data using a second decoder; After the electronic device completes the decoding of the fifth encoded data through the second decoder, the electronic device displays the video frame obtained by decoding the sixth encoded data in the first area.

5. The method according to any one of claims 2 to 4, characterized in that, The process of displaying the first video frame obtained by decoding the first encoded data of the first video in the first area includes: For each piece of first encoded data, the electronic device decodes the first encoded data using a third decoder to obtain YUV data; The electronic device displays video frames corresponding to the YUV data obtained by decoding the first encoded data in the first area.

6. The method as described in claim 5, characterized in that, The process of the electronic device caching the second encoded data of the first video frame includes: For each first encoded data, the electronic device encodes the YUV data obtained from the first encoded data to obtain the second encoded data corresponding to the first encoded data; The electronic device caches the second encoded data corresponding to the first encoded data and the timestamp of the first encoded data in the first storage space.

7. The method as described in claim 6, characterized in that, The first encoded data includes encoded data with a timestamp of a third moment. Before the electronic device encodes the YUV data obtained from the first encoded data at the third moment, the method further includes: The electronic device determines that the second encoded data of the third moment of the first video does not exist in the first storage space.

8. The method according to any one of claims 2 to 7, characterized in that, During the process of moving the first control from the second position to the third position through the first drag operation, the electronic device displays the video frame obtained by decoding the second encoded data in the first area, including: When the first control is moved from the second position to the sixth position by the first drag operation, the electronic device displays a video frame obtained by decoding the seventh encoded data in the second encoded data in the first area. When the first control is in the sixth position, the playback progress of the first video is at the fourth moment, the seventh encoded data is the encoded data corresponding to the fourth moment, and the sixth position is on the first trajectory.

9. The method as described in claim 8, characterized in that, When the first control is moved from the second position to the sixth position via the first drag operation, the electronic device displays a video frame obtained by decoding the seventh encoded data in the second encoded data in the first area, including: When the first control is moved from the second position to the sixth position through the first drag operation, the electronic device determines that the playback progress of the first video is the fourth moment; The electronic device queried that the seventh encoded data corresponding to the fourth moment was stored in the first storage space; The electronic device retrieves the seventh encoded data from the first storage space; The electronic device obtains the seventh encoded data from the first storage space; The electronic device decodes the seventh encoded data; The electronic device displays the video frame obtained by decoding the seventh encoded data in the first area.

10. The method as described in claim 3 or 4, characterized in that, When the first control is dragged to the fourth position via the first drag operation, the electronic device acquires the third encoded data of the first video, including: When the first control is dragged to the fourth position via the first drag operation, the electronic device determines that the playback progress of the first video is the first moment. The electronic device did not find the encoded data corresponding to the first moment in the first storage space; The electronic device acquires third encoded data of the first moment of the first video.

11. The method as described in claim 10, characterized in that, The encoded data corresponding to the first moment is encoded data whose difference between the timestamp and the first moment is less than or equal to a first value, where the first value is half the difference between the timestamps of two adjacent video frames.

12. The method according to any one of claims 1 to 11, characterized in that, The first interface also includes a second area, in which the first control is located and can be moved. The first region and the second region do not overlap, or the second region is within the first region.

13. An electronic device, characterized in that, The device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when executed by the one or more processors, causes the electronic device to perform the method as described in any one of claims 1-12.

14. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-12.

Images