Video playing method, electronic equipment, storage medium and chip

Abstract

The invention provides a video playing method, electronic equipment, a storage medium and a chip, and relates to the technical field of terminal equipment, and the method can create at least two decoders specially used for a frame playing scene; in a frame playing scene, a decoder which is in an idle state at present decodes a video frame to be displayed at present, so that the decoding speed of the video frame is improved; after the decoding speed of the video frame is increased, the video frame is played more smoothly; through the method, the fluency of the video picture of the frame playing scene can be improved.

Description

Technical Field

[0001] This application relates to the technical field of terminal devices, and in particular, to a video playing method, an electronic device, a storage medium, and a chip. Background Art

[0002] The gallery application in an electronic device can provide a picture preview interface, and a user can view each picture stored in the electronic device through the picture preview interface of the gallery application; the gallery application also provides a video playing interface, and a user can play each video stored in the electronic device through the video playing interface in the gallery application.

[0003] The video playing interface can also display the playing progress of a video. For example, the playing progress of the video can be displayed in ways such as a progress bar, a progress ring, a progress display frame, etc.; a dragging operation of the user on the progress bar, the progress ring, or the progress display frame can trigger the electronic device to perform frame playing on the video. However, in the frame playing scenario of the video, the display process of the video picture is relatively stuck and not smooth. Summary of the Invention

[0004] This application provides a video playing method, an electronic device, a storage medium, and a chip, which can reduce the stuck situation of the video picture in the frame playing scenario of the video and provide the smoothness of the video picture in the frame playing scenario.

[0005] To achieve the above object, the first aspect of this application adopts the following technical solution:

[0006] The first aspect of this application provides a video playing method, which is applied to an electronic device. The method includes:

[0007] Display a first interface. The first interface includes a first area and a first control. The first area includes a first video frame of a first video.

[0008] When a dragging operation on the first control is received:

[0009] In response to the dragging operation, move the first control from a first position to a second position along a first trajectory.

[0010] During the process of moving the first control from the first position to the second position along the first trajectory, decode a first segment of video frames of the first video through a first decoder, decode a second segment of video frames of the first video through a second decoder, display the video frames in the decoded first segment of video frames in the first area, and display the video frames in the decoded second segment of video frames in the first area. The first segment of video frames and the second segment of video frames are two consecutive segments of video frames in the first video.

[0011] When displaying video frames in a segment of video frames through the dragging operation of the first control, no matter displaying one video frame or multiple video frames in the segment of video frames, each video frame in the segment of video frames needs to be decoded, so the decoding efficiency is low, and when the next video frame in two consecutive segments of video frames needs to be displayed, the decoder is still decoding the video frames in the first segment of video frames, resulting in failure to decode and display the video frames in the next segment of video frames in time, so the playback process is relatively jerky; in the present application, at least two decoders are set for the process of displaying video frames through the dragging operation of the first control, and the two segments of video frames are decoded by different decoders. Even if the first decoder has not completed decoding the first segment of video frames, it does not affect the timely decoding and display of the second segment of video frames by the second decoder, thereby improving the smoothness of the video screen in the frame playback scene.

[0012] As another implementation of the first aspect, a play control is displayed on the first video frame in the first interface, and the method further includes:

[0013] Upon receiving an action on the playback controls:

[0014] In response to an operation on a playback control, decoding video frames of the first video by a third decoder, and displaying the decoded video frames in the first video in a first area;

[0015] During the process of displaying a third video frame in the decoded first video in the first area, the first control moves from a first position to a second position along a first track, and the third video frame includes the first video frame and the second video frame.

[0016] The embodiment of the present application also provides a third decoder for playing through a playback control. The third decoder is used in a normal playback scenario, so that the normal playback process and the frame playback process are independent of each other and do not interfere with each other. That is, the introduction of the frame playback process does not require changing the native process of normal playback, and the development process is simple.

[0017] Of course, in practical applications, the third decoder may be the same decoder as the first decoder; or the third decoder may be the same decoder as the second decoder.

[0018] As another implementation of the first aspect, displaying the decoded video frames in the first segment of video frames in the first region, and displaying the decoded video frames in the second segment of video frames in the second region, specifically:

[0019] After displaying the decoded video frames in the first section of video frames in the first region, the decoded video frames in the second section of video frames are displayed in the first region.

[0020] During the process of dragging the first control, the display order of the video frames in the first video frame and the video frames in the second video frame is related to the dragging process. When the first passed position corresponds to the video frame in the first video frame, the decoded video frame in the first video frame is displayed first, and the dragging position is related to the displayed video frame.

[0021] As another implementation of the first aspect, the method further includes:

[0022] In response to the drag operation, continue to move the first control from the second position to the third position;

[0023] In the process of moving the first control from the second position to the third position, the fourth video frame of the first video is decoded by the first decoder, and the video frames in the decoded fourth video frame are displayed in the first area, and the fourth video frame and the second video frame are two consecutive video frames in the first video.

[0024] In the present application, when the second video frame is decoded by the second decoder, the third video frame continuous with the second video frame is decoded by the first decoder; these two consecutive video frames are decoded by different decoders, even if the second decoder has not completed decoding the second video frame, it does not affect the first decoder to decode and display the third video frame in a timely manner, thereby improving the smoothness of the video picture in the frame playback scene.

[0025] As another implementation of the first aspect, decoding a first video frame of a first video by a first decoder includes:

[0026] When the state of the first decoder is an idle state, a first video frame of the first video is decoded by the first decoder.

[0027] In the present application, which decoder to use for decoding can be determined based on the state of the decoder. If decoding is performed using an idle decoder, the decoding efficiency can be improved, thereby improving the smoothness of the picture during frame playback.

[0028] As another implementation of the first aspect, when the state of the first decoder is an idle state, decoding a first video frame of the first video by the first decoder includes:

[0029] Get the state of the first decoder;

[0030] Determining that the state of the first decoder is an idle state;

[0031] A first video frame of a first video is decoded by a first decoder in an idle state.

[0032] As another implementation of the first aspect, decoding a second video frame of the first video by a second decoder includes:

[0033] When the state of the first decoder is in use and the state of the second decoder is in idle, the second video frames of the first video are decoded by the second decoder in the idle state.

[0034] In the present application, when the first decoder is not idle, if the second decoder is in the idle state, decoding is performed by the idle second decoder, which can improve the decoding efficiency and thus improve the smoothness of the picture during frame playback.

[0035] As another implementation of the first aspect, when the state of the first decoder is in use and the state of the second decoder is in idle, decoding the second video frames of the first video by the second decoder in the idle state includes:

[0036] Obtain the state of the first decoder;

[0037] Determine that the state of the first decoder is in use;

[0038] When it is determined that the state of the first decoder is in use, obtain the state of the second decoder;

[0039] Determine that the state of the second decoder is in idle;

[0040] Decode the second video frames of the first video by the second decoder in the idle state.

[0041] As another implementation of the first aspect, during the process of the first control moving from the first position along the first trajectory to the second position, the method further includes:

[0042] When the first control moves to the fourth position, display the second video frames in the first video in the first area. When the first control is at the fourth position, the video frame corresponding to the playback progress of the first video is the second video frame, and the fourth position is on the first trajectory and is located between the first position and the second position.

[0043] In the present application, in the frame playback scenario, when it is detected which position the first control is located at, display the video frames related to that position, so that the displayed video frames are displayed along with the user's dragging operation, which is convenient for the user to view the video frames at different moments and improves the user experience.

[0044] As another implementation of the first aspect, the first interface further includes a first indicator, the first indicator is located above the first control and its position on the screen remains unchanged; when the first control is at the fourth position, the video frame corresponding to the playback progress indicated by the position of the first indicator on the first control is the second video frame, and the position of the first control is the position of a point on the first control.

[0045] As another implementation of the first aspect, when the first control is at the fourth position, the playback progress of the first video is at the first moment. When the first control moves to the fourth position, the second video frame in the first video is displayed in the first area, including:

[0046] When the first control moves to the fourth position, parse the first video to obtain the first segment of video frames where the first moment is located;

[0047] Use the first decoder to sequentially decode the video frames in the first segment of video frames according to the decoding order of the video frames in the first segment of video frames;

[0048] After completing the decoding of the second video frame corresponding to the second moment in the first segment of video frames, display the second video frame in the first area.

[0049] In this application, when it is necessary to display the video frame corresponding to a certain moment, it is necessary to decode each video frame in the segment of video frames where the video frame corresponding to this moment is located through a decoder. Of course, after decoding the video frame corresponding to this moment, the video frame corresponding to this moment can be displayed; if there are still undecoded video frames in this segment of video frames, the other undecoded video frames in this segment of video frames can also be continued to be decoded.

[0050] As another implementation of the first aspect, during the process of the first control moving from the first position to the second position along the first trajectory, the method further includes:

[0051] When the first control moves from the fourth position to the fifth position, determine that the playback progress of the first video is at the second moment. The fifth position is on the first trajectory and is between the fourth position and the second position;

[0052] If the video frame at the second moment is within the first segment of video frames, obtain the third video frame corresponding to the second moment from the first decoder;

[0053] After obtaining the third video frame from the first decoder, display the third video frame in the first area.

[0054] In this application, if the video frame at the playback moment corresponding to the currently detected position and the video frame at the playback moment corresponding to the previously detected position are within the same segment of video frames; since the entire segment of video frames will be decoded by the decoder when displaying the video frame at the playback moment corresponding to the previous position, therefore, the video frame at the playback moment corresponding to the currently detected position can be directly obtained from the corresponding decoder. The process of decoding and displaying is faster and smoother without it.

[0055] As another implementation of the first aspect, during the process of the first control moving from the first position to the second position along the first trajectory, the method further includes:

[0056] When the first control moves from the fourth position to the sixth position, determine that the playback progress of the first video is the third moment. The sixth position is on the first track and is between the fourth position and the second position;

[0057] If the fourth video frame corresponding to the third moment is not in the first segment of video frames, parse the first video to obtain the second segment of video frames where the fourth video frame corresponding to the third moment is located;

[0058] Use the second decoder to decode the video frames in the second segment of video frames in the decoding order of the video frames in the second segment of video frames;

[0059] After completing the decoding of the fourth video frame in the second segment of video frames, display the fourth video frame in the first area.

[0060] If the video frames corresponding to the playback moment of the currently detected position and the video frames corresponding to the previous detected position are not in the same segment of video frames, then it is necessary to use another idle decoder to decode the video frames corresponding to the currently detected position. Similarly, it is also necessary to decode each video frame in the segment of video frames where the video frame is located to improve the decoding efficiency.

[0061] As another implementation of the first aspect, the video frames of the first video include multiple I frames, and there are at least one P frame and at least one B frame between any two I frames. The first segment of video frames includes the i-th I frame, and the P frames and B frames between the i-th I frame and the (i + 1)-th I frame. The second segment of video frames includes the (i + 1)-th I frame, and the P frames and B frames between the (i + 1)-th I frame and the (i + 2)-th I frame. The timestamp of the i-th I frame is less than the timestamp of the (i + 1)-th I frame, and the timestamp of the (i + 1)-th I frame is less than the timestamp of the (i + 2)-th I frame, where i is a natural number greater than or equal to 1.

[0062] As another implementation of the first aspect, the first interface further includes a second area. The first control is located within the second area and can move within the second area;

[0063] The first area and the second area do not overlap, or the second area is within the first area.

[0064] As another implementation of the first aspect, displaying the video frames in the decoded first segment of video frames in the first area is specifically:

[0065] Display some or all of the video frames in the decoded first segment of video frames in the first area.

[0066] Displaying the video frames in the decoded second segment of video frames in the first area is specifically:

[0067] Display some or all of the video frames in the decoded second segment of video frames in the first area;

[0068] Display a video frame in the decoded first video in the first area, specifically including:

[0069] Display some or all of the video frames in the decoded first video in the first area.

[0070] In a second aspect, there is provided an electronic device including a processor, which is configured to call a computer program stored in a memory to implement the method according to any one of the first aspects of the present application.

[0071] In a third aspect, there is provided a chip including a processor, the processor being coupled to a memory, and the processor executing a computer program stored in the memory so that the electronic device implements the method according to any one of the first aspects of the present application.

[0072] In a fourth aspect, there is provided a computer-readable storage medium storing a computer program, and when the computer instructions run on an electronic device, the electronic device implements the method according to any one of the first aspects of the present application.

[0073] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the computer program product runs on a device, the electronic device is caused to execute the method according to any one of the first aspects of the present application.

[0074] It can be understood that the beneficial effects of the above second to fifth aspects can refer to the relevant descriptions in the above first aspect, and will not be elaborated herein. Description of the Drawings

[0075] Figure 1 It is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application;

[0076] Figure 2 It is a schematic diagram of an interface during a video playback process provided by an embodiment of the present application;

[0077] Figure 3 It is a schematic diagram of a video playback interface provided by an embodiment of the present application;

[0078] Figure 4 It is another schematic diagram of an interface during a video playback process provided by an embodiment of the present application;

[0079] Figure 5 It is a schematic diagram of an architecture of video playback provided by an embodiment of the present application;

[0080] Figure 6 It is a timing diagram for preparing for normal playback and frame playback provided by an embodiment of the present application;

[0081] Figure 7Schematic diagram of a video frame in a video frame group provided by an embodiment of the present application;

[0082] Figure 8 Timing diagram of a normal playback process of a video provided by an embodiment of the present application;

[0083] Figure 9 Timing diagram of a frame playback process of a video provided by an embodiment of the present application;

[0084] Figure 10 Timing diagram of a frame playback process of another video provided by an embodiment of the present application;

[0085] Figure 11 Timing diagram of a frame playback process of another video provided by an embodiment of the present application;

[0086] Figure 12 Timing diagram of ending frame playback and continuing normal playback provided by an embodiment of the present application;

[0087] Figure 13 For Figure 3 and Figure 4 Timing diagram of the preparation work before normal playback and frame playback in the scenario shown;

[0088] Figure 14 For Figure 3 Timing diagram of the progress display frame switching to the expanded state in the scenario shown;

[0089] Figure 15 Timing diagram of exiting the video playback interface provided by an embodiment of the present application;

[0090] Figure 16 Schematic diagram of a dragging operation on the progress display frame provided by an embodiment of the present application;

[0091] Figure 17 Schematic diagram of multiple contact positions corresponding to the dragging operation provided by an embodiment of the present application;

[0092] Figure 18 For Figure 16 Schematic diagram of the frame playback process corresponding to the dragging operation shown. Detailed implementation manners

[0093] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details.

[0094] It should be understood that, as used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0095] It should also be understood that, in the embodiments of this application, "one or more" means one, two, or more than two; "and / or" describes the association relationship of associated objects and indicates that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0096] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used for differential description and should not be construed as indicating or implying relative importance.

[0097] Referring to "one embodiment" or "some embodiments" described in the specification of this application means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0098] A video playing method provided by an embodiment of this application can be applied to an electronic device, which can be an electronic device such as a tablet computer, a mobile phone, a wearable device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The specific type of the electronic device is not limited in the embodiments of this application.

[0099] Figure 1The structural schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 1211, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0101] The processor 110 may include one or more processing units. For example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the video frame playback method in the embodiments of this application.

[0102] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0103] The internal memory 1211 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 1211. The internal memory 1211 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function (such as an image playback function, etc.). The touch sensor 180K, also called a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch screen, also called a "touch screen", is composed of the touch sensor 180K and the display screen 194. The touch sensor 180K is used to detect a touch operation acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194. For example, click operations on controls provided in the embodiments of the present application, drag operations on progress display frames, etc. can all be recognized by the touch sensor.

[0104] The electronic device 100 implements a display function through a GPU, the display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. For example, the process of rendering YUV data in the embodiments of the present application can be implemented through the GPU.

[0105] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1. For example, in the embodiments of the present application Figure 2 or Figure 3 the interfaces shown are all displayed by the display.

[0106] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats. For example, the video codec can decode the video frames in the H.264 format provided in the embodiments of the present application, or re-encode them into H.264 format video frames all of which are key frames.

[0107] The embodiments of the present application do not particularly limit the specific structure of the execution subject of a video playing method. As long as it can communicate according to a video playing method provided in the embodiments of the present application by running the code recording the video playing method of the embodiments of the present application. For example, the execution subject of a video playing method provided in the embodiments of the present application can be a functional module in the electronic device that can call and execute a program, or a communication device applied to the electronic device, such as a chip.

[0108] The user can view each picture stored in the electronic device and play each video stored in the electronic device through the gallery application in the electronic device.

[0109] Refer to Figure 2 which is a schematic diagram of the interface for playing the videos stored in the electronic device through the gallery application provided in the embodiments of the present application.

[0110] Refer to Figure 2In (a), it is a grid view of pictures and videos stored in an electronic device shown in the gallery application of the electronic device; this grid view can display pictures and videos. Among them, the cover and duration of the video are shown in the grid where the video is located, and the thumbnail of the picture is shown in the grid where the picture is located.

[0111] This grid view can be the interface view shown by the gallery application after the user clicks the icon of the gallery application on the system desktop to open the gallery application, or it can also be the interface view triggered by the gallery application through more steps of operations after the user clicks the icon of the gallery application on the system desktop to open the gallery application.

[0112] When this grid view displays the pictures and / or videos stored in the electronic device, it can be presented in various ways. For example, Figure 2 The way shown in (a) is to display the pictures and videos in the photo group in reverse chronological order; in actual applications, the gallery application can also set multiple object groups. For example, through Figure 2 The control 10 shown in (a) can be used to view multiple groupings in the album: image group, video group, screenshot and screen recording group, etc. The user can also display pictures through the image group in the gallery application, display videos through the video group in the gallery application, and display screenshot pictures and screen recording videos through the screenshot and screen recording group in the gallery application, etc. Each grouping can also display the corresponding pictures and / or videos in the form of the above-mentioned grid view.

[0113] The embodiments of the present application do not limit how many steps of user operations are required from the system desktop to the display of the grid view of the gallery application. Similarly, the way the grid view displays pictures and videos is not limited either.

[0114] Referring to Figure 2 In (b), it is the video playback interface of video A displayed by the electronic device in response to the operation after the user clicks the cover of video A in the grid view shown in (a). This video playback interface includes a video playback area 11 and a progress display area 12. Figure 2 Among them, the video playback area 11 is used to display the video picture during video playback. After entering the video playback interface from the grid view interface, a control 111 is displayed in the video playback area. When the user clicks this control 111, the electronic device responds to the click operation on this control 111 and plays the video picture from the 0th second in this video playback area 11; in the video pause state, this control 111 is also displayed in the video playback area 11. The user can click this control 111, and the electronic device responds to the click operation on this control 111, and the video playback area 11 plays the video picture from the current paused moment.

[0115]

[0116] ​The progress display area 12 is used to display the video playback progress. The progress display area 12 includes a progress display frame 121 and a progress indicator line 122. The position of the progress indicator line 122 can remain unchanged, and the user can drag the progress display frame 121 located below the progress indicator line 122, so that the progress display frame 121 moves left and right without the progress indicator line 122 moving; the position of the progress indicator line 122 above the progress display frame on the progress display frame represents the current video playback progress.

[0117] As an example, the total playback duration of the current video is 15 seconds, and the total length of the progress display frame of the current video is the length represented by L pixel points. Based on these two parameters, the playback duration corresponding to a unit length (or the length corresponding to a unit time) can be obtained. For example, divide the playback duration 15 by the total length L. Therefore, if the leftmost edge position of the progress display frame (the position corresponding to the first pixel point) is set to correspond to the 0th second of the video, then different positions of the progress indicator line on the progress display frame correspond to different playback times. For example, if the distance between the position of the progress indicator line on the progress display frame and the leftmost edge position of the set progress display frame is X, then the playback time corresponding to the progress indicator line is: X * 15 / L.

[0118] For the convenience of subsequent description, the embodiments of the present application are described in the way of "the moment corresponding to the progress indicator line".

[0119] Due to the space limitation of the progress display area 12, some video frames or partial areas of some video frames in the progress display frame 121 can be displayed in the progress display area 12. As shown in (b) of Figure 2 The progress display frame 121 displayed in the progress display area 12 includes: thumbnails of 3 complete video frames and a partial area of the thumbnail of a video frame.

[0120] Generally, the picture played in the video playback area 11 is the picture of the video frame corresponding to the moment corresponding to the progress indicator line 122 in the progress display area 12.

[0121] In a specific implementation, there is a situation where the moment corresponding to the progress indicator line 122 is the moment between the playback moments corresponding to two adjacent video frames, that is, there is no video frame corresponding to the moment of the progress indicator line 122 in the video file.

[0122] As an example, among the video frames arranged in ascending order of the playback time (denoted as the timestamp), the playback time of the i-th video frame is ti, and the playback time of the (i + 1)-th video frame is t(i+1); the time t corresponding to the progress indicator line 122 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval from the time t corresponding to the progress indicator line 122 can be used as the video frame corresponding to the time of the progress indicator line 122.

[0123] Of course, in practical applications, if there is no video frame in the video file whose timestamp is the same as the time corresponding to the progress indicator line, the previous video frame (the video frame corresponding to ti) of the time t corresponding to the progress indicator line 122 can also be used as the video frame corresponding to the time of the progress indicator line 122; or the next video frame (the video frame corresponding to t(i+1)) of the time t corresponding to the progress indicator line 122 can be used as the video frame corresponding to the time of the progress indicator line 122. In practical applications, there are many ways for the progress display area 12 to display the playback progress. For example, the video playback progress can also be displayed by means of a progress bar, or the video playback progress can be displayed in the form of a progress ring. This application provides two playback methods for video playback: normal playback and frame playback. The embodiments of this application only use Figure 2 the progress display area shown in (b) of

[0124] as an example to describe the difference between normal playback and frame playback. Figure 2 Referring to (c) of Figure 2 when the user clicks on the control 111 (this control can also be denoted as the playback control) in the interface shown in (b) of Figure 2 a screenshot during the process of the electronic device playing video A in response to this operation. This screenshot shows that it has played to the 5th second of video A, and the video screen corresponding to the 5th second is displayed in the video playback area 11; at the same time, the progress display frame 121 in the progress display area 12 moves its position so that the time corresponding to the progress indicator line 122 is the 5th second. Among them, the playback process from Figure 2 the (b) of

[0125] A video file includes multiple video frames, each with a timestamp that records the playback moment of the video frame during the playback of the video file. As an example, in ascending order of timestamps, the video frames in the video file are arranged as: video frame 0 (timestamp t0), video frame 1 (timestamp t1), video frame 2 (timestamp t2), video frame 3 (timestamp t3), video frame 4 (timestamp t4), video frame 5 (timestamp t5), video frame 6 (timestamp t6), video frame 7 (timestamp t7), and so on. Normal playback is as follows: in ascending order of timestamps, each video frame is sequentially displayed in the video playback area according to the corresponding playback moment of the timestamp.

[0126] As an example of the normal playback process, at the moment t0 when playback starts, video frame 0 is displayed in video playback area 11, at the moment t1 after playback starts, video frame 1 is displayed in video playback area 11, at the moment t2 after playback starts, video frame 2 is displayed in video playback area 11, at the moment t3 after playback starts, video frame 3 is displayed in video playback area 11, at the moment t4 after playback starts, video frame 4 is displayed in video playback area 11, at the moment t5 after playback starts, video frame 5 is displayed in video playback area 11, at the moment t6 after playback starts, video frame 6 is displayed in video playback area 11, at the moment t7 after playback starts, video frame 7 is displayed in video playback area 11, and so on.

[0127] Combined with Figure 2 in (b) to Figure 2 in (c), the normal playback process of video A is shown: starting from the video frame at the 0th second of video A, each video frame is sequentially displayed according to the playback moment indicated by the timestamp; Figure 2 in (c) is a schematic diagram of the interface showing the video frame at the 5th second. Referring to Figure 2 in (d), the user drags the Figure 2 progress display frame of the interface shown in (c).

[0128] In response to Figure 2 the drag operation shown in (d), the electronic device displays the corresponding screen in the video playback area for the drag operation.

[0129] As an example, referring to Figure 2 in (e), when the user drags the progress display frame so that the time corresponding to the progress indicator line is the 14th second, the video frame corresponding to the 14th second is displayed in the video playback area. The video frame corresponding to the 14th second can be the video frame with a timestamp of the 14th second in video A; it can also be the video frame with the timestamp closest to the 14th second; it can also be the video frame with the closest time interval to the 14th second before the 14th second; or the video frame with the closest time interval to the 14th second after the 14th second.

[0130] It should be noted that in the subsequent embodiments, the specific meaning of the description method of "the video frame corresponding to a certain moment" can be referred to the detailed explanation of "the video frame corresponding to the 14th second", and it will not be repeated hereinafter. From Figure 2 in (d) to Figure 2 the playback process shown in (e) is the frame playback process of the video.

[0131] During the process of the user dragging the progress to display frame 121, a relative position is formed between the progress indicator line 122 and the progress display frame 121, and the position of the progress indicator line 122 on the progress display frame 121 is related to the moment corresponding to the progress indicator line 122.

[0132] In the specific implementation process, assume that before the user moves in the area where the progress display frame 121 on the screen is located, the time indicated by the progress indicator line 122 is t1. When the user moves left or right in the area where the progress display frame on the screen is located, the electronic device can detect the real-time contact position of the user at a certain time interval; the electronic device determines the moving direction and moving distance of the user according to the continuous real-time contact positions. The electronic device determines the time variation amount △t according to the moving distance of the user; determines the operation method for calculating the real-time time t2 according to the moving direction of the user. For example, if the moving direction is left, the starting time t1 minus the time variation amount △t is used to obtain the real-time time t2; if the moving direction is right, the starting time t1 plus the time variation amount △t is used to obtain the real-time time t2. After obtaining the real-time time t2, the electronic device searches for the video frame corresponding to the real-time time t2 and displays the video frame corresponding to t2 in the video playback area.

[0133] On the other hand, the electronic device obtains the moving direction of the progress display frame according to the moving direction determined by the real-time contact position, and determines the moving distance of the progress display frame according to the moving distance of the real-time contact position; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2. As another example, the electronic device also determines the moving direction (if t2 is greater than t1, then move left, if t2 is less than t1, then move right) and moving distance of the progress display frame according to the starting time t1 and the real-time time t2; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0134] It should be noted that the above process is only an example. In actual applications, there can be multiple ways to determine the time corresponding to the progress indicator line 122, the position of the progress display frame 121, and the video frame displayed in the video playback area 11 during the process of the user dragging the progress display frame 121.

[0135] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. The user may move it to the left, to the right, drag it faster, or drag it slower. Correspondingly, the video frame images displayed in the video playback area 11 are random.

[0136] As an example, before the user moves in the area where the progress display frame is located on the screen, the start time indicated by the progress indicator line is t1; during the process of the user dragging the progress to display the frame, the electronic device detects the real-time touch point position of the user at regular time intervals, so as to obtain the real-time times corresponding to multiple consecutive time periods: t2, t3, t7, t4, t3. In the video playback area, the video frames are played in the following order: starting from the video frame corresponding to the t1 moment as the starting video frame, sequentially playing the video frames corresponding to the t2 moment, the t3 moment, the t7 moment, the t4 moment, and the t3 moment. It can be understood that the process of the electronic device playing the video in the video playback area triggered by the user clicking the control 11 is normal playback; the process of the electronic device displaying the video frame corresponding to the moment of the progress indicator line in the video playback area triggered by the user dragging the progress to display the frame is frame playback; in addition, during the normal playback process of video A, if the user drags the progress display frame and then releases it from the area where the progress display frame is located, the electronic device continues to play video A in the normal playback mode.

[0137] In specific implementation, the embodiment of the present application further provides another schematic diagram of the interface for playing the video stored in the electronic device through the gallery application.

[0138] The embodiment of the present application provides a video recording mode: one recording gets multiple results. That is, when performing video recording, photos can also be generated simultaneously (for example, photos automatically captured by the electronic device according to the picture content, and photos manually taken by the user through the photo-taking control on the video recording interface), so as to obtain the photos taken during the video recording process. The recorded video can be associated with the photos taken during the video recording process; similarly, when viewing the video through the gallery application, in the video playback interface, the photos taken during the video recording process can also be displayed in the progress display area.

[0139] Refer to Figure 3 in (a), and this interface can refer to the description in Figure 2 in (a).

[0140] Refer to Figure 3 in (b), which is the video playback interface of video A displayed by the electronic device in response to the operation after the user clicks the cover of video A in the grid diagram shown in Figure 3 in (a). This video playback interface is similar to the video playback interface shown in Figure 2 , the difference lies in the content displayed in the progress display area 12.

[0141] The progress display frame 121 in the progress display area 12 is in a compressed state. The progress display frame in the compressed state can display two video frames, three video frames, etc. Refer to Figure 2 . As shown in (b) of

[0142] , the progress display frame 121 in the compressed state includes a thumbnail of the first frame of the video and a thumbnail of the middle frame. Of course, in practical applications, thumbnails of other frames of the video can also be displayed, such as a thumbnail of the first frame and a thumbnail of the last frame. A thumbnail 123 of a photo obtained during the video recording process is displayed on the right side of the progress display frame 121.

[0143] The user can perform operations (such as clicking or double-clicking, etc.) above the progress display frame to trigger the progress display frame to switch from the compressed state to the expanded state. The progress display frame in the expanded state can include more video frame thumbnails than the progress display frame in the compressed state. The number of video frames displayed in the progress display frame in the expanded state is related to the duration of the video. For example, the longer the duration of the video, the more video frames the progress display frame includes. Figure 3 As shown in (c) of Figure 3 , it is a schematic diagram of the progress display frame in the expanded state displayed by the electronic device in response to the operation after the user clicks on the progress display frame 121 in the compressed state shown in (b) of

[0144] Figure 3 . The interface shown in (c) of Figure 2 is the same as the interface shown in (b) of Figure 3 . Therefore, the normal playback process and the frame playback process based on the interface shown in (c) of Figure 2 can refer to the description from (b) to Figure 2 in (e) of Figure 4 , and will not be elaborated here. Refer to

[0145] Figure 4 . For (a) in Figure 4 and (b) in Figure 3 , it can refer to the description in (a) in Figure 3 and (b) in

[0146] In the interface shown in (b) of Figure 4 , the progress display frame is in the compressed state. The user clicks on the control 111 in the interface shown in (b) of Figure 4 , triggering the electronic device to play video A in the normal playback mode.

[0147] Referring to Figure 4 in (c), it is an interface diagram during the process of playing Video A after the user clicks on the control 111 in the interface shown in (b) of Figure 4 . This interface diagram corresponds to an interface during the normal playback process when the progress display frame is in a compressed state. This interface diagram indicates that it has played to the 5th second of Video A, and the video screen corresponding to the 5th second is displayed in the video playback area; at the same time, the position of the progress display frame in the progress display area moves, so that the moment corresponding to the progress indicator line is the 5th second. Among them, from Figure 4 in (b) to Figure 4 in (c), the playback process shown is the normal playback process of the video.

[0148] Referring to Figure 4 in (d), the user drags the progress display frame on the basis of the interface shown in (c) of Figure 4 .

[0149] Referring to Figure 4 in (e), the electronic device responds to the drag operation shown in (d) of Figure 4 and displays the video frame corresponding to the 14th second at the time corresponding to the progress indicator line in the video playback area. Among them, the playback moment corresponding to the progress indicator line in the progress display area is the 14th second, and the video frame corresponding to the 14th second is displayed in the video playback area.

[0150] It should be noted that if the progress display frame is in a compressed state and the user drags the progress display frame, the progress display frame remains in a compressed state; of course, the user can also, as shown in Figure 3 , first trigger the progress display frame to change from the compressed state to the expanded state through an operation, and then drag the progress display frame in the expanded state, then the progress display frame remains in the expanded state.

[0151] Each operation shown in the embodiments of the present application is an example. In actual applications, other operations can also be set according to specific situations. For example, some click operations can also be double-click operations or other air gestures, etc. Some drag operations can also be air gestures, etc. Similarly, some operation examples in the subsequent embodiments can also be other operations.

[0152] Currently, in the normal playback scenario, each video frame is displayed in ascending order of the time stamps of the video frames. Therefore, parsing, decoding, rendering, etc. can be performed in advance according to the order of the video frames in the video file. Since parsing, decoding, and rendering are performed in advance, the video playback in the normal playback scenario is relatively smooth. In the frame playback scenario, as mentioned above, the process of the user dragging the progress bar to display frames is an uncertain process. The user may move left, move right, drag quickly, or drag slowly. Correspondingly, the video frame images displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display frames, the electronic device needs to determine the moment corresponding to the progress indicator line based on the detected contact position in real time, and then perform operations such as parsing, decoding, and rendering on the video frame corresponding to that moment. When the decoding ability of the electronic device is weak, the frame playback scenario often appears to be stuck and not smooth during playback.

[0153] Combined with Figure 2 Example, from Figure 2 in (b) to Figure 2 in (c) is the normal playback scenario, and the video images displayed in the video playback area are relatively smooth; from Figure 2 in (d) to Figure 2 in (e) is the frame playback scenario, and the video images displayed in the video playback area are relatively stuck and not smooth.

[0154] An embodiment of the present application provides a method for frame playback of a video. The electronic device sets at least two decoders independent of the normal playback process for the frame playback process, and uses the set at least two decoders to perform decoding of the frame playback process, improving the decoding ability, so as to perform frame playback of the video more smoothly.

[0155] Referring to Figure 5 is the technical architecture diagram of the playback engine corresponding to the method for frame playback of a video provided by an embodiment of the present application.

[0156] There are multiple applications in the application layer of the electronic device, such as a gallery application, a file management application, a video playback application, an audio playback application, etc. An embodiment of the present application only lists some applications related to audio and video playback. Of course, in actual applications, only some of the applications listed in the application layer may be used, and other applications not listed in the application layer may also be used to implement the video playback process.

[0157] There is a basic middle platform in the application framework layer, and there is a unified playback engine in this basic middle platform. The unified playback engine is used to provide a basis for upper-layer applications to implement audio and video playback.

[0158] The unified playback engine has an interface layer, and this interface layer is used to transfer data between upper-layer applications and other modules in the unified playback engine.

[0159] The ordinary player is used to implement the ordinary playback process described in the above embodiments. The ordinary player includes: a basic interface module, an ordinary playback module, an audio rendering module, and a video rendering module.

[0160] The basic interface module is used to provide the basic interface for the video playback process. This basic interface can be used by video playback applications without their own interfaces. In the embodiments of this application, the gallery application can choose not to use this basic interface, but use the video playback interface provided by the gallery application itself;

[0161] The audio rendering module is used to call the audio rendering engine to render the audio stream in the video file during the ordinary playback process;

[0162] The video rendering module is used to call the graphics rendering engine to render the video stream in the video file during the ordinary playback process;

[0163] The ordinary playback module is used to execute the playback of audio and video files and call other modules (such as the encoding and decoding module) to implement the ordinary playback process, etc. In practical applications, the audio rendering module and the video rendering module can also be set inside the ordinary playback module.

[0164] The frame player is used to implement the frame playback process. The frame player provides two playback modes: the frame buffer playback mode and the frame concurrent decoding playback mode.

[0165] The frame buffer playback mode involves a frame extraction module and a cache cleaning module; the frame extraction module is used to extract non-key frames for re-encoding and caching the extracted non-key frames; the cache cleaning module is used to clean the relevant cache information (such as linked lists) at an appropriate time;

[0166] The frame concurrent decoding playback mode involves a group of pictures (GOP) parser and a GOP decoder, which are used to implement the parsing of video files and the decoding of video frames.

[0167] The decision module can be a module independent of the ordinary player and the frame player, or a module included in the frame player, and is used to decide whether to perform frame concurrent decoding playback or frame buffer playback. In the embodiments of this application, the decision module is taken as an example in the frame player.

[0168] The media parsing module includes: a metadata module and a thumbnail generation module; the metadata module is used to parse the video duration; the thumbnail generation module is used to generate multiple thumbnails as progress display frames according to the video duration in the one-to-many scenario;

[0169] The data extraction module includes: a subtitle extraction module, a video data extraction module, and an audio data extraction module. The subtitle extraction module is used to extract subtitle data of a video file; the video data extraction module is used to extract video stream data of a video file, and the audio data extraction module is used to extract audio stream data of a video file.

[0170] The encoding and decoding module includes: a video encoding module, a video decoding module, an audio decoding module, and a software and hardware adaptive module.

[0171] Among them, the video encoding module, the video decoding module, and the audio decoding module can all be called by a common player and a frame player, and simultaneously call the encoding and decoding engine below to implement corresponding functions.

[0172] The software and hardware automatic adaptation module is used to implement corresponding functions using software modules when hardware resources are limited.

[0173] The application framework layer further includes an audio rendering engine, a graphics rendering engine, and an encoding and decoding engine;

[0174] The audio rendering engine is used to be called to provide audio rendering; the graphics rendering engine is used to be called to provide picture rendering; the encoding and decoding engine is used to be called to decode and encode video frames.

[0175] Refer to Figure 6 , which is a timing diagram of the preparation process before the common playback and frame playback of the electronic device provided in the embodiment of the present application. This timing diagram can be corresponding to (a) in Figure 2 and (b) in Figure 2 . Of course, video A in this example can be the video corresponding to "one recording, multiple uses", or it can not be the video corresponding to "one recording, multiple uses".

[0176] S101, the gallery application displays a grid view, and the grid view includes the cover of video A.

[0177] The grid view displayed by the gallery application can not only include the cover of video A, but also include the covers of other videos, and of course, can also include thumbnails of pictures. Figure 2 The grid view shown in the interface shown in (a) in

[0178] S102, the gallery application receives a click operation on the cover of video A.

[0179] In the embodiment of the present application, when the electronic device displays the interface shown in (a) in Figure 2 , the user clicks on the cover of any video shown in the grid view in the interface shown in (a) in Figure 2 . In response to this operation, the electronic device displays the video playback interface of this video through the gallery application. This click operation can be referred toFigure 2 The click operation of the user's finger in the interface shown in (a) of

[0180] It should be noted that in actual applications, after other modules of the electronic device receive the click operation, they may transmit the click operation to the gallery application. For the sake of simplicity in description, the intermediate steps are omitted in the embodiments of this application. Taking the example that the gallery application can receive the click operation on the cover of Video A

[0181] S103. After the gallery application receives the click operation on the cover of Video A, it displays the video playback interface of Video A.

[0182] The video playback interface of Video A can refer to Figure 2 the interface shown in (b) of , and the video playback interface of Video A includes a video playback area and a progress display area for displaying the playback progress.

[0183] It should be noted that in response to the click operation, on the one hand, the electronic device is in the process of displaying the video playback interface of Video A; on the other hand, it is also in the process of executing the subsequent steps. Therefore, there is no strict sequence between step S103 and step S104 (and subsequent steps).

[0184] S104. After the gallery application receives the click operation on the cover of Video A, it sends a creation request for the frame concurrent decoding and playback module to the decision module.

[0185] S105. After the decision module receives the creation request for the frame concurrent decoding and playback module, it creates the frame concurrent decoding and playback module in the frame player.

[0186] S106. After the gallery application receives the click operation on the cover of Video A, it sends the file path of Video A to the decision module.

[0187] In the embodiments of this application, the file paths of each picture and each video file in the electronic device are stored in the gallery application of the electronic device, and the gallery application can display pictures and play videos based on the file paths of each picture and video file.

[0188] In the embodiments of this application, step S104 and step S106 can be executed as two independent steps, or can be combined into one step.

[0189] After the decision module receives the file path of Video A, on the one hand, it sends the file path of Video A to the normal playback module to enable the normal playback module to perform the preparatory work for the normal playback of Video A; on the other hand, it sends the file path of Video A to the frame concurrent decoding and playback module to enable the frame concurrent decoding and playback module to perform the preparatory work for the frame playback of Video A.

[0190] First, describe the file path of Video A sent to the general playback module and the subsequent preparation work of the general playback module.

[0191] S107, after the decision module receives the file path of Video A, it sends the file path of Video A to the general playback module.

[0192] S108, after the general playback module receives the file path of Video A, it obtains Video A according to the file path and parses to obtain the encoding format of Video A. In this application, the encoding format of Video A is taken as H.264 as an example.

[0193] S109, the general playback module creates an H.264 decoder for general playback according to the encoding format of Video A. Since this decoder is used for general playback, it can be denoted as a general decoder or a third decoder.

[0194] Next, describe the file path of Video A sent to the frame concurrent decoding and playback module and the subsequent preparation work of the frame concurrent decoding and playback module.

[0195] S110, after the decision module receives the file path of Video A, it sends the file path of Video A to the created frame concurrent decoding and playback module.

[0196] S111, after the frame concurrent decoding and playback module receives the file path of Video A, it obtains Video A according to the file path and parses to obtain the encoding format and file format of Video A. In the embodiment of this application, the encoding format is H.264 and the file format is MP4 as an example.

[0197] S112, the frame concurrent decoding and playback module creates an MP4 parser according to the parsed file format, and at the same time sends the file path of Video A to the created parser, and this parser can be denoted as a GOP parser.

[0198] S113, the frame concurrent decoding and playback module creates the first H.264 decoder according to the parsed encoding format.

[0199] S114, the frame concurrent decoding and playback module creates the second H.264 decoder according to the parsed encoding format.

[0200] For ease of description, the created decoders can be respectively denoted as GOP decoder one and GOP decoder two. Of course, in actual applications, more decoders can also be created to improve the decoding ability of the electronic device. If more decoders for frame playback are created, they can also be denoted as GOP decoder three... and so on in sequence.

[0201] In the embodiment of this application, one decoder can be obtained through one creation operation, and multiple creation operations are executed according to the number of decoders to be created; or multiple decoders can be created simultaneously through one creation operation.

[0202] In the embodiments of the present application, a parser is used to parse an MP4 format file into a video stream; a decoder is used to decode the video stream into video pixel information.

[0203] Refer to Figure 7 As shown, a GOP decoder is used to decode a group of video frames (which can be denoted as a video frame group or a GOP group). Each group of video frames includes a key frame (I-frame) at the beginning and P-frames and B-frames between the key frames (I-frames). That is, each group of video frames can include, in the playback order: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. Whether it is an I-frame, or a P-frame and a B-frame, they are all video frames. When the GOP decoder decodes a video frame at a certain moment, it does not directly decode the video frame at that moment, but decodes the video frame group where the video frame at that moment is located. When decoding a video frame group, it decodes one by one in the decoding order.

[0204] After this step, the preparations for normal playback and the preparations for frame playback are both completed. After the electronic device receives an operation for normal playback, it will perform normal playback of the video. After receiving an operation for frame playback, it will perform frame playback of the video.

[0205] The embodiments of the present application correspond to the playback process of the scenario shown in Figure 2 Taking the case of first receiving an operation for normal playback as an example. Refer to Figure 8 , which is a timing diagram of a method for normal playback of a video provided by the embodiments of the present application. This timing diagram can correspond to the process from the operation shown in (b) in Figure 2 to the display of the corresponding interface shown in (c) in Figure 2 .

[0206] S201, the gallery application receives a click operation on the normal playback control.

[0207] In the embodiments of the present application, when the electronic device displays the video playback interface of Video A shown in (b) in Figure 2 , and the user clicks on the normal playback control in the video playback area shown in (b) in Figure 2 , it can trigger the electronic device to execute the normal playback process of Video A.

[0208] S202, after the gallery application receives the click operation on the normal playback control, it sends a normal playback request for Video A to the decision module. This request carries the start time of normal playback as the 0th second.

[0209] S203, after the decision module receives the normal playback request for Video A, it sends a playback instruction for Video A to the normal playback module. This playback instruction carries the start playback time as the 0th second.

[0210] In S204, after the normal playback module receives the playback instruction of Video A, it parses Video A to obtain video frames in H.264 format.

[0211] S205, the normal playback module sends a decoding request for the video frames to the normal decoder.

[0212] In practical applications, the normal playback module can cache the video frames in H.264 format. The decoding request carries the cache address, and the normal decoder obtains the video frames in H.264 format from the cache address for decoding; of course, the normal playback module can also carry the video frames in H.264 format in the decoding request, and the normal decoder decodes the received video frames in H.264 format. This application does not limit the specific implementation method.

[0213] S206, the normal decoder decodes the video frames to obtain the decoded video data: YUV data.

[0214] The decoded video data is divided into I frames, P frames, and B frames.

[0215] S207, the normal decoder sends the decoded YUV data to the normal player.

[0216] Similarly, the normal decoder can also cache the YUV data and then send the address of the cached YUV data to the normal player. The embodiments of this application do not limit the specific transmission method.

[0217] S208, after the normal player receives the decoded YUV data, it renders the YUV data to obtain a video picture.

[0218] S209, the normal player sends the rendered video picture to the gallery application and carries the timestamp of the video picture.

[0219] S210, after the gallery application receives the video picture, it displays the received video picture in the video playback area, and at the same time updates the position of the progress display frame in the progress display area, so that the moment corresponding to the progress indicator line is consistent with the timestamp of the received video picture.

[0220] In practical applications, the electronic device needs to continuously execute S206 to S210, so as to display each video frame in the video playback interface according to the playback moment represented by the timestamp of each video frame in the video file.

[0221] Refer to Figure 2 As shown in (c) therein, it is the interface diagram when the gallery application displays the video playback interface at the 4th second after the user clicks the normal playback control. Figure 8The normal playback process of Video A shown does not involve the relevant modules in the frame player. At the same time, the normal playback module performs operations such as parsing and rendering. Therefore, the normal playback module may also include a normal parser and a rendering thread used by the normal player.

[0222] As described above, the electronic device has also performed the preparation work for frame playback. Therefore, Video A can also be played in frames through the gallery application.

[0223] In the embodiments of this application, Figure 9 by describing the frame playback process of Video A, Figure 9 the timing diagram shown corresponds to Figure 2 the process from (d) to Figure 2 (e) therein.

[0224] The user can drag Figure 2 the progress display frame in the interface shown in (d) to switch Video A that is being played in the normal mode to frame playback. The user can also first pause Figure 2 Video A that is being played in the normal mode in the interface shown in (c), so that Video A is in a paused state, and then continue to display the video frames in Video A in frame playback mode by dragging the progress display frame. In the embodiments of this application, when Video A is played in the normal mode, the example of the user dragging the progress display frame to switch to the frame playback mode is taken.

[0225] S301. The gallery application receives the drag operation of the progress display frame.

[0226] For an electronic device including a touch screen, the touch driver can sample the user's touch operations according to a period T. The above period, also known as the polling period and reporting point period, is the period for the touch driver to collect the user's touch operations.

[0227] Taking the touch driver sampling 240 times per second as an example. The period T = 1 / 240 = 4.16 ms, that is, the touch driver samples the touch data every 4.16 ms. If in a certain period, the user presses the touch screen with a single finger or two fingers, then the touch driver can determine the touch position of the single finger or two fingers according to voltage changes, etc.

[0228] Exemplarily, when the user wants to change the video frames displayed in the video playback area by dragging the position of the progress display frame, the touch driver receives the click operation of the user corresponding to the progress display frame and passes the down event to the application processor. The application processor passes the down event to the gallery application. After receiving the click event, the touch driver continues to receive the continuous drag operation of the user corresponding to the progress display frame and passes the move event to the application processor. The application processor passes the move event to the gallery application.

[0229] S302, the gallery application determines the timestamp t1 for frame playback of Video A according to the drag position of the drag operation.

[0230] In the embodiments of the present application, a continuous drag operation includes: the user's finger touches the progress display frame (down event), the user's finger moves (move event), and the user's finger releases from the progress display frame (up event). Figure 7 The illustrated embodiment describes the frame playback process by taking the drag operation corresponding to the first two contact positions detected by the electronic device after the user's finger touches the progress display frame as an example.

[0231] S303, when Video A is played in the normal playback mode, the gallery application sends a normal playback pause instruction and a frame playback request to the decision module, and the frame playback request carries the timestamp t1 of Video A.

[0232] After receiving the normal playback pause instruction and the frame playback request of Video A, the decision module needs to pause the normal playback of Video A on the one hand and start the frame playback of Video A on the other hand.

[0233] S304, after receiving the normal playback pause instruction, the decision module sends an instruction to pause playing Video A to the normal playback module.

[0234] S305, after receiving the instruction to pause playing Video A, the normal playback module sends a pause decoding instruction to the normal decoder.

[0235] S306, after receiving the temporary decoding instruction, the normal decoder pauses the decoding of Video A.

[0236] S307, after receiving the instruction to pause playing Video A, the player pauses the rendering of Video A.

[0237] S308, after receiving the frame playback request, the decision module sends a frame concurrent decoding and playback instruction to the frame concurrent decoding and playback module, and the instruction carries the timestamp t1 of Video A.

[0238] S309, after receiving the frame playback instruction, the frame concurrent decoding and playback module sends a parsing request to the GOP parser, and the parsing request carries the timestamp t1 of Video A.

[0239] S310, after receiving the frame playback instruction, the frame concurrent decoding and playback module sends a rendering request to the rendering thread, and the rendering request carries the timestamp t1 of Video A.

[0240] S311, after receiving the parsing request, the GOP parser then parses Video A to obtain the H.264 format video frame in the GOP group where the video frame at the current timestamp t1 in Video A is located, and records the timestamp ts of the first video frame and the timestamp te of the last video frame in the GOP group.

[0241]

[0242] When implemented specifically, the GOP parser can cache the decoded H.264 video frames, and the decoding request carries the cache address, or directly send the parsed H.264 video frames.

[0243]

[0244]

[0242]

[0245] In the embodiment of the present application, when the status of GOP decoder 1 is the idle state, decoding is performed by GOP decoder 1; if the status of GOP decoder 1 is not the idle state, it is necessary to check whether other GOP decoders are in the idle state to decode through the decoder in the idle state, so as to decode the video frames at a relatively fast speed.

[0246] This example takes decoder 1 being in the idle state as an example.

[0247]

[0248]

[0243]

[0249] The H.264 video frames obtained by the GOP decoder are all the video frames within the GOP group where the t1 video frame is located, and need to be decoded in sequence according to the decoding order until all the video frames in the group are decoded, and the decoded data is YUV data.

[0250] In practical applications, GOP decoder 1 can decode the YUV data of each video frame and store the YUV data of the video frame in cache space 1. Correspondingly, GOP decoder 2 has a corresponding cache space 2 to store the YUV data of the video frames decoded by GOP decoder 2. When GOP decoder 2 needs to decode, GOP decoder 2 can store the YUV data of each decoded video frame in cache space 2.

[0251]

[0252] In practical applications, the rendering thread sends query requests to each GOP decoder in a polling manner. The query request carries the timestamp t1, and the GOP decoder corresponding to the video frame with timestamp t1 is determined according to the query result sent by the decoder based on the query request.

[0253] The rendering thread can also search for the video frame at time t1 from the cache spaces corresponding to each GOP decoder in a polling manner until the video frame at time t1 is found in one of the cache spaces, so as to determine the GOP decoder corresponding to the video frame with timestamp t1.

[0254] Taking the rendering thread sending query requests to the GOP decoder in a polling manner as an example, in the embodiment of the present application, after receiving the rendering request, the rendering thread:

[0255] The rendering thread first sends a query request (carrying t1) to GOP decoder one; GOP decoder one queries cache space one and determines that there is no video frame at time t1, and sends the information of no video frame at time t1 to the rendering thread. In practical applications, if GOP decoder one is not the decoder for decoding the video frame with timestamp t1, then there is no video frame at time t1 in cache space one, and GOP decoder one sends the information of no video frame with timestamp t1 to the rendering thread; if GOP decoder one is the decoder for decoding the video frame with timestamp t1, but has not decoded the video frame of t1 according to the decoding order, then there is also no video frame at time t1 in cache space one, and GOP decoder one sends the information of no video frame with timestamp t1 to the rendering thread. This is because when the GOP decoder decodes the video frames in the group of pictures, it decodes them according to the decoding order of the video frames. When GOP decoder one receives the query request, the decoder may not have decoded the video frame at time t1 yet. Therefore, there is no video frame at time t1 in cache space one either, and it is also necessary to send the information of no video frame with timestamp t1 to the rendering thread. If the rendering thread receives the information of no video frame with timestamp t1 sent by GOP decoder one, it sends a query request (carrying t1) to GOP decoder two; GOP decoder two queries cache space two and determines that there is no video frame at time t1, and sends the information of no video frame at time t1 to the rendering thread. Similarly, if GOP decoder two is not the decoder for decoding the video frame with timestamp t1, then there is no video frame at time t1 in cache space two, and GOP decoder two sends the information of no video frame with timestamp t1 to the rendering thread; if GOP decoder two is the decoder for decoding the video frame with timestamp t1, but has not decoded the video frame of t1 yet, then there is also no video frame at time t1 in cache space two, and GOP decoder two sends the information of no video frame with timestamp t1 to the rendering thread.

[0256] If the rendering thread receives the information of the video frame without timestamp t1 sent by GOP decoder two, it continues to send a query request (carrying t1) to GOP decoder one...;

[0257] In this polling manner, until the rendering thread receives the information of the video frame at t1 sent by any decoder. Taking the example that the rendering thread receives the information of the video frame at t1 sent by GOP decoder one in the embodiment of this application, the rendering thread can determine that the decoder for decoding the video frame with timestamp t1 is GOP decoder one.

[0258] S318, the rendering thread obtains the YUV data at t1 from the cache (cache space one) of the determined GOP decoder one.

[0259] S319, the rendering thread renders the obtained YUV data to obtain the video frame picture after rendering at t1.

[0260] S320, the rendering thread sends the video frame picture corresponding to t1 to the gallery application.

[0261] S321, the gallery application displays the video frame picture of video A at t1 in the video playing area, and at the same time updates the position of the progress display frame, so that the moment corresponding to the progress indicator line is consistent with the timestamp of the received video picture.

[0262] During the process of the user dragging the progress display frame, the contact position of the user's finger detected by the electronic device may change. Correspondingly, the positional relationship between the progress indicator line and the progress display frame also changes, and the moment corresponding to the progress indicator line also changes. If the moments corresponding to the two consecutive contact positions detected are t1 and t2 respectively, the process of the electronic device displaying the video frame at t2 can refer to Figure 10 the frame playing process shown, where the GOP group where the video frame at t2 is located is the same as the GOP group where the video frame at t1 is located.

[0263] S401, the gallery application receives the drag operation acting on the progress display frame.

[0264] It should be noted that the drag operations in S401 and step S301 are the same continuous operation, that is, the user's drag operation is not released between step S401 and step S301, but only the moment corresponding to the progress indicator line changes due to the drag operation.

[0265] S402, the gallery application determines the timestamp t2 of the frame playing of video A according to the drag position.

[0266] S403. When the gallery application determines that Video A is not played in the normal mode, it sends a frame playback request to the decision module. The frame playback request carries the timestamp t2 of Video A.

[0267] In the embodiment of this application, when Video A is played in the normal mode, the gallery application needs to send a normal playback pause instruction and a frame playback request to the decision module. Specifically, reference can be made to Figure 7 S303 shown.

[0268] S404. After receiving the frame playback request, the decision module sends a frame concurrent decoding and playback instruction to the frame concurrent decoding and playback module. The instruction carries the timestamp t2 of Video A.

[0269] S405. After receiving the frame playback instruction, the frame concurrent decoding and playback module sends a parsing request to the GOP parser. The parsing request carries the timestamp t2 of Video A.

[0270] S406. After receiving the frame playback instruction, the frame concurrent decoding and playback module sends a rendering request to the rendering thread. The rendering request carries the timestamp t2 of Video A.

[0271] S407. After receiving the parsing request, the GOP parser determines whether the video frame corresponding to the timestamp t2 in the parsing request is in the latest determined GOP group.

[0272] In this application, in S311, the timestamp ts of the first video frame and the timestamp te of the last video frame in the GOP group where the last obtained t1 moment is located have been recorded. It can be determined whether t2 is greater than or equal to ts and less than or equal to te. If t2 is greater than or equal to ts and less than or equal to te, it means that the video frame corresponding to t2 is in the latest GOP group. If t2 is less than ts or greater than te, it means that the video frame corresponding to t2 is not in the latest GOP group.

[0273] After the GOP parser determines that the video frame with timestamp t2 is in the latest determined GOP group, it will no longer parse Video A nor send a decoding instruction to the decoding thread because the GOP group where the video frame with timestamp t2 is located has been parsed before, and the decoder also starts to decode the GOP group where the video frame with t2 is located in sequence.

[0274] S408. The rendering thread finds the GOP decoder corresponding to the timestamp t2 according to the timestamp t2 in the received rendering request: GOP Decoder One.

[0275] Among them, the process of the rendering thread determining the GOP decoder corresponding to the timestamp t2 is the same as the process of determining the GOP decoder corresponding to the timestamp t1.

[0276] It should be noted that since in Figure 7In the timing diagram shown, in step S315, the video frame decoded by GOP decoder 1 is a video frame in the GOP group where the t1 video frame is located, and the t2 video frame is also in this GOP group. Therefore, in step S315, the video frame corresponding to t2 will also be decoded.

[0277] When the rendering thread queries GOP decoder 1 to check if there is YUV data corresponding to the t2 timestamp during specific implementation, it is possible that GOP decoder 1 has already decoded the YUV data corresponding to t2, or it is also possible that the YUV data corresponding to t2 has not been decoded yet.

[0278] S409. The rendering thread obtains the YUV data corresponding to t2 from GOP decoder 1.

[0279] S410. The rendering thread renders the obtained YUV data at time t2 to obtain the video frame image corresponding to time t2.

[0280] S411. The rendering thread sends the video frame image corresponding to time t2 to the gallery application.

[0281] S412. After receiving the video frame image corresponding to time t2 sent by the rendering thread, the gallery application displays the video frame image of video A at time t2 in the video playback area, and at the same time updates the position of the progress display frame so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video image.

[0282] Of course, during the specific implementation process, if the t2 video frame and the t1 video frame do not belong to the same group of video frames, the parsing, decoding, and rendering processes of the t1 video frame in Figure 9 will continue to be executed.

[0283] In addition, when the decoding thread determines the GOP decoder to be used for decoding:

[0284] It can first query GOP decoder 1;

[0285] If GOP decoder 1 is in an idle state, decoding is performed through GOP decoder 1;

[0286] If GOP decoder 1 is in a used state, query GOP decoder 2;

[0287] If GOP decoder 2 is in an idle state, decoding is performed through GOP decoder 2;

[0288] Of course, in practical applications, to make the frame playback smoother, more GOP decoders can also be set. For example:

[0289] If GOP decoder 2 is in a used state, query GOP decoder 3;

[0290] …….

[0291] In the embodiments of the present application, by Figure 11 As an example, when the t3 video frame is not in the GOP group where the t1 video frame is located, continue to execute according to Figure 9 the parsing, decoding, and rendering processes of the t1 video frame in

[0292] Of course, in practical applications, it does not mean that when the t3 video frame is not in the GOP group where the t1 video frame is located, different GOP decoders must be used for decoding.

[0293] S501, The gallery application receives a drag operation on the progress display frame.

[0294] It should be noted that the drag operations in S501, S401, and S301 are the same continuous drag operation, that is, the user's drag operation is not released between step S501 and step S301, but only the drag position is changed.

[0295] S502, The gallery application determines the timestamp t3 of the frame playback of video A according to the drag position.

[0296] S503, When the gallery application determines that video A is not played in the normal mode, it sends a frame playback request to the decision module, and the frame playback request carries the timestamp t3 of video A.

[0297] S504, After receiving the frame playback request, the decision module sends a frame concurrent decoding and playback instruction to the frame concurrent decoding and playback module, and the instruction carries the timestamp t3 of video A.

[0298] S505, After receiving the frame playback instruction, the frame concurrent decoding and playback module sends a parsing request to the GOP parser, and the parsing request carries the timestamp t3 of video A.

[0299] S506, After receiving the frame playback instruction, the frame concurrent decoding and playback module sends a rendering request to the rendering thread, and the rendering request carries the timestamp t3 of video A.

[0300] S507, After receiving the parsing request, the GOP parser determines that the video frame with timestamp t3 in the parsing request is not in the latest determined GOP group.

[0301] It can be according to Figure 10In the manner shown in step S407, compare t3 with ts and the relationship between t3 and te to determine whether the video frame of t3 is in the latest determined GOP group. If t3 is greater than or equal to ts and less than or equal to te, it means that the video frame corresponding to t3 is in the latest GOP group; if t3 is less than ts or greater than te, it means that the video frame corresponding to t3 is not in the latest GOP group.

[0302] S508, after the GOP parser determines that the video frame with timestamp t3 is not in the latest determined GOP group, start parsing video A to obtain the H.264 format video frames within the GOP group where the video frame of t3 is located.

[0303] S509, the GOP parser sends a decoding request to the GPO decoding thread.

[0304] In specific implementation, the GOP parser can cache the decoded H.264 format video frames, and the decoding request carries the cache address, or directly send the parsed H.264 format video frames.

[0305] S510, the decoding thread sends a status acquisition request to GOP decoder 1.

[0306] S511, the decoding thread receives that the status of GOP decoder 1 is in use.

[0307] S512, when the decoding thread determines that GOP decoder 1 is in use, it sends a status acquisition request to GOP decoder 2.

[0308] S513, the decoding thread receives that the status of GOP decoder 2 is in the idle state.

[0309] S514, when the decoding thread determines that the status of GOP decoder 2 is in the idle state, it sends the H.264 format video frame or the cache address of the H.264 format video frame to the idle GOP decoder 2.

[0310] S515, GOP decoder 2 decodes the H.264 format video frame to obtain YUV data and caches the YUV data.

[0311] The H.264 format video frames obtained by the GOP decoder are all the video frames within the GOP group where the video frame of t3 is located, and need to be decoded in sequence according to the decoding order until all the video frames within the GOP group are decoded. The data of each decoded video frame is YUV data.

[0312] S516, the rendering thread finds the GOP decoder corresponding to the video frame with timestamp t3 according to the timestamp t3 in the received rendering request: GOP decoder 2.

[0313] The method for the rendering thread to determine the decoder for the video frame at t3 is the same as the method for determining the decoder for the video frame at t1, and will not be elaborated here.

[0314] S517. The rendering thread obtains the YUV data at time t3 from the cache of the GOP decoder two according to the determined GOP decoder two.

[0315] S518. The rendering thread renders the obtained YUV data to obtain the video frame image after rendering at time t3.

[0316] S519. The rendering thread sends the video frame image corresponding to time t3 to the gallery application.

[0317] S520. The gallery application displays the video frame image at time t3 of video A in the video playback area, and simultaneously updates the position of the progress display frame, so that the time corresponding to the progress indicator line is consistent with the timestamp of the received video image.

[0318] In the specific implementation process, Figure 2 the interface shown in (e) in can be the video frame playback interface corresponding to the timestamp t3.

[0319] In addition, after the user drags and ends the progress display frame, that is, after the user's finger lifts from the area of the progress display frame, the playback process of video A will resume to the state before switching to frame playback: normal playback or paused playback.

[0320] In the specific implementation, the first case: If the user drags the progress display frame during the normal mode playback of video A, then after the user's finger lifts from the area of the progress display frame, the playback process of video A switches from frame playback to the normal playback mode, and continues to play in the normal mode starting from the moment indicated by the progress indicator line when the user's finger lifts from the progress display frame.

[0321] The second case: If the user first pauses the normal playback process of video A during the normal mode playback of video A, and then drags the progress display frame in the paused state of video A, then after the user's finger lifts from the area of the progress display frame, the playback process of video A switches from frame playback to the paused state, and after the user clicks the normal playback control in the video playback area of the interface, it continues to play in the normal mode starting from the moment indicated by the progress indicator line (the moment indicated by the progress indicator line when the user's finger lifts from the area of the progress display frame).

[0322] In the embodiment of the present application, the first case is taken as an example. After the user's finger lifts from the area of the progress display frame to release the progress display frame at time t3, video A switches from frame playback to normal playback.

[0323] Refer to Figure 12, which is a timing diagram for switching from frame playback to normal playback when the moment corresponding to the progress indicator line during the release operation on the progress display frame provided in the embodiment of the present application is the moment t3.

[0324] S601. The gallery application receives a release operation on the progress display frame.

[0325] In the embodiment of the present application, the release operation in S601 (after dragging ends, the user's finger lifts from the screen) and the previous dragging operation are continuous gesture operations, that is, the user's finger drags the progress display frame so that the moment represented by the progress indicator line goes from t1 to t3 and then lifts the finger, and this finger-lifting operation is the release operation on the progress display frame.

[0326] S602. In response to this operation, the gallery application sends the frame playback end information of video A and the timestamp t3 at the end to the decision module.

[0327] In specific implementation, a flag bit can be set in the gallery application or the decision module to mark whether the state of video A before frame playback is normal playback state or paused playback state; if it is in the paused state, the subsequent process of resuming normal playback will not be executed; if it is in the playback state, since the decision module pauses the normal playback when switching to frame playback (the normal playback paused when switching to frame playback does not modify the flag bit), it is necessary to send a playback instruction to the normal player again, that is, execute the subsequent operations.

[0328] S603. After receiving the frame playback end information and the timestamp t3, the decision module sends a video A start playback instruction to the normal playback module, and this instruction carries the start playback time t3.

[0329] S604. After receiving the video A start playback instruction and the timestamp t3, the normal playback module starts to parse the file of video A based on the moment t3 to obtain video frames in H.264 format.

[0330] After S604, the process of normal playback is executed and Figure 6 is similar to the steps executed after S204 in Figure 6 , except that Figure 10 in

[0331] decodes the video frames starting from the 0th second;

[0332] After S604 in Figure 6 starts to play from the video frame at the moment t3.

[0333] S606, The general decoder decodes the video frame to obtain the decoded video data: YUV data.

[0334] S607, The general decoder sends the obtained YUV data to the general player.

[0335] Similarly, the general decoder can also cache the YUV data and then send the address of the cached YUV data to the general player. The embodiments of the present application do not limit the specific transmission method.

[0336] S608, After receiving the decoded YUV data, the general player renders the YUV data to obtain a video picture.

[0337] S609, The general player sends the rendered video picture to the gallery application and carries the timestamp of the video picture.

[0338] S610, After receiving the video picture, the gallery application displays the received video picture in the video playing area, and at the same time updates the position of the progress display frame in the progress display area, so that the moment corresponding to the progress indicator line is consistent with the timestamp of the received video picture.

[0339] Through the above method, the normal playback and frame playback of Video A can be realized, as well as the switching between normal playback and frame playback. And because two additional configured decoders are used for decoding during the frame playback process, the decoding efficiency during frame playback is improved, making the video picture during the frame playback process smoother.

[0340] As mentioned above, the embodiments of the present application also provide another type of video: a one-record-many video. Corresponding to the playback process of the one-record-many video shown in Figure 3 When the electronic device displays the video playback interface shown in (b) in Figure 3 , it also needs to create a thumbnail parsing module to be able to parse and obtain the number of thumbnails matching the video duration as the progress display frames when switching from the progress display frames corresponding to the home frame and the middle frame to the progress display frames corresponding to multiple thumbnails.

[0341] Referring to Figure 13 , on the basis of the timing diagram shown in Figure 5 , after step S103, it further includes:

[0342] S115, After receiving the click operation on the cover of Video A, the gallery application creates a thumbnail parsing module.

[0343] Of course, in the specific implementation, there is no strict sequence for the steps of creating the frame concurrent decoding and playing module and the thumbnail parsing module after S103. Similarly, after the gallery application receives a click operation on the cover of video A, on the one hand, it executes the process of displaying the video playing interface of video A, on the other hand, it executes the process of creating the thumbnail parsing module, and on the other hand, it executes the process of creating the frame concurrent decoding and playing module. There is no strict sequence for the above three processes.

[0344] Referring to Figure 14 , when the progress display frames displayed in the progress display area of the video playing interface are the home frame and the middle frame, the user can trigger the progress display frames to be updated to multiple thumbnails that match the video duration through a click operation or other operations. For the sake of convenience of description, only the gallery application that interacts with each other and the thumbnail parsing module for obtaining multiple thumbnails that match the video duration are shown.

[0345] S701, the gallery application receives a click operation on the progress display frame of video A.

[0346] This click operation is a click operation on the progress display frame that is displayed as the home frame and the middle frame. For example, it can be the click operation on the progress display frame shown in (b) of Figure 3 , and the progress display frame is the home frame and the middle frame.

[0347] S702, after the gallery application receives this click operation, it sends the file path of video A to the thumbnail parsing module to trigger the thumbnail parsing module to parse video A to obtain multiple thumbnails that match the duration of video A.

[0348] S703, the gallery application sends a thumbnail acquisition request for video A to the thumbnail parsing module.

[0349] Among them, S703 and S702 can be executed as one step or as two independent steps. The embodiments of the present application do not limit this.

[0350] S704, after the thumbnail parsing module receives the thumbnail acquisition request for video A, it obtains video A according to the file path of video A and parses video A to obtain n thumbnails, where n is related to the duration of video A. The longer the duration of video A, the larger n is.

[0351] S705, the thumbnail parsing module sends the n parsed thumbnails to the gallery application.

[0352] S706, after the gallery application receives the n thumbnails, it updates the progress display frame composed of the home frame and the middle frame to the progress display frame composed of the n thumbnails.

[0353] After S706, the electronic device displays Figure 3 the interface shown in (c) in which, the progress display frame is updated to thumbnails of 4 video frames.

[0354] Through Figure 14 the timing diagram shown, the display mode of the progress display frame can be switched in the multi-channel video playback interface. The normal playback and frame playback processes of the multi-channel video can refer to the normal playback and frame playback processes of the video as described above.

[0355] The normal playback process and frame playback process of Video A are described above. As described above, when opening the video playback interface of Video A, a parser, a decoder, etc. need to be created; similarly, when exiting the video playback interface of Video A, the created parser, decoder, etc. also need to be destroyed.

[0356] Referring to Figure 15 is the timing diagram for destroying the parser and decoder when exiting the video playback interface of Video A provided in the embodiments of the present application.

[0357] S801, in the case of displaying the video playback interface of Video A, the gallery application receives a return operation.

[0358] In the embodiments of the present application, the return operation can be an operation of swiping left starting from the rightmost side of the screen on the interface, and this operation can trigger the gallery application to return to Figure 2 the grid diagram interface shown in (a) in.

[0359] S802, after receiving the return operation, the gallery application displays a grid diagram, and the interface where the grid diagram is located is an interface before entering the video playback interface of Video A shown in (b) in, and the grid diagram includes the cover of Video A. Figure 2

[0360] S803, after receiving the return operation, the gallery application sends a destruction request to the decision module, and the destruction request is used to destroy each module created when entering the video playback interface of Video A.

[0361] It should be noted that there is no strict sequence between S802 and S803. They can be two parallel steps, or one step can be executed first and then the other step.

[0362] After receiving the destruction request, the decision module, on the one hand, destroys the relevant modules (GOP parser and GOP decoder) in the frame player, and on the other hand, destroys the relevant modules (normal decoder) in the normal player. The destruction operations on both sides can be executed simultaneously, or the destruction operation on one side can be executed first and then the destruction operation on the other side.

[0363] ​The process of destroying the relevant modules (GOP parser and GOP decoder) in the frame player is described through S804 to S809.

[0364] In S804, after the decision module receives the destruction request, it sends the destruction requests for the parser and decoder to the frame concurrent decoding and playing module.

[0365] In S805, after the frame concurrent decoding and playing module receives the destruction requests for the parser and decoder, it destroys the GOP parser.

[0366] In S806, after the frame concurrent decoding and playing module receives the destruction requests for the parser and decoder, it destroys the first GOP decoder.

[0367] In S807, after the frame concurrent decoding and playing module receives the destruction requests for the parser and decoder, it destroys the second GOP decoder.

[0368] Among them, there is no strict sequence for the steps of destroying the GOP parser and GOP decoder.

[0369] In S808, after the frame concurrent decoding and playing module destroys the GOP parser, the first GOP decoder, and the second GOP decoder, it sends the information that the destruction of the parser and decoder is successful to the decision module.

[0370] In S809, after the decision module receives the information that the destruction of the parser and decoder is successful sent by the frame concurrent decoder playing module, it destroys the frame concurrent decoding and playing module.

[0371] The process of destroying the relevant module (decoder) in the ordinary player is described through S810 to S812.

[0372] In S810, after the decision module receives the destruction request, it sends the destruction request for the decoder to the ordinary playing module.

[0373] In S811, after the ordinary playing module receives the destruction request for the decoder, it destroys the ordinary decoder.

[0374] In S812, after the ordinary playing module destroys the ordinary decoder, it sends the information that the destruction of the decoder is successful to the decision module.

[0375] In S813, after the decision module destroys the frame concurrent decoding and playing module and receives the information that the destruction of the decoder sent by the ordinary playing module is successful, it sends the information that the destruction is successful to the gallery application.

[0376] The embodiments of this application are not limited to being used in Figures 2 to 4In the shown scenario, when playing a video through a gallery application, the frame playing method provided by the embodiments of the present application can be used to achieve frame playing of the video to improve the smoothness of frame playing. Of course, when playing a video through other video playing applications, the frame playing method provided by the embodiments of the present application can also be used.

[0377] Referring to Figure 16 , which is a schematic diagram of a dragging operation on the progress display frame provided by the embodiments of the present application. Figure 17 For Figure 16 shown is a schematic diagram of the frame playing process corresponding to the dragging operation. The video playing method provided by the embodiments of the present application includes:

[0378] Display a first interface, where the first interface includes a first area and a first control, and the first area includes the first video frame of a first video;

[0379] When a dragging operation on the first control is received:

[0380] In response to the dragging operation, move the first control from a first position to a second position;

[0381] During the process of the first control moving from the first position to the second position along a first trajectory, decode a first segment of video frames of the first video through a first decoder, decode a second segment of video frames of the first video through a second decoder, display the video frames in the decoded first segment of video frames in the first area, and display the video frames in the decoded second segment of video frames in the first area. The first segment of video frames and the second segment of video frames are two consecutive segments of video frames in the first video.

[0382] In the embodiments of the present application, the first video can be Video A in the above example, and the first video frame can be any frame in the first video. For example, it can be the first video frame. The first interface can be the video playing interface in the above embodiments, and the first area can be the video playing area in the above embodiments. Of course, the first interface also includes a second area, that is, the progress display area. The above example takes the progress display area and the video playing area as two non-overlapping areas as an example. In actual applications, the progress display area can also be within the video playing area.

[0383] Referring to Figure 16 , the first control can be the progress display frame in the above embodiments. The position of the first control can be any point on the progress display frame as the position reference point. The position of the first control is the position of this position reference point. Of course, this position reference point can also be the position corresponding to the user touch point on the progress display frame. The first trajectory can be a straight line trajectory between the first position and the second position.

[0384] As described above, different positions of the progress display frame represent different playback progress. With the position of the progress indicator line unchanged, drag the progress display frame so that the progress indicator line is at different positions on the progress display frame; different positions of the progress indicator line on the progress display frame represent different playback progress; therefore, with the position of the progress indicator line unchanged, during a single drag operation, different positions of the progress display frame can also correspond to different playback progress.

[0385] If the playback progress is displayed in the form of a progress bar, the first control is a control that can be dragged on the progress bar, and the position of the progress bar remains unchanged. Similarly, different positions on the progress bar represent different playback progress.

[0386] During the process that the user drags the progress display frame so that the position of the progress display frame moves from the first position along the first trajectory to the third position, it successively passes through the fourth position, the fifth position, the sixth position, and the second position in sequence.

[0387] Refer to Figure 17 , the contact positions of the user's finger sequentially pass through the first position, the fourth position, the fifth position, the sixth position, the second position, and the third position from right to left.

[0388] If the video frames are sorted from left to right according to the playback order, then during the process that the contact position of the user's finger is dragged from right to left, the corresponding video frames are displayed from left to right. The video frames corresponding to different positions of the first control can refer to Figure 17 as shown.

[0389] Refer to Figure 18 , each segment of video frames (corresponding to a GOP group or a video frame group in the above example) includes an I frame, and P frames and B frames encoded based on this I frame.

[0390] As an embodiment of the present application, in practical applications, different segments of video frames can be alternately decoded by the first decoder and the second decoder. For example, the first segment of video frames, the second segment of video frames, and the fourth segment of video frames are three consecutive segments of video frames. When the first decoder decodes the first segment of video frames, the second decoder decodes the second segment of video frames, and the first decoder decodes the fourth segment of video frames.

[0391] As another embodiment of the present application, it is also possible to select a decoder in the idle state for decoding through the state of the decoder. For example, each video frame in the first group of video frames is decoded by the first decoder in the idle state; when the drag operation makes the position of the first control correspond to the second segment of video frames, the first decoder may still be decoding the video frames in the first segment of video frames, so each video frame in the second segment of video frames can be decoded by the second decoder in the idle state; when the drag operation makes the position of the first control correspond to the third segment of video frames, the second decoder may still be decoding the video frames in the second segment of video frames, but the first decoder has completed decoding the video frames in the first segment of video frames, so each video frame in the third segment of video frames can be decoded by the first decoder in the idle state.

[0392] In this embodiment, at least two decoders can be dedicated to the frame playback scenario; in the frame playback scenario, the currently to-be-displayed video frame is decoded by the decoder that is currently in the idle state (or alternately using decoders for decoding), thereby improving the decoding speed of the video frame; after the decoding speed of the video frame is improved, the display of the video frame is relatively smooth; by this method, the smoothness of the video picture in the frame playback scenario can be improved.

[0393] In the embodiment of the present application, there may be other video frames before the first segment of video frames of the first video, and there may also be other video frames after the fourth segment of video frames.

[0394] When the first control is in the first position, the video frame corresponding to the playback at this moment can be the I-frame in the first segment of video frames or the video frame before the I-frame in the first segment of video frames;

[0395] When the first control is in the second position, the video frame corresponding to the playback at this moment can be the last video frame in the second segment of video frames;

[0396] When the first control is in the third position, the video frame corresponding to the playback at this moment can be the last video frame in the third segment of video frames or the video frame after the last video frame in the third segment of video frames;

[0397] The user can drag the first control from the first position to the third position. During the dragging process, it specifically passes through the fourth position, the fifth position, the sixth position, and the second position from the first position to the third position. Of course, it also passes through other positions in the middle, which will not be exemplified one by one.

[0398] The following describes the dragging process from the first position to the second position.

[0399] When dragged to the fourth position, determine that the corresponding playback time is the first time. At this time, the previous decoded video frame group is not the first segment of video frames. Therefore, parse the first video to obtain the first segment of video frames where the video frame corresponding to the first time is located; decode the video frames in the first segment of video frames in sequence according to the decoding order of the video frames in the first segment of video frames through the first decoder; after decoding the second video frame corresponding to the first time in the first segment of video frames, display the second video frame in the first area.

[0400] When dragged to the fifth position, determine that the corresponding playback time is the second time. At this time, the previous decoded video frame group is the first segment of video frames, and the video frame corresponding to the second time is within the first segment of video frames. Since the first decoder will decode each video frame in the first segment of video frames, the third video frame corresponding to the second time can be obtained from the first decoder; after obtaining the third video frame from the first decoder, display the third video frame in the first area.

[0401] When dragged to the sixth position, determine that the corresponding playback time is the third time. At this time, the previous decoded video frame group is the first segment of video frames, and the video frame corresponding to the third time is not in the first segment of video frames. Parse the first video to obtain the second segment of video frames where the fourth video frame corresponding to the third time is located; decode the video frames in the second segment of video frames in sequence according to the decoding order of the video frames in the second segment of video frames through the second decoder; after decoding the fourth video frame in the second segment of video frames, display the fourth video frame in the first area.

[0402] It should be noted that the second video frame can be a B frame in the example or not a B frame in the example; the third video frame can also be a B frame in the example or not a B frame in the example; however, the playback order of the second video frame is before the third video frame. Similarly, the fourth video frame can be a B frame in the figure or not a B frame in the example.

[0403] Since the above dragging process is a frame playback process, the video frames displayed during the dragging process are related to the detected contact position. Therefore, some or all of the video frames in the first segment of video frames displayed during the dragging process; similarly, some or all of the video frames in the second segment of video frames displayed during the dragging process.

[0404] In addition, in the process of playing the first video in the ordinary playback mode of the embodiment of the present application, a third segment of video frames including the first segment of video frames and the second segment of video frames may also be displayed, and the position of the first control will also be moved during the display process; in addition, since the first video can be fully played during the ordinary playback process, the video frames played in the ordinary playback mode can also include other segments of video frames. Of course, the ordinary playback process can also play some video frames of the first video.

[0405] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0406] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program runs on an electronic device, it can implement the steps in the above method embodiments.

[0407] The embodiments of the present application also provide a computer program product. When the computer program product runs on an electronic device or a wireless router, the electronic device can implement the steps in the above method embodiments.

[0408] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps in the above method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the first device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0409] The embodiments of the present application also provide a chip, which includes a processor. The processor is coupled to a memory, and the processor calls the computer program stored in the memory to implement the steps in any method embodiment of the present application. The chip can be a single chip or a chip module composed of multiple chips.

[0410] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0411] Those of ordinary skill in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0412] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A video playback method, characterized in that, Applied to an electronic device, the method includes: Displaying a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video; When a drag operation on the first control is received: In response to the drag operation, moving the first control from a first position along a first trajectory to a second position; During the process of moving the first control from the first position along the first trajectory to the second position, decoding a first segment of video frames of the first video by a first decoder, decoding a second segment of video frames of the first video by a second decoder, displaying in the first area the video frames in the decoded first segment of video frames, and displaying in the first area the video frames in the decoded second segment of video frames, the first segment of video frames and the second segment of video frames being two consecutive segments of video frames in the first video.

2. The method according to claim 1, wherein A play control is displayed on the first video frame in the first interface, and the method further includes: When an operation on the play control is received: In response to the operation on the play control, decoding the video frames of the first video by a third decoder, and displaying in the first area the video frames in the decoded first video; During the process of displaying in the first area the third segment of video frames in the decoded first video, the first control moves from the first position along the first trajectory to the second position, and the third segment of video frames includes the first segment of video frames and the second segment of video frames.

3. The method according to claim 1 or 2, characterized in that, The displaying in the first area the video frames in the decoded first segment of video frames and the displaying in the first area the video frames in the decoded second segment of video frames specifically are: After displaying in the first area the video frames in the decoded first segment of video frames, displaying in the first area the video frames in the decoded second segment of video frames.

4. The method according to claim 3, characterized in that, The method further includes: In response to the drag operation, continuing to move the first control from the second position to a third position; During the process of moving the first control from the second position to the third position, decoding a fourth segment of video frames of the first video by the first decoder, and displaying in the first area the video frames in the decoded fourth segment of video frames, the fourth segment of video frames and the second segment of video frames being two consecutive segments of video frames in the first video.

5. The method according to any one of claims 1 to 4, characterized in that The decoding the first segment of video frames of the first video by the first decoder includes: When the state of the first decoder is an idle state, decoding the first segment of video frames of the first video by the first decoder.

6. The method according to claim 5, characterized in that, The decoding the first segment of video frames of the first video by the first decoder when the state of the first decoder is an idle state includes: Obtaining the state of the first decoder; Determining that the state of the first decoder is an idle state; Decoding the first segment of video frames of the first video by the first decoder in the idle state.

7. The method according to any one of claims 1 to 6, characterized in that, The decoding the second segment of video frames of the first video by the second decoder includes: When the state of the first decoder is in use and the state of the second decoder is idle, the second decoder in the idle state decodes the second video frame of the first video.

8. The method according to claim 7, wherein When the state of the first decoder is in use and the state of the second decoder is idle, the second decoder in the idle state decodes the second video frame of the first video, including: Obtain the state of the first decoder; Determine that the state of the first decoder is in use; When it is determined that the state of the first decoder is in use, obtain the state of the second decoder; Determine that the state of the second decoder is idle; The second decoder in the idle state decodes the second video frame of the first video.

9. The method according to any one of claims 1 to 8, characterized in that, During the process that the first control moves from the first position along the first trajectory to the second position, the method further includes: When the first control moves to the fourth position, the second video frame in the first video is displayed in the first area. When the first control is at the fourth position, the video frame corresponding to the playback progress of the first video is the second video frame, and the fourth position is on the first trajectory and between the first position and the second position.

10. The method according to claim 9, characterized in that The first interface further includes a first indicator, which is located above the first control and has a fixed position on the screen; when the first control is at the fourth position, the video frame corresponding to the playback progress indicated by the position of the first indicator on the first control is the second video frame, and the position of the first control is the position of a point on the first control.

11. The method according to claim 9 or 10, characterized in that, When the first control is at the fourth position, the playback progress of the first video is the first moment. When the first control moves to the fourth position, the step of displaying the second video frame in the first video in the first area includes: When the first control moves to the fourth position, parse the first video to obtain the first video frame where the video frame corresponding to the first moment is located; The first decoder sequentially decodes the video frames in the first video frame according to the decoding order of the video frames in the first video frame; After completing the decoding of the second video frame corresponding to the first moment in the first video frame, display the second video frame in the first area.

12. The method according to claim 11, wherein During the process that the first control moves from the first position along the first trajectory to the second position, the method further includes: When the first control moves from the fourth position to the fifth position, determine that the playback progress of the first video is the second moment, and the fifth position is on the first trajectory and between the fourth position and the second position; If the video frame corresponding to the second moment is within the first video frame, obtain the third video frame corresponding to the second moment from the first buffer space of the first decoder; After obtaining the third video frame from the first decoder, display the third video frame in the first area.

13. The method according to claim 11 or 12, characterized in that During the process that the first control moves from the first position to the second position along the first trajectory, the method further includes: When the first control moves from the fourth position to the sixth position, determining that the playing progress of the first video is the third moment, where the sixth position is on the first trajectory and is located between the fourth position and the second position; If the video frame corresponding to the third moment is not in the first segment of video frames, parsing the first video to obtain the second segment of video frames where the fourth video frame corresponding to the third moment is located; Sequentially decoding the video frames in the second segment of video frames by the second decoder according to the decoding order of the video frames in the second segment of video frames; After completing the decoding of the fourth video frame in the second segment of video frames, displaying the fourth video frame in the first area.

14. The method according to any one of claims 1 to 13, characterized in that, The video frames of the first video include a plurality of I frames, and between any two I frames, there are at least one P frame and at least one B frame. The first segment of video frames includes the i-th I frame, and the P frames and B frames between the i-th I frame and the (i + 1)-th I frame. The second segment of time frames includes the (i + 1)-th I frame, and the P frames and B frames between the (i + 1)-th I frame and the (i + 2)-th I frame. The timestamp of the i-th I frame is less than the timestamp of the (i + 1)-th I frame, and the timestamp of the (i + 1)-th I frame is less than the timestamp of the (i + 2)-th I frame, where i is a natural number greater than or equal to 1.

15. The method according to any one of claims 1 to 14, characterized in that, The first interface further includes a second area, and the first control is located within the second area and can move within the second area; The first area and the second area do not overlap, or the second area is within the first area.

16. The method according to any one of claims 2 to 15, characterized in that, The displaying the video frames in the decoded first segment of video frames in the first area is specifically: Displaying some or all of the video frames in the decoded first segment of video frames in the first area; The displaying the video frames in the decoded second segment of video frames in the first area is specifically: Displaying some or all of the video frames in the decoded second segment of video frames in the first area; The displaying the video frames in the decoded first video in the first area specifically includes: Displaying some or all of the video frames in the decoded first video in the first area.

17. An electronic device, characterized in that, Including one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program. When the one or more processors execute the computer program, the electronic device executes the method according to any one of claims 1-16.

18. A chip system, the chip system being applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to call computer instructions to enable the electronic device to execute the method according to any one of claims 1-16.

19. A computer-readable storage medium, comprising a computer program, characterized in that, When the computer program runs on the electronic device, the electronic device executes the method according to any one of claims 1-16.

Images