A video playback method, electronic device, chip system, and storage medium

Abstract

This application provides a video playback method, electronic device, chip system, and storage medium, relating to the field of terminal device technology. In a frame playback scenario, when a sliding operation triggers the electronic device to display a video frame at a certain moment, the first decoder decodes the video frame in the video frame group (first frame group) where the video frame at that moment is located, and the second decoder decodes the video frame in the next video frame group (second frame group) of the first frame group. When a dragging operation triggers the electronic device to display a video frame in the second frame group, since the decoder has already decoded and cached it in advance, the decoded video frame can be directly obtained from the cache space corresponding to the second decoder for rendering and display. Of course, it is also necessary to decode the video frame in the next video frame group (third frame group) of the second frame group through the first decoder, so that video frames in the video frame group are always decoded in advance, improving the smoothness of the picture in the frame playback scenario.

Description

Technical Field

[0001] This application relates to the field of terminal equipment technology, and in particular to a video playback method, electronic device, chip system and storage medium. Background Technology

[0002] Some applications installed on electronic devices (such as gallery apps, video editing apps, etc.) provide video playback and video editing functions. When playing a video, the playback progress can be displayed, for example, through a progress bar, progress ring, or progress display frame. When editing a video, different positions of the video can also be displayed through a progress bar, progress ring, or progress display frame.

[0003] Whether in video playback or video editing scenarios, users can trigger electronic devices to play videos frame by frame by dragging on the progress bar, progress ring, or progress display frame; however, in frame-by-frame video playback scenarios, the display of the video is relatively choppy and not smooth. Summary of the Invention

[0004] This application provides a video playback method, electronic device, chip system, and storage medium, which can reduce video stuttering in frame playback scenarios and improve the smoothness of video playback in frame playback scenarios.

[0005] To achieve the above objectives, the first aspect of this application adopts the following technical solution:

[0006] The first aspect of this application provides a video playback method applied to an electronic device, the method comprising:

[0007] Display a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video;

[0008] A swipe operation is received on the first interface;

[0009] In response to the sliding operation, the first control is moved from the first position along the first trajectory;

[0010] During the process of moving the first control from the first position along the first trajectory:

[0011] When the first control is moved from the first position to the second position on the first trajectory, the video frame in the first frame group is decoded by the first decoder, the video frame in the second frame group is decoded by the second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress represented by the first control when it is in the second position is within the progress range represented by the first frame group;

[0012] When the first control is moved from the second position to the third position on the first trajectory, the video frame in the third frame group is decoded by the first decoder, and the decoded third video frame in the second frame group is displayed in the first area. The playback progress indicated by the first control at the third position is within the progress range indicated by the second frame group. The first frame group, the second frame group, and the third frame group are three groups of video frames that are consecutive in time in the first video.

[0013] In this application, when a sliding operation triggers the electronic device to display a video frame at a certain moment, the first decoder decodes the video frame in the video frame group (first frame group) where the video frame at that moment is located, and the second decoder decodes the video frame in the next video frame group (second frame group) of the first frame group. When a dragging operation triggers the electronic device to display a video frame in the second frame group, since the decoder has already decoded and cached it in advance, the decoded video frame can be directly obtained from the cache space corresponding to the second decoder for rendering and display, which improves the speed of displaying the screen, reduces stuttering, and improves the smoothness of the screen. Of course, it is also necessary to decode the video frame in the next video frame group (third frame group) of the second frame group through the first decoder, so that video frames in the video frame group are always decoded in advance, which improves the smoothness of the screen in the frame playback scenario.

[0014] As another implementation of the first aspect, during the process of moving the first control from the first position along the first trajectory, the method further includes:

[0015] The first control is moved from the third position to the fourth position on the first trajectory. Figure 4 In the case of position 4), the video frame in the fourth frame group is decoded by the second decoder, and the decoded fourth video frame in the third frame group is displayed in the first area. The playback progress indicated by the first control at the fourth position (position 4) is within the progress range indicated by the third frame group. The first frame group, the second frame group, the third frame group and the fourth frame group are four groups of video frames that are consecutive in time.

[0016] In this application, as the first control continues to move with the sliding operation, the decoder is used alternately to decode the video frames in the next video frame group, so that there is always a decoder that decodes the video frames in advance. When it is necessary to display the subsequent video frames, the decoded video frames can be directly obtained from the decoder for rendering and display, thereby improving the smoothness of the picture.

[0017] As another implementation of the first aspect, the method further includes, before decoding the video frames in the first frame group by the first decoder:

[0018] The playback progress of the first video is determined as the first moment based on the position of the first touch point of the sliding operation;

[0019] Analyze the first video and determine the video frame group containing the video frame corresponding to the first moment as the first frame group;

[0020] Before decoding the video frames in the second frame group using the second decoder, the method further includes:

[0021] The playback mode of the video frame corresponding to the first moment is determined to be forward playback;

[0022] If the video frame at the first moment is played in forward mode, the next video frame group in the first video that is located in the first frame group is determined to be the second frame group.

[0023] In this application, the playback progress of the video can be determined by the real-time touch position of the sliding operation. Then, the video frame group corresponding to the playback progress is found. If the video frame group corresponding to the playback progress has not yet been decoded, it is decoded by a decoder. Correspondingly, it can also be determined whether the frame playback process is forward playback (the playback time determined by the current touch position is greater than the playback time determined by the previous touch position) or backward playback (the playback time determined by the current touch position is less than the playback time determined by the previous touch position). In the case of forward playback, the next video frame group is decoded by another decoder to decode in advance.

[0024] In forward playback, the next video frame group is one that meets the following conditions: it is adjacent to the current video frame group and its timestamp is greater than the timestamp of the current video frame group. The timestamp of the video frame group can be the timestamp of any video frame in the video frame group.

[0025] In reverse playback, the previous video frame group is decoded by another decoder in advance. The previous video frame group is a video frame group that meets the following conditions: it is adjacent to the current video frame group and its timestamp is less than the timestamp of the current video frame group.

[0026] As another implementation of the first aspect, after determining the playback progress of the first video as the first moment based on the first touch point position of the sliding operation, the method further includes:

[0027] The position of the first control at the first moment is determined to be the second position when the playback progress of the first video is determined to be at the first moment.

[0028] Move the first control to the second position along the first trajectory.

[0029] In this application, the position of the first control on the first track is related to the playback progress, which provides good operability and intuitiveness from the user's perspective.

[0030] As another implementation of the first aspect, the method further includes, before decoding the video frames in the third frame group by the first decoder:

[0031] The playback progress of the first video is determined as the second moment based on the position of the second touch point of the sliding operation;

[0032] Analyze the first video to determine the video frame group containing the video frame corresponding to the second moment as the second frame group;

[0033] It has been determined that the second frame group has been assigned to the second decoder for decoding;

[0034] The playback mode of the video frame corresponding to the second moment is determined to be forward playback;

[0035] If the second frame group has been assigned to the second decoder for decoding, and the video frame corresponding to the second moment is played in forward mode, then the next video frame group in the first video located in the second frame group is determined to be the third frame group.

[0036] In this application, if the video frame group corresponding to the current playback progress has been assigned a decoder, the video frame in the next or previous video frame group can be decoded by another decoder according to the playback method, so as to pre-decode the next or previous video frame group that may be displayed as the user slides.

[0037] As another implementation of the first aspect, after determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method further includes:

[0038] The position of the first control at the second moment when the playback progress of the first video is determined is the third position;

[0039] Move the first control along the first trajectory to the third position.

[0040] As another implementation of the first aspect, determining that the playback mode of the video frame corresponding to the second moment is forward playback includes:

[0041] It is determined that the second time point is greater than the first time point;

[0042] Accordingly, if it is determined that the second time is less than the first time, the playback mode of the video frame corresponding to the second time is determined to be reverse playback.

[0043] In this application, the playback mode of the video frame at the current moment can be determined based on the relationship between the playback times of two consecutive playback progresses, and the determination process is simple and quick.

[0044] As another implementation of the first aspect, after determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method further includes:

[0045] Calculate the absolute value of the first difference between the second time point and the first time point;

[0046] The absolute value of the first difference is determined to be greater than the first duration.

[0047] As another implementation of the first aspect, after calculating the absolute value of the first difference between the second time point and the first time point, the method further includes:

[0048] If the absolute value of the first difference is not greater than the first duration, then the video frame group in which the second moment is located is no longer determined, nor is the next video frame group in which the second moment is located determined.

[0049] In this application, since the human eye's perception of frame rate is not very high, for video frames corresponding to two very close playback progresses (time difference not exceeding the first duration), the video frame corresponding to the later playback progress can be skipped. For video frames corresponding to two not very close playback progresses (time difference greater than the first duration), the video frame corresponding to the later playback progress needs to be played. The first duration is related to the human eye's perception of frame rate; for example, it can be set to 16.7ms, 15ms, 18ms, 25ms, etc. Therefore, the first duration is used to filter video frames to keep the frame rate within a certain range.

[0050] As another implementation of the first aspect, after determining that the video frame group corresponding to the first moment belongs to the first frame group, the method further includes:

[0051] Record the timestamp of the first video frame and the timestamp of the last video frame in the first frame group;

[0052] After determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, and before parsing the first video and determining that the video frame group corresponding to the second moment belongs to the second frame group, the method further includes:

[0053] It is determined that the second moment is not within the time range of the timestamps of the first and last video frames in the first frame group.

[0054] As another implementation of the first aspect, after determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method includes:

[0055] If the second time point falls within the time stamp range of the first video frame and the last video frame in the first frame group, then the decoded video frame corresponding to the second time point is obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second time point is displayed in the first area.

[0056] In this application, if the second moment falls within the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, it indicates that the video frame corresponding to the second moment is in the first frame group. Since the first frame group has already been assigned a decoder for decoding when the video frame corresponding to the first moment is to be displayed, it is not necessary to assign a decoder to decode the first frame group again. Instead, the decoded video frame corresponding to the second moment can be directly obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second moment can be displayed in the first area. Of course, if the second moment is not within the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, it indicates that the video frame corresponding to the second moment is not a video frame in the first frame group. It is necessary to determine the video frame group in which the video frame corresponding to the second moment belongs, and then obtain the decoded video frame from the decoder corresponding to the determined video frame group.

[0057] As another implementation of the first aspect, after determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method further includes:

[0058] Calculate the absolute value of the first difference between the second time point and the first time point;

[0059] If the absolute value of the first difference is greater than the second duration, both the first decoder and the second decoder are set not to decode non-key frames in the video frame group.

[0060] If the absolute value of the first difference is not greater than the second duration, the first decoder and the second decoder are configured to decode the video frames in the video frame group in the decoding order.

[0061] If the second duration is greater than the first duration, and the absolute value of the first difference is greater than the second duration (e.g., 500ms, 400ms, 600ms, 700ms, etc.), it indicates that the user's swiping operation is faster. In this case, the user does not need to perceive the rapid switching process of the screen. The decoder can decode only the keyframes in one video frame group, thereby enabling the decoder to quickly complete the decoding of the current video frame group. Of course, when displaying video frames, the keyframes in one video frame group can also be displayed, thereby improving the decoding speed and the smoothness of the screen.

[0062] As another implementation of the first aspect, the decoding of video frames in the first frame group by the first decoder includes:

[0063] The first decoder decodes the multiple video frames in the first frame group according to the decoding order of the multiple video frames in the first frame group;

[0064] The first decoder decodes one video frame from the first frame group and stores the decoded video frame in the first buffer space.

[0065] The video frames corresponding to the first moment include:

[0066] In the first cache space, video frames whose timestamp differs from the first moment by less than or equal to a first value, where the first value is half the difference between the timestamps of two adjacent video frames;

[0067] Alternatively, the video frame with the smallest difference between its timestamp and the first moment among the decoded video frames already cached in the first cache space;

[0068] Alternatively, keyframes already cached in the first cache space.

[0069] In practical implementation, different video frames can be displayed based on the speed of the user's swiping operation. For example, when the user's swiping operation is slow, the decoder has enough time to decode each video frame in a video frame group and can retrieve the video frame whose timestamp differs from the first time point from the cache space by a value less than or equal to the first value. When the swiping operation is fast, the decoder does not have enough time to decode each video frame in a video frame group and can retrieve the video frame with the smallest difference between the timestamp and the first time point from the cache space. When the swiping operation is very fast, the first decoded keyframe (I-frame) in the decoding order can be retrieved from the cache space. Correspondingly, non-keyframes include p-frames and B-frames. Of course, if a group of video frames does not have B-frames, then the non-keyframes are P-frames.

[0070] As another implementation of the first aspect, before displaying the decoded second video frame from the first frame group in the first region, the method further includes:

[0071] The decoder that decodes the first frame group is identified as the first decoder;

[0072] The video frame corresponding to the first time moment is obtained from the first decoder, and the video frame corresponding to the first time moment is the second video frame.

[0073] As another implementation of the first aspect, determining the decoder for decoding the first frame group as the first decoder includes:

[0074] Calculate the second difference between the timestamp of the first video frame cached in the first buffer space corresponding to the first time step and the first time step at the first time step;

[0075] When the second difference is within the first time range, the decoder that decodes the first frame group is determined to be the first decoder.

[0076] As another implementation of the first aspect, after calculating the second difference between the timestamp of the first video frame cached in the first buffer space corresponding to the first time point and the first decoder, the method further includes:

[0077] If the second difference is not within the first time range, then calculate the third difference between the timestamp of the first video frame cached in the second buffer space corresponding to the first time and the second decoder, to determine whether the decoder that decodes the first frame group is the second decoder.

[0078] In this application, since decoding is performed using at least two decoders, the decoder for the video frame group corresponding to the playback progress can be determined first by the relationship between the timestamp of the first video frame in each group of video frames and the determined playback progress (time). Then, the specific video frame is obtained from the cache space of the decoder.

[0079] In a second aspect, an electronic device is provided, including a processor for calling a computer program stored in a memory to implement the method of any one of the first aspects of this application.

[0080] Thirdly, a chip is provided, including a processor coupled to a memory, the processor executing a computer program stored in the memory to cause an electronic device to implement the method of any of the first aspects of this application.

[0081] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when computer instructions are executed on an electronic device, causes the electronic device to implement the method of any one of the first aspects of this application.

[0082] Fifthly, embodiments of this application provide a computer program product that, when run on a device, causes the electronic device to execute the method of any one of the first aspects of this application.

[0083] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0084] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0085] Figure 2 A schematic diagram of a video playback process provided in an embodiment of this application;

[0086] Figure 3 An example of a set of video frames provided for an embodiment of this application, and a schematic diagram of the playback order and decoding order of the set of video frames;

[0087] Figure 4 A flowchart illustrating the video playback method provided in this application embodiment;

[0088] Figure 5 A timing diagram provided for the preparatory work before frame playback in an embodiment of this application;

[0089] Figure 6 A timing diagram of the video frames corresponding to position 1 in the video playback process provided in this embodiment of the application;

[0090] Figure 7 A timing diagram of the video frames corresponding to position 2 in the video playback process provided in this embodiment of the application;

[0091] Figure 8 A schematic diagram illustrating the frame rate filtering process during video playback provided in this embodiment of the application;

[0092] Figure 9 A timing diagram of the video frames corresponding to position 3 in the video playback process provided in this embodiment of the application;

[0093] Figure 10 A timing diagram of the video frame corresponding to position k during video playback provided in this embodiment of the application;

[0094] Figure 11 This is a timing diagram of the video frame corresponding to position k+1 in the video playback process provided in the embodiments of this application. Detailed Implementation

[0095] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0096] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0097] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0098] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0099] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0100] This application provides a video playback method that can be applied to electronic devices, such as tablets, mobile phones, wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not limit the specific type of electronic device.

[0101] Figure 1A schematic diagram of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 1211, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0102] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0103] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the video frame playback method in the embodiments of this application.

[0104] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0105] Internal memory 1211 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 1211. Internal memory 1211 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback function). Touch sensor 180K, also called a "touch panel". Touch sensor 180K can be set on display screen 194, and touch sensor 180K and display screen 194 form a touch screen, also called a "touch screen". Touch sensor 180K is used to detect touch operations on or near it. Touch sensor can transmit the detected touch operation to application processor to determine the touch event type. Visual output related to touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be set on the surface of electronic device 100, in a different position than display screen 194. For example, click operations of controls and drag operations of progress display frames provided in this application embodiment can be recognized by touch sensor.

[0106] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. For example, in this embodiment, the process of rendering YUV data can be implemented using a GPU.

[0107] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1. For example, in the embodiments of this application... Figure 2 or Figure 3 All interfaces shown are displayed on the monitor.

[0108] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats. For example, the video codec can decode H.264 format video frames as provided in the embodiments of this application, or re-encode them into H.264 format video frames that are all keyframes.

[0109] This application does not specifically limit the structure of the execution entity of a video playback method. As long as communication can be performed according to the video playback method provided by this application by running code containing the code of this application's video playback method, it is acceptable. For example, the execution entity of the video playback method provided by this application can be a functional module in an electronic device capable of calling and executing programs, or a communication device applied in an electronic device, such as a chip.

[0110] Users can play and edit videos stored on their electronic devices using applications (such as gallery apps, video editing apps, etc.). Whether playing or editing videos, users can trigger frame-by-frame playback by dragging (or swiping) on ​​the progress bar, progress ring, or progress display frame. In practice, swiping on the interface displaying the progress bar, progress ring, or progress display frame may also trigger frame-by-frame playback.

[0111] The following example uses a gallery application to play videos.

[0112] Reference Figure 2 This is a schematic diagram of an interface for playing videos stored on an electronic device through a gallery application, as provided in an embodiment of this application.

[0113] Reference Figure 2 (a) in the image gallery is a grid of images and videos stored on an electronic device, displayed by the image gallery application of the electronic device. The grid can display images and videos, where the cell containing a video displays the cover and duration of the video, and the cell containing an image displays a thumbnail of the image.

[0114] The grid image can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app, or it can be the interface displayed by the Gallery app after the user clicks the Gallery app icon on the system desktop to open the Gallery app and triggers more steps.

[0115] When displaying images and / or videos stored on an electronic device, this grid chart can be presented in various ways, for example, Figure 2 The method shown in (a) displays photos and videos in a photo group in reverse chronological order; in practical applications, gallery applications can also set up multiple object groups, for example, through... Figure 2 The control 10 shown in (a) can view multiple groups in the album: image group, video group, screenshot and screen recording group, etc. Users can also display pictures through the image group in the gallery application, videos through the video group in the gallery application, and screenshots and screen recordings through the screenshot and screen recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the grid diagram shown above.

[0116] This application does not limit the number of user operations required from the system desktop to the display of the grid chart in the gallery application, nor does it limit the way the grid chart displays images and videos.

[0117] Reference Figure 2 (b) in the text represents the user's click. Figure 2 Following the cover of video A in the grid diagram shown in (a), the electronic device displays the video playback interface of video A in response to the operation (this interface can be taken as an example of a first interface). The video playback interface includes a video playback area 11 (this area can be taken as an example of a first area) and a progress display area 12.

[0118] The video playback area 11 is used to display the video frame during video playback. After entering the video playback interface from the grid interface, the video playback area displays control 111 and video cover (which can be used as an example of the first video frame). When the user clicks control 111, the electronic device responds to the click operation on control 111 and starts playing the video frame in the video playback area 11 from the 0th second. When the video is paused, control 111 is also displayed in the video playback area 11. When the user clicks control 111, the electronic device responds to the click operation on control 111 and starts playing the video frame in the video playback area 11 from the current paused moment.

[0119] The progress display area 12 is used to display the video playback progress. The progress display area 12 includes a progress display frame 121 (which can be used as an example of a first control) and a progress indicator line 122. The position of the progress indicator line 122 can remain unchanged. The user can drag the progress display frame 121 located below the progress indicator line 122 so that the progress display frame 121 moves left and right below the progress indicator line 122 without moving the progress indicator line 122. The position of the progress indicator line 122 above the progress display frame indicates the current playback progress of the video.

[0120] In this example, a progress display frame and a progress indicator line are used to demonstrate the playback progress of a video. The progress display frame can serve as an example of the first control. In practical applications, other progress display methods can also be used, such as progress bars and progress rings. If a progress bar displays the playback progress of a video, then the first control is an indicator on the progress bar that displays the playback progress and can be moved on the progress bar.

[0121] As an example, the total playback duration of the current video is 15 seconds, and the total length of the current video's progress display frames is represented by L pixels. Based on these two parameters, we can obtain the playback duration corresponding to a unit length (or the length corresponding to a unit of time). For example, divide the playback duration 15 by the total length L. Therefore, if the leftmost edge of the progress display frame (the position corresponding to the first pixel) corresponds to second 0 of the video, then different positions of the progress indicator line on the progress display frame will correspond to different playback times. For example, if the distance between the position of the progress indicator line on the progress display frame and the leftmost edge of the progress display frame is X, then the playback time corresponding to the progress indicator line is: X*15 / L.

[0122] For ease of subsequent description, the embodiments in this application are described in the manner of "the time corresponding to the progress indicator line". Of course, it can also be the playback progress represented by the position of the first control or the playback progress of the video represented by the first control at a certain position.

[0123] Due to space limitations in the progress display area 12, only a portion of the video frames or a portion of the video frames from the progress display frame 121 can be displayed in the progress display area 12. For example... Figure 2 As shown in (b), the progress display frame 121 displayed in the progress display area 12 includes: thumbnails of three complete video frames and a partial area of ​​a thumbnail of a video frame.

[0124] Typically, the video playback area 11 plays the video frame corresponding to the time indicated by the progress indicator line 122 in the progress display area 12.

[0125] In a specific implementation, there is a situation where the time corresponding to the progress indicator line 122 is the time between the playback times of two adjacent video frames, that is, there is no video frame in the video file corresponding to the time of the progress indicator line 122.

[0126] As an example, in a video frame sorted by playback time (denoted as timestamp) from smallest to largest, the playback time of the i-th video frame is ti, and the playback time of the (i+1)-th video frame is t(i+1); the time t corresponding to the progress indicator line 122 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval to the time t corresponding to the progress indicator line 122 can be taken as the video frame at the time corresponding to the progress indicator line 122. Therefore, it can also be understood that the video frame corresponding to a certain time is the one where the absolute value of the difference between the timestamp and that time is less than or equal to a first value (half the difference between the timestamps of two adjacent video frames in the video).

[0127] Of course, in practical applications, if there are no video frames in the video file whose timestamps and progress indicator lines correspond to the same time, the video frame preceding time t (the video frame corresponding to ti) of progress indicator line 122 can be used as the video frame corresponding to the time t of progress indicator line 122; or the video frame following time t (the video frame corresponding to tj) of progress indicator line 122 can be used as the video frame corresponding to the time t of progress indicator line 122. In practical applications, the progress display area 12 can display the playback progress in many ways. For example, it can also display the video playback progress using a progress bar or a progress ring. This application provides two playback modes for video playback: normal playback and frame playback.

[0128] The embodiments in this application are only for reference. Figure 2 The progress display area shown in (b) (in practical applications, other forms of progress display areas can also be used) serves as an example to illustrate the difference between normal playback and frame playback.

[0129] Reference Figure 2 (c) in the text represents the user's click. Figure 2After the control 111 (which can also be referred to as the playback control) in the interface shown in (b) is displayed, the electronic device responds to the operation by displaying an interface diagram during the playback of video A. This interface diagram indicates that the video A has been played to the 5th second, and the video frame corresponding to the 5th second is displayed in the video playback area 11; at the same time, the progress display frame 121 in the progress display area 12 moves its position so that the time corresponding to the progress indicator line 122 is the 5th second. Among them, from Figure 2 (b) in Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0130] A video file consists of multiple video frames, each with a timestamp. The timestamp records the playback time of each video frame during the video file's playback process. For example, arranged by timestamp from smallest to largest, the video frames in the file are: Video Frame 0 (timestamp t0), Video Frame 1 (timestamp t1), Video Frame 2 (timestamp t2), Video Frame 3 (timestamp t3), Video Frame 4 (timestamp t4), Video Frame 5 (timestamp t5), Video Frame 6 (timestamp t6), Video Frame 7 (timestamp t7)... Normal playback involves displaying each video frame sequentially in the playback area according to its corresponding timestamp, from smallest to largest.

[0131] As an example of a normal playback process, video frame 0 is displayed in video playback area 11 at time t0 after playback begins; video frame 1 is displayed in video playback area 11 at time t1 after playback begins; video frame 2 is displayed in video playback area 11 at time t2 after playback begins; video frame 3 is displayed in video playback area 11 at time t3 after playback begins; video frame 4 is displayed in video playback area 11 at time t4 after playback begins; video frame 5 is displayed in video playback area 11 at time t5 after playback begins; video frame 6 is displayed in video playback area 11 at time t6 after playback begins; video frame 7 is displayed in video playback area 11 at time t7 after playback begins...

[0132] Combination Figure 2 (b) in Figure 2 (c) shows the normal playback process of video A: starting from the 0th second of video A, each video frame is displayed sequentially according to the playback time indicated by the timestamp; Figure 2 (c) in the diagram is a schematic of the interface displaying the video frame up to the 5th second.

[0133] Reference Figure 2 In (d), the user drags... Figure 2 The progress display frame of the interface shown in (c) is shown in the image.

[0134] In response to Figure 2The drag operation (also referred to as the slide operation) shown in (d) in the figure displays the corresponding screen in the video playback area of ​​the electronic device.

[0135] As an example, refer to Figure 2 In step (e), when the user drags the progress bar to display a frame such that the progress indicator line corresponds to the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be a video frame in video A with a timestamp of 14 seconds; it can also be a video frame with a timestamp closest to 14 seconds; it can also be a video frame before 14 seconds whose time interval between timestamp and 14 seconds is closest; or it can be a video frame after 14 seconds whose time interval between timestamp and 14 seconds is closest.

[0136] It should be noted that, in subsequent embodiments, the specific meaning of the description method of "video frame corresponding to a certain moment" can be found in the detailed explanation of "video frame corresponding to the 14th second". From Figure 2 (d) in Figure 2 The playback process shown in (e) is the frame playback process of the video.

[0137] As the user drags the progress display frame 121, a relative position is formed between the progress indicator line 122 and the progress display frame 121. The position of the progress indicator line 122 on the progress display frame 121 is related to the time corresponding to the progress indicator line 122.

[0138] In the specific implementation, assuming that the time indicated by the progress indicator line 122 is t1 before the user moves in the area where the progress display frame 121 is located on the screen. When the user moves left or right in the area where the progress display frame is located, the electronic device can detect the user's real-time touch position at certain time intervals. The electronic device determines the user's movement direction and distance based on the continuous real-time touch positions. The electronic device determines the time change Δt based on the user's movement distance; and determines the calculation method for real-time time t2 based on the user's movement direction. For example, if the movement direction is left, the starting time t1 is subtracted from the time change Δt to obtain the real-time time t2; if the movement direction is right, the starting time t1 is added to the time change Δt to obtain the real-time time t2. After obtaining the real-time time t2, the electronic device finds the video frame corresponding to the real-time time t2 and displays the video frame corresponding to t2 in the video playback area.

[0139] On the other hand, the electronic device determines the movement direction of the progress display frame based on the movement direction determined by the real-time touch point position, and determines the movement distance of the progress display frame based on the movement distance of the real-time touch point position. The electronic device then moves the progress display frame in the progress display area according to the movement direction and distance, so that the time corresponding to the progress indicator line is t2. As another example, the electronic device also determines the movement direction (left if t2 is greater than t1, right if t2 is less than t1) and movement distance of the progress display frame based on the start time t1 and the real-time time t2. The electronic device then moves the progress display frame in the progress display area according to the movement direction and distance, so that the time corresponding to the progress indicator line is t2.

[0140] It should be noted that the above process is only an example. In actual applications, there are multiple ways for the user to determine the time corresponding to the progress indicator line 122, the position of the progress display frame 121, and the video frame displayed in the video playback area 11 while dragging the progress display frame 121.

[0141] Based on the above frame playback process, it can be understood that the process of the user dragging the progress to display the frame is an uncertain process. It may move to the left, it may move to the right, it may be dragged quickly, or it may be dragged slowly. Accordingly, the video frame images displayed in the video playback area 11 have randomness.

[0142] As an example, before the user moves within the area of ​​the progress display frame on the screen, the starting time indicated by the progress indicator is t1. During the user's dragging of the progress display frame, the electronic device detects the user's real-time touch position at regular time intervals, thus obtaining multiple consecutive time periods corresponding to the real-time times: t2, t3, t7, t4, and t3. In the video playback area, video frames are played in the following order: starting with the video frame corresponding to time t1, the video frames corresponding to time t2, t3, t7, t4, and t3 are played sequentially.

[0143] It can be understood that the process of a user clicking control 111 to trigger the electronic device to play video in the video playback area is normal playback; the process of a user dragging the progress display frame to trigger the electronic device to display the video frame corresponding to the progress indicator line in the video playback area is frame playback.

[0144] The operations shown in the embodiments of this application are merely examples. In practical applications, other operations can be set according to specific circumstances. For example, some single-click operations can also be double-click operations or other air gestures. Some drag operations can also be air gestures. Similarly, some examples of operations in subsequent embodiments can also be other operations.

[0145] In the above embodiments, whether it is normal playback or frame playback, the video frames need to be decoded before they are displayed in the video playback area. Usually, the video frames are decoded in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a keyframe (I-frame) and at least one forward reference frame (P-frame); of course, in practical applications, each video frame may also include at least one bidirectional reference frame (B-frame).

[0146] Reference Figure 3 This is a schematic diagram illustrating the playback and decoding order of a set of video frames provided in an embodiment of this application. A set of video frames can also be referred to as a group of pictures (GOP) or a group of video frames.

[0147] Each group of video frames, in playback order, includes: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame, and P-frame. I-frames, as keyframes, are decoded to obtain a complete image, but decoding is slower. P-frames, as non-keyframes, record changes relative to the previous frame (not a B-frame); information from the previous frame is required to decode the P-frame. Similarly, information from previous P-frames (which may also be I-frames) and subsequent P-frames is needed to decode the B-frame. P-frames and B-frames improve smoothness while reducing the video file size. Both P-frames and B-frames are non-keyframes.

[0148] Whether it's an I-frame, P-frame, or B-frame, they are all video frames. When decoding a video frame at a specific moment, you don't directly decode the video frame at that moment. Instead, you decode the group of video frames that the video frame at that moment belongs to. When decoding video frames within a group of video frames, you decode them one by one in the decoding order.

[0149] As an example, if we need to decode the video frame at time t1, and the video frame at time t1 is the 6th video frame in the playback order of a group of video frames: frame B; then the decoding order is:

[0150] First decoding: the video frame played in the first playback order: I-frame;

[0151] The second decoding: the 4th video frame in playback order: P-frame;

[0152] The third decoding step: the second video frame played in the playback order: frame B;

[0153] The fourth decoding: the third video frame in playback order: frame B;

[0154] The fifth decoding: the 7th video frame in playback order: P-frame;

[0155] The sixth decoding: the 5th video frame in playback order: frame B;

[0156] The seventh decoding: the 6th video frame in playback order: frame B;

[0157] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding operation to decode the video frame at time t1.

[0158] Currently, in normal playback scenarios, the video frames in each video frame group are parsed, decoded, rendered, and displayed sequentially according to the playback order of the video frame groups. Since normal playback follows the order of the video frame groups, pre-decoding and buffering are performed. Of course, when decoding the video frames in each video frame group, according to... Figure 3 The decoding order shown decodes each video frame in the video frame group sequentially. Therefore, in a normal playback scenario, each video frame can be displayed according to its timestamp, resulting in a relatively smooth video playback.

[0159] In frame-based playback scenarios, as mentioned earlier, the process of a user dragging the progress bar to display frames is unpredictable. The user may drag the progress bar to the left, or to the right, or drag it quickly or slowly. Consequently, the video frames displayed in the video playback area 11 are random. Therefore, during the process of the user dragging the progress bar to display frames, the electronic device needs to determine the moment the progress indicator line points based on the real-time detected touch point position. Then, it needs to parse, decode, render, and display the video frame at the moment the progress indicator line points. As mentioned earlier, before decoding the video frame at that moment, it may be necessary to decode other video frames in the same video frame group. Therefore, frame-based playback scenarios are prone to stuttering and unsmoothness.

[0160] Therefore, embodiments of this application provide a video playback method. The electronic device is equipped with at least two decoders for the frame playback process. By utilizing these at least two decoders to decode the frame playback process, decoding capabilities are improved, thereby enabling smoother frame playback of the video.

[0161] Reference Figure 4 This describes a video playback process that uses at least two decoders for decoding, as provided in the embodiments of this application.

[0162] When a user drags a progress display frame, triggering the electronic device to display a video frame at position 1 (different from the actual position of the progress display frame on the screen; position 1 is an abstract position corresponding to a video frame), decoder 1 (also referred to as the first decoder) in the electronic device begins decoding video frame group 1 (also considered an example of the first frame group) containing the video frame at position 1. The decoded video frame (the video frame in video frame group 1) is then cached in the first cache space corresponding to decoder 1. If decoder 1 determines that video frame group 1 containing position 1 is being decoded, decoder 2 (also referred to as the second decoder) begins decoding the next video frame group 2 (also considered an example of the second frame group) within video frame group 1, and caches the decoded video frame (the video frame in video frame group 2) in the second cache space corresponding to decoder 2. The first cache space is where the video frames decoded by decoder 1 are stored; the second cache space is where the video frames decoded by decoder 2 are stored. Typically, when a decoder decodes video frames in a video frame group, it decodes the video frames in the decoding order, and for each decoded video frame, the decoded video frame is cached in the corresponding cache space.

[0163] In this example, the relationship between position 1 and the position of the first control on the screen is as follows: when the user drags the progress display frame, triggering the electronic device to display the video frame corresponding to position 1 (which is different from the actual position of the progress display frame on the screen; position 1 is an abstract position corresponding to the video frame), the first control will be moved to the second position. The position of the first control can be the position of the top-left corner of the area where the first control is located, or the position of the top-right corner of the area where the first control is located, or even the position of any pixel within the first control.

[0164] The progress display frame, as an example of the first control, can move along a first trajectory in response to the user's swipe. The position of the first control before moving with the user's swipe can be recorded as the first position (i.e., the initial position). In other words, with the user's swipe, the first control can start moving from the first position.

[0165] As the user swipes, the first control moves to a certain position. The playback progress (or playback moment) represented by the first control at that position corresponds to the video frame displayed in the first area. For example, the video frame displayed in the first area is the video frame corresponding to that playback moment.

[0166] When the user drags the progress bar to display a frame, triggering the electronic device to show the video frame corresponding to position 2, the video frame group 1 containing the video frame at position 2 has already been assigned to decoder 1 for decoding. In practical applications, the video frame at position 2 may already be cached in the first cache space. Therefore, the electronic device can retrieve the decoded video frame at position 2 from the first cache space and perform subsequent rendering and display. Since the decoded video frame can be directly retrieved from the first cache space and then rendered and displayed, the smoothness of the picture can be improved and stuttering can be reduced. Of course, since the next video frame group 2 in the video frame group 1 corresponding to position 2 has already been assigned to the decoder for decoding, there is no need to decode it again.

[0167] When the user drags the progress display frame, triggering the electronic device to display the video frame corresponding to position 3, the first control will be moved to the third position (the difference between the playback progress indicated by the first control when it is in the third position and the timestamp of the video frame corresponding to position 3 is less than or equal to the first duration). The video frame corresponding to position 3 has already been assigned to decoder 2 for decoding (it was assigned to decoder 2 for decoding when the first control was moved to the second position by the user's swipe operation). In practical applications, the video frame corresponding to position 3 may already be cached in the second cache space. Therefore, the electronic device can directly obtain the decoded video frame corresponding to position 3 from the second cache space and perform subsequent rendering and display. In addition, when the electronic device is triggered to display the video frame corresponding to position 3, the electronic device will trigger decoder 1 to start decoding the next video frame group 3 (which can be referred to as the third frame group) of the video frame group 2 containing the video frame corresponding to position 3, and cache the decoded video frame in video frame group 3 in the first cache space.

[0168] When the user drags the progress display frame, triggering the electronic device to display the video frame corresponding to position 4, the first control will be moved to the fourth position (the difference between the playback progress indicated by the first control when it is in the fourth position and the timestamp of the video frame corresponding to position 4 is less than or equal to the first duration). The video frame corresponding to position 4 has already been assigned to decoder 1 for decoding (it was assigned to decoder 1 for decoding when the first control was moved to the third position by the user's swipe operation). Decoder 1 has cached the decoded video frame in video frame group 3 (as an example of the fourth frame group) in the first cache space. The electronic device can directly obtain the video frame corresponding to position 4 (as an example of the fourth video frame) from the first cache space and perform subsequent rendering and display. Of course, the decoding task of video frame group 4 also needs to be assigned to decoder 2.

[0169] Of course, as the user slides, when the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 5, the electronic device begins to decode video frame group k through the first decoder, decode video frame group k+1 through the second decoder, and simultaneously displays the video frame in video frame group k (for example, the video frame corresponding to position 5).

[0170] In this embodiment of the application, when the playback progress determined by two consecutive touch point positions corresponds to the playback progress corresponding to position 4 and position 5 respectively, since the video frame group k corresponding to position 5 and the video frame group 3 corresponding to position 4 are not two adjacent video frame groups, when determining the video frame corresponding to position 5 to be displayed based on the touch point position, the video frame group K may not have been decoded in advance. In this case, the parser can assign the video frame K to other decoders other than the decoder assigned when the decoding task was assigned last time.

[0171] For example, in the previous allocation of decoding tasks, the decoding task for video frame group 4 was assigned, and this task was assigned to decoder 2. Therefore, in this allocation of decoding tasks, the task of decoding video frame group k will be assigned to decoder 1, and the task of decoding video frame group k+1 will be assigned to decoder 2. Figure 4 In the example shown, K is a natural number greater than 5.

[0172] Based on the above process, it can be understood that when a drag operation triggers the electronic device to display video frames in video frame group j, another decoder (a decoder different from the decoder corresponding to video frame group j) will decode video frame group j+1. When the drag operation triggers the electronic device to display video frames in video frame group j+1, the video frames in video frame group j+1 have already been decoded when the display of video frames in video frame group j was triggered and cached in the corresponding cache space. The electronic device can directly obtain the decoded video frames in video frame group j+1 from the cache space for rendering and display, thereby improving the smoothness of the picture when playing the frames.

[0173] Of course, when the drag operation triggers the electronic device to display a video frame in video frame group j+1, there will also be a decoder decoding the video frame in video frame group j+2. In this way, there will always be a decoder that performs the decoding work in advance. When the user actually drags to the corresponding position to trigger the display of a certain video frame, the corresponding video frame at that moment may have already been decoded and cached, thus improving the smoothness of the picture when the frame is played.

[0174] In addition, in practical applications, only one decoder can be set up for frame playback scenarios, and each video frame group is decoded by this decoder. However, if the decoding capability of this decoder is weak (the decoding speed is slow, and the decoding task of the next video frame group arrives before the decoding of the previous video frame group is finished), then two or even more decoders can be set up to decode in turn.

[0175] As an example, when three decoders are set, if the user's drag operation triggers the electronic device to continuously display video frames in video frame group j, video frame j+1, video frame group j+2, and video frame group j+3, then video frame group j is decoded by the first decoder, video frame group j+1 is decoded by the second decoder, video frame group j+2 is decoded by the third decoder, and video frame group j+3 is decoded by the first decoder.

[0176] Of course, user dragging operations can be left or right; therefore, correspondingly, when the dragging process triggers the display of a video frame with a smaller timestamp (smaller than the timestamp of the previously displayed video frame), for example, when the dragging operation triggers the electronic device to display a video frame in video frame group j, another decoder (a decoder different from the decoder corresponding to video frame group i) will decode video frame group j-1; when the dragging operation triggers the electronic device to display a video frame in video frame group j-1, the video frame in video frame group j-1 may have already been decoded and cached in the cache space, and the electronic device can directly obtain the decoded video frame from the cache space for rendering and display, thereby improving the smoothness of the picture when the frame is played.

[0177] Of course, when the drag operation triggers the electronic device to display a video frame in video frame group j-1, there will also be a decoder decoding the video frame in video frame group j-2. In this way, there will always be a decoder that performs the decoding work in advance. When the user actually drags to the corresponding position to trigger the display of a certain video frame, the video frame may have already been decoded and cached, thus improving the smoothness of the picture when the frame is played.

[0178] In the example above, if a swipe operation triggers the electronic device to display a video frame at a certain moment, the video frame whose timestamp differs from that moment from the cache space is less than a first value (half the difference between the timestamps of two adjacent video frames in the video). That is, the video frame with the smallest difference between its timestamp and that moment among the video frames in the video file.

[0179] In addition, in practical applications, since the decoder also needs time to decode a set of video frames, when the user drags quickly and triggers the electronic device to display a video frame at a certain moment, the decoder may not have yet decoded the video frame corresponding to that moment in the set of video frames. Therefore, it can also be set that when the user drags quickly, if the video frame to be displayed has not been successfully decoded, the most recent (smallest difference) decoded video frame corresponding to the drag operation can be obtained from the cache space for rendering and display; when the user drags quickly enough, only the key frames in a set of video frames may be decoded and rendered for display.

[0180] The above embodiments all use two decoders to decode alternately, and always keep one decoder to decode one video frame group in advance. In practical applications, more than two decoders can be set to decode alternately, for example, decoding in the order of decoder one, decoder two and decoder three, decoder one, decoder two, decoder three, while always keeping at least one decoder to decode at least one video frame group in advance (for example, it can also be set that two decoders always decode two video frame groups in advance). The embodiments of this application will not be listed one by one.

[0181] To better illustrate the video playback method provided in this application, the following describes the video playback method provided in the embodiments of this application in detail using timing diagrams.

[0182] Figure 5 A timing diagram illustrating the preparatory work before frame playback provided in this embodiment of the application. This process includes the following steps:

[0183] S101, the gallery application displays a grid chart, which includes the cover of video A (as an example of the first video).

[0184] The gallery app can display grid images that include not only the cover art for video A, but also the cover art for other videos, and of course, thumbnails of images. Figure 2 The grid diagram shown in (a) can serve as an example of the grid diagram used in this step.

[0185] S102, the gallery application receives a click operation on the cover of video A.

[0186] In this embodiment of the application, the electronic device displays... Figure 2 When the user clicks on the interface shown in (a), Figure 2 In the interface shown in (a), the grid display shows the cover art of any video. In response to this action, the electronic device displays the video playback interface for that video via the gallery application. This click action can be referenced... Figure 2 The user's finger tapping action in the interface shown in (a).

[0187] It should be noted that in practical applications, other modules of the electronic device may receive the click operation and then transmit the click operation to the gallery application. For the sake of simplicity, the intermediate steps are omitted in the embodiments of this application. Taking the gallery application as an example, it can receive the click operation on the cover of video A.

[0188] S103, after receiving a click operation on the cover of video A, the gallery application sends the file path of video A to the frame playback controller.

[0189] In this embodiment of the application, the gallery application of the electronic device stores the file paths of various pictures and video files in the electronic device. The gallery application can display pictures and play videos based on the file paths of various pictures and video files.

[0190] S104, after receiving the file path of video A, the frame playback controller sends the file path of video A to the frame playback scheduler.

[0191] S105, after receiving the file path of video A, the frame playback scheduler sends the file path of video A to the parser.

[0192] S201, After sending the file path of video A to the frame playback controller, the gallery application sends a frame playback start request to the frame playback controller.

[0193] In this embodiment of the application, S201 is executed after step S103, but there is no strict order of execution between S201, S104 and S105.

[0194] S202, after receiving the frame playback start request, the frame playback controller sends the frame playback start request to the frame playback scheduler.

[0195] S203, after receiving the frame playback start request, the frame playback scheduler sends a parsing thread start request to the parser.

[0196] S204, After receiving the request to start the parsing thread, the parser starts the parsing thread.

[0197] S205, after successfully starting the parsing thread, the parser sends a message to the frame playback scheduler indicating that the parsing thread has started successfully.

[0198] S206, the frame playback scheduler sends a request to decoder 1 to start decoding thread 1.

[0199] S207, after receiving the request to start decoding thread one, decoder one starts decoding thread one.

[0200] S208, after successfully starting decoding thread one, decoder one sends a message to the frame playback scheduler indicating that decoding thread one has started successfully.

[0201] S209, the frame playback scheduler sends a request to decoder 2 to start decoding thread 2.

[0202] S210, after receiving the request to start decoding thread two, decoder two starts decoding thread two.

[0203] S211, after successfully starting decoding thread two, decoder two sends a message to the frame playback scheduler indicating that decoding thread two has started successfully.

[0204] In this embodiment of the application, more decoders can also be set to improve the decoding capability of the electronic device. If more decoders are set for frame playback, they can be sequentially referred to as decoder three, etc.

[0205] S212, the frame playback scheduler sends a rendering thread start request to the rendering module.

[0206] S213, After receiving the rendering thread start request, the rendering module starts the rendering thread.

[0207] S214, after successfully starting the rendering thread, the rendering module sends a message to the frame playback scheduler indicating that the rendering thread has started successfully.

[0208] S215, after receiving a click operation on the cover of video A, the gallery application displays the video playback interface of video A.

[0209] The video playback interface of video A can be referenced. Figure 2 The interface shown in (b) of the video A includes a video playback area and a progress display area for displaying the playback progress.

[0210] It should be noted that in response to the click operation, the electronic device is simultaneously executing steps S103 to S214 and displaying the video playback interface of video A (S215). Therefore, there is no strict order between steps S103 (and subsequent steps up to S214) and step S215.

[0211] After this step, preparations for frame playback are complete. Upon receiving the command for frame playback, the electronic device will execute frame playback of the video.

[0212] This application mainly describes the frame playback process of video to illustrate how the smoothness of the frame playback process can be improved by using at least two decoders.

[0213] Reference Figure 6 , Figure 6 This is a timing diagram of the video frame corresponding to position 1 in the video playback process provided in this embodiment. S301, the gallery application receives a drag operation on the progress display frame.

[0214] For electronic devices including touchscreens, the touch driver can sample user touch operations at intervals T. This interval, also known as the deployment cycle or reporting cycle, is the period during which the touch driver collects user touch operations.

[0215] Taking a touch driver that collects data 240 times per second as an example, the period T = 1 / 240 = 4.16 ms, meaning the touch driver collects touch data once every 4.16 ms. If, during a certain period, the user presses the touchscreen with one or two fingers, the touch driver can determine the touch position of the single or two fingers based on voltage changes, etc.

[0216] For example, when a user wants to change the position of the progress display frame by dragging it to change the video frame displayed in the video playback area, the touch driver receives the user's click operation corresponding to the progress display frame and passes a down event to the application processor. The application processor passes the down event to the gallery application. After receiving the click event, the touch driver continues to receive the user's continuous drag operation corresponding to the progress display frame and passes a move event to the application processor. The application processor passes the move event to the gallery application.

[0217] S302, the gallery application determines the time ti for the playback of a frame of video A based on the drag position (as an example of the first moment).

[0218] The frame playback time *ti* is not the current time, but rather the video frame corresponding to time *ti* in the video file. The explanations for subsequent times *ti+1*, *ti+2*, etc., will not be repeated.

[0219] In this embodiment, a continuous drag operation includes: the user's finger touching the progress display frame (down event), the user's finger moving (move event), and the user's finger releasing from the progress display frame (up event). The time ti can be the playback progress corresponding to the first touch point position in the move event, or it can be the playback progress corresponding to any touch point position in the move event.

[0220] After the gallery application determines the playback time ti of video A's frame based on the drag position, it needs to move the position of the progress display frame and display the video frame corresponding to time ti in the video playback area.

[0221] As an example, and Figure 4 The example shown corresponds to the video frame at time ti. Figure 4 The video frame corresponding to position 1 in the image.

[0222] S303, the gallery application sends a frame playback request to the frame playback controller, carrying the time ti of video A.

[0223] S304 After receiving the frame playback request, the frame playback controller sends the frame playback request to the frame playback scheduler, carrying the time ti of video A.

[0224] S306, after receiving a frame playback request, the frame playback scheduler sends a parsing request to the parser, carrying the time ti of video A.

[0225] S307, after receiving a frame playback request, the frame playback scheduler sends the moment ti of video A to the rendering module.

[0226] S309, after receiving the parsing request, the parser parses video A through the parsing thread to obtain the video frame group j (and the video frame group j corresponding to the video frame ti) of video frame A. Figure 4 The example shown corresponds to the encoded video frames in video frame group 1), and the start time ts and end time te of video frame group j are recorded.

[0227] The timestamp of the first video frame in video frame group j can be used as the start time ts, and the timestamp of the last video frame in video frame group j can be used as the end time te.

[0228] In practical applications, ts-1 / 2*T2 and te+1 / 2*T2 can also be recorded as the start and end times. T2 is the difference between the timestamps of two adjacent video frames.

[0229] S310, the parser sends a decoding request to the decoder, carrying the encoded video frames in video frame group j (which can be all encoded video frames in video frame group j).

[0230] When the parsing module determines that the encoded video frame in video frame group j is decoded by decoder one, it determines that another decoder (decoder two) is used to decode the video frame in the next video frame group j+1 of video frame group j. That is, the decoded video frame is obtained by decoding the encoded video frame in video frame group j+1 by decoder two.

[0231] In the actual implementation, it is also necessary to distinguish between forward playback and reverse playback. As an example, if ti is greater than ti-1, it is forward playback. The playback progress corresponding to subsequent drag operations is likely to be greater than ti as well. Therefore, the next video frame group j+1 of the video frame group j where ti is located can be pre-decoded. The next video frame group is the video frame group adjacent to the video frame group j where the video frame at time ti is located, and whose timestamp is greater than that of the video frame group j.

[0232] Of course, if ti is less than ti-1, then it is reverse playback. The playback progress corresponding to subsequent drag operations is likely to be less than ti. Therefore, the previous video frame group j-1 of the video frame group j where ti is located can be pre-decoded. The previous video frame group is the video frame group adjacent to the video frame group j where the video frame corresponding to time ti is located, and whose timestamp is less than that of video frame group j.

[0233] The steps described above for determining whether to play forward or backward can be performed by the gallery application. Then, the playback mode (forward or backward) is transmitted to the parser. The parser determines the previous video frame group (when playing backward) or the next video frame group (when playing forward) of the video frame corresponding to time ti based on the playback mode.

[0234] As another example, the playback method can also be determined by a frame playback controller, frame playback scheduler, or parser based on the relationship between the currently received time and the last received time. This application will not provide examples of each of these in the illustrations.

[0235] S311, the parser sends a decoding request to decoder 2, carrying the encoded video frame in video frame group j+1.

[0236] and Figure 4 In the embodiment shown, video frame group j+1 can be understood as... Figure 4 Video frame group 2 in the video.

[0237] S312, after receiving the decoding request, decoder 1 decodes the encoded video frame in video frame group j through decoding thread 1 to obtain the decoded video frame.

[0238] In this embodiment, decoder one can cache the decoded video frames in the first cache space according to the decoding order.

[0239] S313, after receiving the decoding request, decoder 2 decodes the encoded video frame in video frame group j+1 through decoding thread 2 to obtain the decoded video frame.

[0240] In this embodiment of the application, decoder 2 can cache the decoded video frames in the second cache space according to the decoding order.

[0241] S314, the rendering module determines that the decoder of the video frame group j corresponding to the video frame at decoding time ti is decoder one.

[0242] In this embodiment, the rendering module can determine the decoder of the video frame group j to which the video frame corresponding to ti belongs by polling.

[0243] As an example, the rendering thread searches for the first video frame (the first video frame is the key frame in the video frame group) in the first cache space corresponding to decoder 1. If the difference between ti and the timestamp of the first video frame in the first cache space is less than the time span T1 of a video frame group (the difference between the timestamps of the key frames in two adjacent video frame groups), it means that the video frame group j corresponding to the video frame at time ti is decoded by decoder 1.

[0244] In this example, the timestamp of the first video frame in the first buffer space is subtracted from ti to obtain the second difference; if the second difference is in the range [0, T1), then the video frame group j corresponding to the video frame at time ti is determined to be decoded by decoder one.

[0245] That is, [0, T1) can be used as an example of the first time range.

[0246] Alternatively, [0-1 / 2*T2, T1-1 / 2*T2) can be set as the first time range, where T2 is the difference between the timestamps of two adjacent video frames.

[0247] Correspondingly, if the difference between the timestamp of ti and the first video frame in the first buffer space is not less than the time span of a video frame group (the difference in timestamps of keyframes in two adjacent video frame groups), then the first video frame (the first video frame is the keyframe in the video frame group) is retrieved from the second buffer space corresponding to decoder 2. If the difference between the timestamp of ti and the first video frame in the second buffer space is less than the time span of a video frame group (the difference in timestamps of keyframes in two adjacent video frame groups), it means that the video frame group j containing the video frame at time ti is decoded by decoder 2. In this way, the decoder corresponding to the video frame group j containing the video frame at time ti can be determined.

[0248] S315, the rendering module obtains the video frame corresponding to time ti from decoder one.

[0249] In this embodiment, the decoder containing the video frame group j corresponding to time ti is taken as decoder one. After determining that the decoder containing video frame group j is decoder one, the video frame whose timestamp is less than or equal to 1 / 2*T2 (the difference between the timestamps of two adjacent video frames) is obtained from decoder one as the video frame corresponding to time ti. This is because ti is the time determined by the user's drag operation. In practical applications, there may not be a video frame with timestamp ti. In this case, the video frame with the timestamp closest to ti in the group of video frames is obtained as the video frame corresponding to time ti.

[0250] S316, the rendering module renders the video frame corresponding to time ti through the rendering thread to obtain the picture corresponding to time ti.

[0251] S317, the rendering module sends the frame corresponding to time ti to the frame playback scheduler.

[0252] S318, after the frame playback scheduler receives the frame corresponding to time ti, it sends the frame playback controller the frame corresponding to time ti.

[0253] S319, after receiving the frame corresponding to time ti, the frame playback controller sends the frame corresponding to time ti to the gallery application.

[0254] S320, after receiving the image corresponding to time ti, the gallery application displays the image corresponding to time ti in the video playback area (as an example of the image corresponding to the second video frame).

[0255] In practical applications, after S302, the gallery application also needs to determine the position of the first control based on the real-time touch point position of the swipe operation, thereby moving the first control in the video playback interface so that the first control moves to a specific position, and the playback progress represented by the first control at that specific position is time ti.

[0256] Of course, taking a single swipe operation by the user as an example (including down, move, and up events), this swipe operation will trigger the first control to move along the first trajectory. When the down event is detected, the first control is currently at the first position, and the first control moves along the fixed first trajectory. The move event in this swipe operation will trigger the first control to move from the first position along the first trajectory.

[0257] As the user drags the progress display frame, the position of the user's finger touch detected by the electronic device may change. Correspondingly, the positional relationship between the progress indicator line and the progress display frame also changes, and the time corresponding to the progress indicator line also changes. If the times corresponding to two consecutive detected touch positions are ti and ti+1, the process of the electronic device displaying the video frame corresponding to ti+1 can be referenced... Figure 7 The frame playback process shown is such that the video frame group corresponding to time ti+1 is the same as the video frame group corresponding to time ti. However, the video frame corresponding to time ti+1 can be... Figure 4 The video frame corresponding to position 2 in the text. Figure 7 The frame playback process shown includes the following steps:

[0258] S401, the gallery application receives a drag operation applied to the progress display frame.

[0259] It should be noted that the drag operation in step S401 and step S301 is the same continuous operation. That is, the user's drag operation is not released between step S401 and step S301. It is only because the drag operation causes the time corresponding to the progress indicator line to change.

[0260] S402, the gallery application determines the playback time ti+1 of the frame of video A based on the drag position (which can be used as an example of the second moment).

[0261] As an example in this application, the video frame with the smallest difference between the timestamp and the time ti can be... Figure 4 The video frame corresponding to position 2 in the image.

[0262] S403, the gallery application sends a frame playback request to the frame playback controller, carrying the time ti+1 of video A.

[0263] S404: After receiving a frame playback request, the frame playback controller sends a frame playback request to the frame playback scheduler, carrying the time ti+1 of video A.

[0264] S405, after receiving a frame playback request, the frame playback scheduler calculates the difference between time ti+1 in the current frame playback request and time ti in the previous frame playback request.

[0265] If the difference between ti+1 and ti is less than 16.7ms, ti+1 is discarded, and S406 and subsequent steps are not executed.

[0266] S406, if the difference between ti+1 and ti is greater than or equal to 16.7ms, the frame playback scheduler sends a parsing request to the parser, carrying the time ti+1 of video A, and records the time ti+1.

[0267] Generally, the human eye cannot perceive differences in video playback at frame rates of 30fps and above. Differences are only perceptible when the frame rate is below 30fps. Therefore, the video frame rate can be filtered to a suitable range. For example, filtering to 30fps results in a frame interval of 1000ms / 30 = 33.3ms, where 33.3ms is the difference threshold. In practical applications, if the frame rate is 120fps, it can also be filtered to 60fps, resulting in a frame interval of 1000ms / 60 = 16.7ms, where 16.7ms is the difference threshold.

[0268] Alternatively, as a compromise, filtering the frame rate to between 30fps and 60fps can also reduce the number of video frames in the video, which can significantly reduce the power consumption of the decoder.

[0269] Based on the above principles, 16.6ms and 33.3ms are just examples of difference thresholds. In practical applications, it can also be a value around 16.6ms, such as 15ms, 17ms, 18ms, 20ms, etc., or a value around 33.3ms; of course, it can also be any value between 16.6ms and 33.3ms.

[0270] Reference Figure 8This describes the video filtering process. Taking filtering to 60fps as an example, the interval between video frames is 1000ms / 60 = 16.7ms. The difference threshold is 16.7ms.

[0271] As the user drags the progress bar to display frames, the electronic device determines the following moments based on the successive touch point positions:

[0272] 11:43:46.646-3013000, 11:43:46.659-3784000, 11:43:46.676-4758000, 11:43:46.693-5528000, 11:43:46.710-6389000;

[0273] The first touch point location is determined at 11:43:46.646-3013000; this time is retained for subsequent decoding, rendering, and display.

[0274] The second contact point position was determined at 11:43:46.659-3784000. The difference between this and the previously retained time (11:43:46.646) is 659-646=13ms, which is less than 16.7ms. Therefore, this time is discarded.

[0275] The third touch point position is determined at 11:43:46.676-4758000. The difference between this and the previous reserved time (11:43:46.659) is 676-659=17ms, which is greater than 16.7ms. This time is reserved for subsequent decoding, rendering and display.

[0276] The fourth touch point position is determined at 11:43:46.693-5528000. The difference between this and the previous reserved time (11:43:46.676) is 693-676=17ms, which is greater than 16.7ms. This time is reserved for subsequent decoding, rendering and display.

[0277] The fifth touch point position is determined at 11:43:46.710-6389000. The difference between this and the previous reserved time (11:43:46.693) is 710-693=17ms, which is greater than 16.7ms. This time is reserved for subsequent decoding, rendering and display.

[0278] S407, if the difference between ti+1 and ti is greater than or equal to 16.7ms, the frame playback scheduler sends the time ti+1 of video A to the rendering module.

[0279] S408, after receiving the parsing request, the parser determines that ti+1 is within the recorded start time ts and end time te.

[0280] In this application, the timestamps of the first video frame and the timestamp te of the last video frame in the video frame group where the previously obtained time ti is located have been recorded in S309. It can be determined whether ti+1 is greater than or equal to ts and less than or equal to te. If ti+1 is greater than or equal to ts and less than or equal to te, it means that the video frame corresponding to ti+1 is in video frame group j (the video frame group where the video frame corresponding to ti is located). If ti+1 is less than ts or greater than te, it means that the video frame corresponding to ti+1 is not in video frame group j.

[0281] After the parser determines that the video frame corresponding to timestamp ti+1 is in video frame group j, since video frame group j has already been handed over to decoder one for decoding, video A is no longer parsed again, nor is the decoding instruction of video frame group j sent to the decoding module, because the video frame group j where the video frame ti+1 is located has already been parsed, and the decoder has also decoded the video frame group j where the video frame corresponding to ti+1 is located in order.

[0282] In the embodiments of this application, Figure 4 The video frame corresponding to position 2 in the video frame j can be understood as the video frame with the smallest difference between the timestamp and ti+1.

[0283] S414, the rendering module determines that the video frame group j corresponding to time ti+1 is decoded by decoder one.

[0284] The process of the rendering module determining the video frame group j corresponding to time ti+1 and decoding it by the decoder can be referred to the description in the above embodiment, and will not be repeated here.

[0285] S415, the rendering module obtains the video frame corresponding to time ti+1 from decoder one.

[0286] For example, a video frame whose timestamp and ti+1 differ from the first buffer space corresponding to decoder 1 is obtained, and the first value is half the difference between the timestamps of two adjacent video frames.

[0287] S416, the rendering module renders the video frame corresponding to time ti+1 through the rendering thread to obtain the picture corresponding to time ti+1.

[0288] S417, the rendering module sends the frame corresponding to time ti+1 to the frame playback scheduler.

[0289] S418, after the frame playback scheduler receives the frame corresponding to time ti+1, it sends the frame playback controller the frame corresponding to time ti+1.

[0290] S419, after receiving the frame corresponding to time ti+1, the frame playback controller sends the frame corresponding to time ti+1 to the gallery application.

[0291] S420: After receiving the image corresponding to time ti+1, the gallery application displays the image corresponding to time ti+1 in the video playback area.

[0292] Similarly, the gallery application also needs to determine the position of the first control based on the real-time touch point position of the swipe operation, so as to move the first control in the video playback interface, so that the first control moves to a specific position, and the playback progress represented by the first control at that specific position is time ti+1.

[0293] The following is based on Figure 9 Describe the video playback process performed by the electronic device when a drag operation triggers the display of the video frame corresponding to position 3 (the video frame whose timestamp is less than the difference between timestamp and ti+2 is less than the first value).

[0294] S501, the gallery application receives a drag operation applied to the progress display frame.

[0295] It should be noted that the drag operations in steps S501, S401, and S301 are the same continuous operations. That is, the user's drag operation between steps S501 and S301 is not released, but the time corresponding to the progress indicator line changes due to the drag operation.

[0296] S502, the gallery application determines the playback time ti+2 of the frame of video A based on the drag position (which can be used as an example of the second moment).

[0297] S503, the gallery application sends a frame playback request to the frame playback controller, carrying the time ti+2 of video A.

[0298] S504: After receiving a frame playback request, the frame playback controller sends a frame playback request to the frame playback scheduler, carrying the time ti+2 of video A.

[0299] S505, after receiving a frame playback request, the frame playback scheduler calculates the difference between the time ti+2 in the current parsing request and the time ti+1 carried in the previous parsing request, and determines that the difference is greater than 16.7ms.

[0300] S506, if the difference between ti+2 and ti+1 is greater than or equal to 16.7ms, the frame playback scheduler sends a parsing request to the parser, carrying the time ti+2 of video A, and records the time ti+2.

[0301] In practical applications, if the times corresponding to consecutive contact positions are ti and ti+2, then it is necessary to calculate the difference between ti+2 and ti. This embodiment of the application takes the times corresponding to consecutive contact positions as ti, ti+1, and ti+2 as an example.

[0302] S507, the frame playback scheduler sends a rendering request to the rendering module, carrying the time ti+2.

[0303] S508, after receiving the parsing request sent by the frame playback scheduler, the parser determines that the time ti+2 is not within the range of the recorded start time ts and te.

[0304] S509, the parser parses video A and obtains the video frame group j+1 (video frame group 2) where the video frame corresponding to ti+2 is located, and records the start time ts and te of video frame group j+1.

[0305] In this application, after parsing video A to obtain a new group of video frames, the ts and te of the newly obtained video frame group are recorded. The previously recorded ts and te are then overwritten.

[0306] S510, the parser queries that video frame group j+1 has been decoded by decoder two.

[0307] In this case, it is not necessary to repeatedly decode video frame group j+1 through the decoder.

[0308] S511, the parser sends a decoding request to decoder 1, carrying video frame group j+2 (equivalent to...). Figure 4 The encoded video frames in video frame group 3).

[0309] S513, decoder one decodes the encoded video frame in video frame group j+2 through decoding thread one to obtain the decoded video frame.

[0310] S514, the rendering module determines that the video frame group j+1 corresponding to ti+2 is decoded by decoder two.

[0311] The process by which the rendering module determines the decoder corresponding to video frame group j+1 containing the video frame corresponding to ti+2 is as follows: First, the timestamp of the first keyframe cached in decoder one (or in the first cache space corresponding to decoder one) is queried. Based on the relationship between this timestamp and ti+2, it is determined that the decoder of video frame group j+1 is not decoder one (the second difference between ti+2 and the queried timestamp is not within the first time range). If it is determined that it is not decoder one, then the timestamp of the first keyframe cached in decoder two (or in the second cache space corresponding to decoder two) is queried. Based on the relationship between this timestamp and ti+2, it is determined that the decoder of video frame group j+1 is decoder two (the second difference between ti+2 and the queried timestamp is within the first time range). The specific process of determining the decoder can also refer to the description in the above embodiment, and will not be repeated here.

[0312] S515, the rendering module obtains the video frame corresponding to ti+2 from decoder 2. As another embodiment of this application, in practical applications, when the user drags quickly, the following situation may occur:

[0313] Decoder 2 needs to follow Figure 3 The decoding order shown is used. If the difference between the timestamp and ti+2 is less than the first value (half the difference between the timestamps of two adjacent video frames), the video frame is the 7th decoded video frame (B frame). When the user's drag operation triggers the electronic device to display the video frame corresponding to ti+2 (the video frame corresponding to position 3), due to the fast drag operation, although decoder 2 decodes this group of video frames in advance, after the electronic device executes S514, decoder 2 has not yet decoded the video frame corresponding to position 3 (the 7th video frame). In this case, it can also be set that the rendering module obtains the video frame with the smallest difference between the timestamp and ti+2 from the second buffer space (the buffer space corresponding to decoder 2), and this video frame is used as the video frame corresponding to time ti+2. That is, the video frame with the smallest difference between the currently successfully decoded video frames and ti+2 is used as the video frame corresponding to time ti+2 for subsequent rendering and display. This can also obtain video frames for rendering and display as soon as possible, reducing screen stuttering.

[0314] As an example, the rendering module can sequentially calculate the difference between the timestamp of each video frame already cached in the second cache space and ti+2, and select the video frame with the smallest difference as the video frame corresponding to time ti+2.

[0315] S516, the rendering module renders the video frame corresponding to time ti+2 through the rendering thread to obtain the picture corresponding to time ti+2.

[0316] S517, the rendering module sends the frame corresponding to time ti+2 to the frame playback scheduler.

[0317] S518, after the frame playback scheduler receives the frame corresponding to time ti+2, it sends the frame playback controller the frame corresponding to time ti+2.

[0318] S519: After receiving the frame corresponding to time ti+2, the frame playback controller sends the frame corresponding to time ti+2 to the gallery application.

[0319] S520, after the gallery application receives the frame corresponding to time ti+2, it displays the frame corresponding to time ti+2 in the video playback area (as an example of the frame corresponding to the third video frame).

[0320] In practical applications, users' dragging operations may be very fast. In this case, in order to enable electronic devices to quickly display the decoded screen, it can be set that each decoder only decodes the I-frames in a group of video frames. Accordingly, when the rendering thread obtains the video frame corresponding to a certain moment, it can obtain the I-frame from the cache space of the determined decoder (the decoder used to decode the video frame group where the video frame corresponding to that moment is located), and the obtained I-frame is used as the video frame corresponding to that moment.

[0321] As an example, when a drag operation triggers the electronic device to display the video frame corresponding to time tk (e.g., Figure 4 The video frame corresponding to position 5 in the diagram is shown below. Positions 5 and 4 are the video frames corresponding to the playback times of two consecutive touch points detected by the electronic device. For example, if the playback time corresponding to the previous touch point detected by the electronic device is ti+3, and position 4 is a video frame whose difference between the timestamp and ti+3 is less than the first value, theoretically, the electronic device should display the video frame corresponding to position 4. Similarly, if the playback time corresponding to the next touch point detected by the electronic device is ti+4, and position 5 is a video frame whose difference between the timestamp and ti+4 is less than the first value, theoretically, the electronic device should display the video frame corresponding to position 5. However, due to the user's fast dragging operation, the following situation may occur:

[0322] After the electronic device executes the decoder that determines the video frame group where ti+4 is located, if the decoder has not yet decoded the video frame corresponding to position 5, it can obtain the video frame with the smallest difference between the timestamp and ti+4 from the cache space of the determined decoder as the video frame corresponding to time ti+4 for rendering and display; or it can directly obtain the first video frame (keyframe) from the cache space of the determined decoder as the video frame corresponding to time ti+4 for rendering and display.

[0323] As an example, the rendering thread can calculate the difference between the currently received time ti+4 and the last received time ti+3. When the difference is large, the first video frame (keyframe) is directly obtained from the cache space of the determined decoder as the video frame corresponding to time ti+4 for rendering and display. When the difference is small, the video frame with the smallest difference between the timestamp and ti+4 is obtained from the cache space of the determined decoder as the video frame corresponding to time ti+4 for rendering and display.

[0324] As another example, when the difference is large, since keyframes are directly obtained, to avoid the decoder decoding non-keyframes (which will not be used later but will waste resources), the rendering module can instruct the decoder to decode only the keyframes in a set of video frames when the difference is large. When the difference is small, the decoder is instructed to decode the video frames in a set of video frames in the decoding order. This process can be described in detail in [reference needed]. Figure 10 and Figure 11 The illustrated embodiment.

[0325] The process executed by the electronic device is similar to that in the above embodiments, with the following differences:

[0326] S601, the rendering module received the time ti+4.

[0327] S602, the rendering module calculates the difference between the currently received ti+4 and the previously received ti+3, and determines that the difference is greater than 500ms (an example of the second duration).

[0328] 500ms is just an example; in practical applications, it can be 400ms, 600ms, etc. This value can be determined based on the decoder's decoding speed and empirical values ​​to achieve a balance between decoding capability and video smoothness. The second duration is used to determine whether to decode only keyframes.

[0329] S603, if the rendering module determines that the difference is greater than 500ms, it sends an instruction to decoder 1 to instruct decoder 1 to decode only the I-frames in the video frame group.

[0330] S604, if the rendering module determines that the difference is greater than 500ms, it sends an instruction to decoder 2 to instruct decoder 2 to decode only the I-frames in the video frame group.

[0331] S605, the rendering module determines that the decoder of the video frame group k corresponding to the video frame ti+4 is decoder one.

[0332] S606, the rendering module retrieves the video frame with the smallest difference between the timestamp and t i+4 from the cache space where decoder 1 is located, and uses it as the video frame corresponding to time t i+4.

[0333] When the decoder only decodes I-frames in a group of video frames, the video frame with the smallest difference between the timestamp obtained from the buffer space and t i+4 is likely to be a keyframe (I-frame).

[0334] S607, the rendering module renders the decoded video frame at time t i+4 through the rendering thread to obtain the image at time t i+4.

[0335] Subsequently, the rendering module sends the image at time t i+4 to the gallery application for display.

[0336] Of course, in practical applications, there is no strict order of execution for processes S602 to S604 and processes S605 to S607. Processes S602 to S604 can be executed first, followed by processes S605 to S607; or processes S605 to S607 can be executed first, followed by processes S602 to S604; or the two processes can be executed simultaneously through two independent threads.

[0337] and Figure 10 In the embodiment shown, when the user's dragging operation is not fast enough, it is also necessary to restore the normal decoding logic (decode the video frames in the video frame group according to the decoding order).

[0338] S701, the rendering module received the time ti+5.

[0339] S702, the rendering module calculates the difference between the currently received ti+5 and the previously received ti+4, and determines that the difference is no greater than 500ms.

[0340] S703, if the rendering module determines that the difference is no more than 500ms, it sends a command to decoder 1 to instruct decoder 1 to cancel decoding only I-frames.

[0341] S704, if the rendering module determines that the difference is no more than 500ms, it sends a command to decoder 2 to instruct decoder 2 to cancel decoding only I-frames.

[0342] S705, the rendering module determines that the decoder of video frame group k+1, where the video frame corresponding to ti+5 is located, is decoder two.

[0343] S706, the rendering module obtains the video frame with the smallest difference between the timestamp and t i+5 from the cache space where decoder 2 is located, and uses it as the video frame corresponding to time t i+5.

[0344] S707, the rendering module renders the decoded video frame at time t i+5 through the rendering thread to obtain the image at time t i+5.

[0345] Subsequently, the rendering module sends the image at time t i+5 to the gallery application for display.

[0346] Similarly, in practical applications, there is no strict order of execution for processes S702 to S704 and S705 to S707. Processes S702 to S704 can be executed first, followed by processes S705 to S707; or processes S705 to S707 can be executed first, followed by processes S702 to S704; or the two processes can be executed simultaneously through two independent threads.

[0347] certainly, Figure 10 and Figure 11 In the illustrated embodiment, it is also necessary to notify another decoder to decode the video frames in the next video frame group, but this is not shown in the illustration.

[0348] Additionally, in some extreme scenarios, such as when dragging is very fast, during the process of displaying the video frame corresponding to the time ti+m (which belongs to the video frame group j+n), the video frame group corresponding to the playback progress (playback time) of the next touch point is likely not an adjacent video frame group of video frame group j+n. Therefore, the decoder may not be able to decode in advance. However, since only keyframes are decoded, the decoding time before rendering is greatly reduced, thereby reducing screen stuttering and improving screen smoothness.

[0349] In the above embodiments, the two moments at which the positions of any two consecutive touch points are determined can be used as examples of the first moment and the second moment; the position of the first control, the determined video frame group, the displayed video frame, etc., related to the corresponding first moment (or second moment) can all be determined based on the inherent logic corresponding to the above video playback process.

[0350] In the above embodiments, the forward playback scenario is used as an example. The process of reverse playback is similar to that of forward playback, except that when determining the next video frame group, the previous video frame group is determined. For a detailed explanation of the next and previous video frame groups, please refer to the descriptions in the above embodiments. In practical applications, there may also be scenarios where playback alternates between forward and reverse.

[0351] In the above embodiments, the first buffer space can cache only the video frames of one video frame group. When the decoder decodes video frames in a new video frame group, the video frames of the new video frame group overwrite the video frames of the old video frame group. Similarly, the second buffer space can cache only the video frames of one video frame group. When the decoder decodes video frames in a new video frame group, the video frames of the new video frame group overwrite the video frames of the old video frame group.

[0352] As another embodiment of this application, the video playback method includes:

[0353] Display a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video;

[0354] A swipe operation is received on the first interface;

[0355] In response to the sliding operation, the first control is moved from the first position along the first trajectory;

[0356] During the process of moving the first control from the first position along the first trajectory:

[0357] When the first control is moved from the first position to the second position on the first trajectory, the video frame in the first frame group is decoded by the first decoder, the video frame in the second frame group is decoded by the second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress represented by the first control when it is in the second position is within the progress range represented by the first frame group;

[0358] When the first control is moved from the second position to the third position on the first trajectory, the video frame in the third frame group is decoded by the first decoder, and the decoded third video frame in the second frame group is displayed in the first area. The playback progress indicated by the first control at the third position is within the progress range indicated by the second frame group. The first frame group, the second frame group, and the third frame group are three groups of video frames that are consecutive in time in the first video.

[0359] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0360] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, can implement the steps in the above-described method embodiments.

[0361] This application also provides a computer program product that, when run on an electronic device or a wireless router, enables the electronic device to perform the steps described in the various method embodiments above.

[0362] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0363] This application also provides a chip, which includes a processor coupled to a memory. The processor calls a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip can be a single chip or a chip module composed of multiple chips.

[0364] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0365] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0366] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video playback method, characterized in that, Applied to electronic devices, the method includes: Display a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video; A swipe operation is received on the first interface; In response to the sliding operation, the first control is moved from the first position along the first trajectory; During the process of moving the first control from the first position along the first trajectory: When the first control is moved from the first position to the second position on the first trajectory, the video frame in the first frame group is decoded by the first decoder, the video frame in the second frame group is decoded by the second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress represented by the first control when it is in the second position is within the progress range represented by the first frame group; When the first control is moved from the second position to the third position on the first trajectory, the video frame in the third frame group is decoded by the first decoder, and the decoded third video frame in the second frame group is displayed in the first area. The playback progress indicated by the first control at the third position is within the progress range indicated by the second frame group. The first frame group, the second frame group, and the third frame group are three time-continuous video frame groups in the first video. Wherein, the playback progress corresponding to the first control when it is in the second position is the first moment, and the playback progress corresponding to the first control when it is in the third position is the second moment; If the absolute value of the first difference between the first time point and the second time point is greater than the second duration, the sliding speed of the sliding operation is the first sliding speed, and both the first decoder and the second decoder are set not to decode non-key frames in the video frame group; If the absolute value of the first difference between the first time point and the second time point is not greater than the second duration, the sliding speed of the sliding operation is the second sliding speed. The first decoder and the second decoder are set to decode the video frames in the video frame group in the decoding order, and the first sliding speed is greater than the second sliding speed.

2. The method as described in claim 1, characterized in that, During the process of moving the first control from the first position along the first trajectory, the method further includes: When the first control is moved from the third position to the fourth position on the first trajectory, the video frame in the fourth frame group is decoded by the second decoder, and the decoded fourth video frame in the third frame group is displayed in the first area. The playback progress indicated by the first control at the fourth position is within the progress range indicated by the third frame group. The first frame group, the second frame group, the third frame group, and the fourth frame group are four video frame groups that are consecutive in time.

3. The method as described in claim 1 or 2, characterized in that, Before decoding the video frames in the first frame group by the first decoder, the method further includes: The playback progress of the first video is determined as the first moment based on the position of the first touch point of the sliding operation; Analyze the first video and determine the video frame group containing the video frame corresponding to the first moment as the first frame group; Before decoding the video frames in the second frame group using the second decoder, the method further includes: The playback mode of the video frame corresponding to the first moment is determined to be forward playback; If the video frame at the first moment is played in forward mode, the next video frame group in the first video that is located in the first frame group is determined to be the second frame group.

4. The method as described in claim 3, characterized in that, After determining the playback progress of the first video as the first moment based on the first touch point position of the sliding operation, the method further includes: The position of the first control at the first moment is determined to be the second position when the playback progress of the first video is determined to be at the first moment. Move the first control to the second position along the first trajectory.

5. The method as described in claim 3, characterized in that, Before decoding the video frames in the third frame group using the first decoder, the method further includes: The playback progress of the first video is determined as the second moment based on the position of the second touch point of the sliding operation; Analyze the first video to determine the video frame group containing the video frame corresponding to the second moment as the second frame group; It has been determined that the second frame group has been assigned to the second decoder for decoding; The playback mode of the video frame corresponding to the second moment is determined to be forward playback; If the second frame group has been assigned to the second decoder for decoding, and the video frame corresponding to the second moment is played in forward mode, then the next video frame group in the first video located in the second frame group is determined to be the third frame group.

6. The method as described in claim 5, characterized in that, After determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method further includes: The position of the first control at the second moment when the playback progress of the first video is determined is the third position; Move the first control along the first trajectory to the third position.

7. The method as described in claim 5 or 6, characterized in that, The step of determining that the playback mode of the video frame corresponding to the second moment is forward playback includes: It is determined that the second time point is greater than the first time point; Accordingly, if it is determined that the second time is less than the first time, the playback mode of the video frame corresponding to the second time is determined to be reverse playback.

8. The method as described in claim 5 or 6, characterized in that, After determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method further includes: Calculate the absolute value of the first difference between the second time point and the first time point; The absolute value of the first difference is determined to be greater than the first duration.

9. The method as described in claim 8, characterized in that, After calculating the absolute value of the first difference between the second time point and the first time point, the method further includes: If the absolute value of the first difference is not greater than the first duration, then the video frame group in which the second moment is located is no longer determined, nor is the next video frame group in which the second moment is located determined.

10. The method as described in claim 5 or 6, characterized in that, After determining that the video frame group corresponding to the first time moment belongs to the first frame group, the method further includes: Record the timestamp of the first video frame and the timestamp of the last video frame in the first frame group; After determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, and before parsing the first video and determining that the video frame group corresponding to the second moment belongs to the second frame group, the method further includes: It is determined that the second moment is not within the time range of the timestamps of the first and last video frames in the first frame group.

11. The method as described in claim 10, characterized in that, After determining the playback progress of the first video as the second moment based on the second touch point position of the sliding operation, the method includes: If the second time point falls within the time stamp range of the first video frame and the last video frame in the first frame group, then the decoded video frame corresponding to the second time point is obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second time point is displayed in the first area.

12. The method as described in claim 3, characterized in that, Decoding the video frames in the first frame group using the first decoder includes: The first decoder decodes the multiple video frames in the first frame group according to the decoding order of the multiple video frames in the first frame group; The first decoder decodes one video frame from the first frame group and stores the decoded video frame in the first buffer space. The video frames corresponding to the first moment include: In the first cache space, video frames whose timestamp differs from the first moment by less than or equal to a first value, where the first value is half the difference between the timestamps of two adjacent video frames; Alternatively, the video frame with the smallest difference between its timestamp and the first moment among the decoded video frames already cached in the first cache space; Alternatively, keyframes already cached in the first cache space.

13. The method as described in claim 12, characterized in that, Before displaying the decoded second video frame from the first frame group in the first region, the method further includes: The decoder that decodes the first frame group is identified as the first decoder; The video frame corresponding to the first time moment is obtained from the first decoder, and the video frame corresponding to the first time moment is the second video frame.

14. The method as described in claim 13, characterized in that, The step of determining the decoder that decodes the first frame group as the first decoder includes: Calculate the second difference between the timestamp of the first video frame cached in the first buffer space corresponding to the first time step and the first time step at the first time step; When the second difference is within the first time range, the decoder that decodes the first frame group is determined to be the first decoder.

15. The method as described in claim 14, characterized in that, After calculating the second difference between the timestamp of the first video frame cached in the first buffer space corresponding to the first time point and the first decoder, the method further includes: If the second difference is not within the first time range, then the third difference between the timestamp of the first video frame cached in the second buffer space corresponding to the first time and the second decoder is calculated to determine whether the decoder that decodes the first frame group is the second decoder.

16. An electronic device, characterized in that, The device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when executed by the one or more processors, causes the electronic device to perform the method as described in any one of claims 1-15.

17. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-15.

18. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-15.

Images