Video playing method, electronic equipment, chip system and storage medium

Abstract

The invention provides a video playing method, electronic equipment, a chip system and a storage medium, and relates to the technical field of terminal equipment, in a frame playing scene, when a sliding operation triggers the electronic equipment to display a video frame at a certain moment, a video frame in a video frame group (a first frame group) where the video frame at the moment is located is decoded through a first decoder, and the video frame is played. Decoding, by a second decoder, a video frame in a next video frame group (second frame group) of the first frame group; when the electronic equipment is triggered to display a certain video frame in the second frame group due to the dragging operation, the decoded video frame can be directly obtained from the cache space corresponding to the second decoder to be rendered and displayed due to the fact that the decoder decodes and caches the video frame in advance, and certainly, the decoding efficiency is greatly improved. And meanwhile, the video frames in the next video frame group (the third frame group) of the second frame group need to be decoded through the first decoder, so that the video frames in the video frame group are always decoded in advance, and the fluency of the picture in the frame playing scene is improved.

Description

Technical Field

[0001] The present application relates to the technical field of terminal devices, and in particular to a video playback method, electronic device, chip system and storage medium. Background Art

[0002] Some applications installed in electronic devices (for example, gallery applications, video editing applications, etc.) can provide video playback and video editing functions. When playing a video through the video playback function, the video playback progress can be displayed, for example, the video playback progress can be displayed through a progress bar, a progress ring, a progress display frame, etc.; when editing a video through the video editing function, the screen at different positions of the video can also be displayed through a progress bar, a progress ring, a progress display frame, etc.

[0003] Whether it is a video playback scenario or a video editing scenario, the user can trigger the electronic device to play the video frame by frame by dragging on the progress bar, progress ring or progress display frame; however, in the video frame playback scenario, the display process of the video screen is relatively jerky and not smooth. Summary of the invention

[0004] The present application provides a video playback method, an electronic device, a chip system and a storage medium, which can reduce the freeze of the video picture in the frame playback scene of the video and improve the smoothness of the video picture in the frame playback scene.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application adopts the following technical solutions:

[0006] A first aspect of the present application provides a video playback method, applied to an electronic device, the method comprising:

[0007] Displaying a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video;

[0008] Receiving a sliding operation on the first interface;

[0009] In response to the sliding operation, moving the first control from a first position along a first track;

[0010] During the process of moving the first control from the first position along the first track:

[0011] When the first control is moved from the first position to a second position on the first track, a video frame in a first frame group is decoded by a first decoder, a video frame in a second frame group is decoded by a second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress indicated by the first control at the second position is within the progress interval indicated by the first frame group;

[0012] When the first control is moved from the second position to the third position on the first trajectory, the video frames in the third frame group are decoded by the first decoder, and the decoded third video frames in the second frame group are displayed in the first area, wherein the playback progress represented by the first control at the third position is within the progress interval represented by the second frame group, and the first frame group, the second frame group and the third frame group are three temporally consecutive video frame groups in the first video.

[0013] In the present application, when a sliding operation triggers an electronic device to display a video frame at a certain moment, the video frame in the video frame group (first frame group) to which the video frame at that moment belongs is decoded by the first decoder, and the video frame in the next video frame group (second frame group) of the first frame group is decoded by the second decoder; when the electronic device is triggered to display a video frame in the second frame group due to a drag operation, since the decoder has been decoded and cached in advance, the decoded video frame can be directly obtained from the cache space corresponding to the second decoder for rendering and display, which improves the speed of displaying the picture, reduces the freeze, and improves the smoothness of the picture. Of course, it is also necessary to decode the video frames in the next video frame group (third frame group) of the second frame group through the first decoder, so that the video frames in the video frame group are always decoded in advance, thereby improving the smoothness of the picture in the frame playback scenario.

[0014] As another implementation of the first aspect, in a process of moving the first control from the first position along the first trajectory, the method further includes:

[0015] In moving the first control from the third position to a fourth position on the first track ( Figure 4 4), the video frames in the fourth frame group are decoded by the second decoder, and the decoded fourth video frames in the third frame group are displayed in the first area, wherein the playback progress represented by the first control at the fourth position (position 4) is within the progress interval represented by the third frame group, and the first frame group, the second frame group, the third frame group and the fourth frame group are four temporally continuous video frame groups.

[0016] In the present application, as the first control continues to move with the sliding operation, the decoders are also used alternately to decode the video frames in the next video frame group, so that there is always a decoder to decode the video frames in advance. When it is really necessary to display the subsequent video frames, the decoded video frames can be directly obtained from the decoder for rendering and display, thereby improving the smoothness of the picture.

[0017] As another implementation of the first aspect, before decoding the video frames in the first frame group by the first decoder, the method further includes:

[0018] Determining the playback progress of the first video as a first moment according to the first contact point position of the sliding operation;

[0019] Parsing the first video, determining that a video frame group including a video frame corresponding to the first moment is the first frame group;

[0020] Before decoding the video frames in the second frame group by the second decoder, the method further includes:

[0021] Determining that the playback mode of the video frame corresponding to the first moment is forward playback;

[0022] When the playback mode of the video frame corresponding to the first moment is forward playback, the next video frame group located in the first frame group in the first video is determined to be the second frame group.

[0023] In the present application, the playback progress of the video can be determined by the real-time contact position of the sliding operation, and then the video frame group where the video frame corresponding to the playback progress is located is found. If the video frame group where the video frame corresponding to the playback progress is located has not been decoded, it is decoded by a decoder; accordingly, it can also be determined whether the frame playback process is forward playback (the playback time determined by the current contact position is greater than the playback time determined by the previous contact position) or reverse playback (the playback time determined by the current contact position is less than the playback time determined by the previous contact position). In the case of forward playback, the next video frame group is decoded by another decoder for early decoding.

[0024] In the case of forward playback, the next video frame group is a video frame group that meets the following conditions: adjacent to the current video frame group and with a timestamp greater than the timestamp of the current video frame group. The timestamp of the video frame group may be the timestamp of any video frame in the video frame group.

[0025] In the case of reverse playback, a previous video frame group is decoded by another decoder for early decoding, and the previous video frame group is a video frame group that meets the following conditions: adjacent to the current video frame group and with a timestamp smaller than the timestamp of the current video frame group.

[0026] As another implementation of the first aspect, after determining that the playback progress of the first video is a first moment according to the first contact point position of the sliding operation, the method further includes:

[0027] Determine that the position of the first control when the playback progress of the first video is the first moment is the second position;

[0028] The first control is moved along the first trajectory to the second position.

[0029] In the present application, the position of the first control on the first track is related to the playback progress, and from the user's perspective, the controllability is good and intuitive.

[0030] As another implementation of the first aspect, before decoding the video frames in the third frame group by the first decoder, the method further includes:

[0031] Determining the playback progress of the first video as a second moment according to the position of the second contact point of the sliding operation;

[0032] Parsing the first video, determining that a video frame group including a video frame corresponding to the second moment is the second frame group;

[0033] Determining that the second frame group has been assigned to the second decoder for decoding;

[0034] Determining that the playback mode of the video frame corresponding to the second moment is forward playback;

[0035] When the second frame group has been assigned to the second decoder for decoding and the playback mode of the video frame corresponding to the second moment is forward playback, the next song video frame group in the first video located in the second frame group is determined to be the third frame group.

[0036] In the present application, when a decoder has been assigned to the video frame group to which the video frame corresponding to the current playback progress belongs, the video frame in the next or previous video frame group can be decoded by another decoder according to the playback mode, so that the next or previous video frame group that may be displayed can be decoded in advance as the user slides.

[0037] As another implementation of the first aspect, after determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes:

[0038] Determine that the position of the first control when the playback progress of the first video is the second moment is the third position;

[0039] The first control is moved along the first trajectory to the third position.

[0040] As another implementation of the first aspect, determining that the playback mode of the video frame corresponding to the second moment is forward playback includes:

[0041] Determining that the second moment is greater than the first moment;

[0042] Correspondingly, when it is determined that the second moment is less than the first moment, it is determined that the playback mode of the video frame corresponding to the second moment is reverse playback.

[0043] In the present application, the playback mode of the video frame corresponding to the current moment can be determined according to the relationship between the playback moments corresponding to two consecutive playback progresses, and the determination process is simple and quick.

[0044] As another implementation of the first aspect, after determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes:

[0045] Calculate an absolute value of a first difference between the second moment and the first moment;

[0046] It is determined that the absolute value of the first difference is greater than the first duration.

[0047] As another implementation of the first aspect, after calculating the absolute value of the first difference between the second moment and the first moment, the method further includes:

[0048] If the absolute value of the first difference is not greater than the first duration, the video frame group where the second moment is located is no longer determined, and the video frame group next to the video frame group where the second moment is located is no longer determined.

[0049] In this application, since the human eye's perception of frame rate is not that high, for video frames corresponding to two particularly close playback progresses (the time difference is not greater than the first duration), the video frame corresponding to the latter playback progress may not be played. For video frames corresponding to two not particularly close playback progresses (the time difference is greater than the first duration), the video frame corresponding to the latter playback progress needs to be played. The first duration is related to the human eye's perception of frame rate. For example, it can be set to 16.7ms, 15ms, 18ms, 25ms, etc. Therefore, the first duration is used to filter video frames so that the frame rate is within a certain range.

[0050] As another implementation of the first aspect, after determining that the video frame group to which the video frame corresponding to the first moment belongs is the first frame group, the method further includes:

[0051] Recording the timestamp of the first video frame and the timestamp of the last video frame in the first frame group;

[0052] After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, and before parsing the first video and determining that the video frame group to which the video frame corresponding to the second moment belongs is the second frame group, the method further includes:

[0053] It is determined that the second moment is not within a time range between a timestamp of a first video frame and a timestamp of a last video frame in the first frame group.

[0054] As another implementation of the first aspect, after determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method includes:

[0055] If the second moment is within the range of the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, the decoded video frame corresponding to the second moment is obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second moment is displayed in the first area.

[0056] In the present application, if the second moment is within the range of the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, it means that the video frame corresponding to the second moment is in the first frame group. Since the first frame group has been assigned a decoder for decoding when the video frame corresponding to the first moment is to be displayed, there is no need to assign a decoder to decode the first frame group again. Instead, the decoded video frame corresponding to the second moment can be directly obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second moment can be displayed in the first area; of course, if the second moment is not within the range of the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, it means that the video frame corresponding to the second moment is not a video frame in the first frame group, and it is necessary to determine the video frame group in which the video frame corresponding to the second moment is located, so as to obtain the decoded video frame from the decoder corresponding to the determined video frame group.

[0057] As another implementation of the first aspect, after determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes:

[0058] Calculate an absolute value of a first difference between the second moment and the first moment;

[0059] When the absolute value of the first difference is greater than the second duration, setting the first decoder and the second decoder not to decode non-key frames in the video frame group;

[0060] When the absolute value of the first difference is not greater than the second duration, the first decoder and the second decoder are set to decode the video frames in the video frame group according to a decoding order.

[0061] The second duration is greater than the first duration. When the absolute value of the first difference is greater than the second duration (for example, 500ms, 400ms, 600ms, 700ms, etc.), it means that the user's sliding operation is faster. In this case, the user does not need to perceive the rapid switching process of the screen. The decoder can only decode the key frames in a video frame group, so that the decoder can quickly complete the decoding of the current video frame group. Of course, when displaying video frames, the key frames in a video frame group can also be displayed, thereby improving the decoding speed and picture smoothness.

[0062] As another implementation of the first aspect, decoding the video frames in the first frame group by the first decoder includes:

[0063] decoding, by the first decoder, a plurality of video frames in the first frame group in a decoding order of the plurality of video frames in the first frame group;

[0064] Decoding one video frame in the first frame group by the first decoder, and storing the decoded video frame in a first cache space;

[0065] The video frame corresponding to the first moment includes:

[0066] a video frame in the first cache space whose timestamp difference with the first moment is less than or equal to a first value, wherein the first value is half of the timestamp difference between two adjacent video frames;

[0067] Or, a video frame having a minimum difference between a timestamp and the first moment among the decoded video frames cached in the first cache space;

[0068] Alternatively, the key frames have been cached in the first cache space.

[0069] In a specific implementation, different video frames can be set to be displayed according to the speed of the user's sliding operation. For example, when the user's sliding operation is slow, the decoder has enough time to decode each video frame in a video frame group, and can obtain a video frame whose difference between a timestamp and the first moment is less than or equal to a first value from the cache space; when the sliding operation is fast, the decoder does not have enough time to decode each video frame in a video frame group, and can obtain a video frame whose difference between a timestamp and the first moment among the decoded video frames is the smallest from the cache space; when the sliding operation is very fast, the first decoded key frame (I frame) in the decoding order can be obtained from the cache space. Correspondingly, non-key frames include p frames and B frames. Of course, if a group of video frames does not have a B frame, the non-key frame is a P frame.

[0070] As another implementation of the first aspect, before the first area displays the decoded second video frame in the first frame group, the method further includes:

[0071] Determining that a decoder for decoding the first frame group is a first decoder;

[0072] A video frame corresponding to the first moment is obtained from the first decoder, where the video frame corresponding to the first moment is the second video frame.

[0073] As another implementation of the first aspect, determining that a decoder for decoding the first frame group is a first decoder includes:

[0074] Calculate a second difference between the first moment and the timestamp of the first video frame cached in the first cache space corresponding to the first decoder;

[0075] When the second difference is within a first time range, it is determined that the decoder for decoding the first frame group is a first decoder.

[0076] As another implementation of the first aspect, after calculating the second difference between the first moment and the timestamp of the first video frame cached in the first cache space corresponding to the first decoder, the method further includes:

[0077] If the second difference is not within the first time range, the third difference between the first time point and the timestamp of the first video frame cached in the second cache space corresponding to the second decoder is calculated to determine whether the decoder decoding the first frame group is the second decoder.

[0078] In the present application, since decoding is performed through at least two decoders, the decoder for decoding the video frame group corresponding to the playback progress can be determined first through the relationship between the timestamp of the first video frame in each group of video frames and the determined playback progress (moment), and then the specific video frame can be obtained from the cache space of the decoder.

[0079] In a second aspect, an electronic device is provided, comprising a processor, wherein the processor is used to call a computer program stored in a memory to implement any method of the first aspect of the present application.

[0080] In a third aspect, a chip is provided, including a processor, wherein the processor is coupled to a memory, and the processor executes a computer program stored in the memory so that the electronic device implements any method of the first aspect of the present application.

[0081] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer instructions are executed on an electronic device, the electronic device implements any one of the methods in the first aspect of the present application.

[0082] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a device, enables the electronic device to execute any one of the methods in the first aspect of the present application.

[0083] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of an interface of a video playback process provided in an embodiment of the present application;

[0086] Figure 3 An example of a group of video frames provided in an embodiment of the present application and a schematic diagram of the playback order and decoding order of the group of video frames;

[0087] Figure 4 A schematic diagram of a flow chart of a video playback method provided in an embodiment of the present application;

[0088] Figure 5 A timing diagram of the preparation work before frame playback provided in an embodiment of the present application;

[0089] Figure 6 A timing diagram of the video frame corresponding to position 1 during the video playback process provided in the embodiment of the present application;

[0090] Figure 7 A timing diagram of the video frame corresponding to position 2 during the video playback process provided in an embodiment of the present application;

[0091] Figure 8 A schematic diagram of a process of performing frame rate filtering during video playback provided in an embodiment of the present application;

[0092] Fig. 9 A timing diagram of the video frame corresponding to position 3 during the video playback process provided in an embodiment of the present application;

[0093] Fig.10 A timing diagram of a video frame corresponding to a display position k during a video playback process provided in an embodiment of the present application;

[0094] Fig.11 A timing diagram of the video frame corresponding to the display position k+1 during the video playback process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0096] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0097] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0098] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0099] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0100] The embodiment of the present application provides a video playback method, which can be applied to an electronic device, such as a tablet computer, a mobile phone, a wearable device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0101] Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 1211, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0102] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0103] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the frame playback method of the video in the embodiment of the present application.

[0104] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0105] The internal memory 1211 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 1211. The internal memory 1211 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.). Touch sensor 180K, also known as a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided by the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194. For example, the click operation of the control provided in the embodiment of the present application, the drag operation of the progress display frame, etc. can all be recognized by the touch sensor.

[0106] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. For example, the process of rendering YUV data in the embodiment of the present application can be implemented by a GPU.

[0107] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. For example, in the embodiments of the present application, Figure 2 or Figure 3 The interfaces shown are all displayed by the monitor.

[0108] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats. For example, the video codec can implement the decoding of the H.264 format video frames provided in the embodiments of the present application, or re-encode them into H.264 format video frames with all key frames.

[0109] The embodiment of the present application does not particularly limit the specific structure of the execution subject of a video playback method, as long as the code of a video playback method of the embodiment of the present application can be run to communicate according to a video playback method provided by the embodiment of the present application. For example, the execution subject of a video playback method provided by the embodiment of the present application can be a functional module in an electronic device that can call and execute a program, or a communication device applied to an electronic device, such as a chip.

[0110] The user can play various videos stored in the electronic device through applications in the electronic device (for example, gallery applications, video editing applications, etc.), and edit various videos stored in the electronic device. Regardless of the video playback scene or the video editing scene, the user can trigger the electronic device to play the video frame by dragging the progress bar, progress ring or progress display frame (which can also be understood as a sliding operation). Of course, in actual applications, the sliding operation on the interface displaying the progress bar, progress ring or progress display frame may also trigger the electronic device to play the video frame.

[0111] The following takes the scenario of playing videos in the gallery application as an example.

[0112] Reference Figure 2 , is a schematic diagram of an interface for playing videos stored in an electronic device through a gallery application provided in an embodiment of the present application.

[0113] Reference Figure 2 (a) in the figure is a grid diagram of pictures and videos stored in the electronic device displayed by the gallery application of the electronic device; the grid diagram can display pictures and videos, wherein the grid where the video is located displays the cover of the video and the length of the video, and the grid where the picture is located displays the thumbnail of the picture.

[0114] The grid image can be an interface image displayed by the gallery application after the user clicks the gallery application icon on the system desktop to open the gallery application, or it can be an interface image displayed by the gallery application triggered by more steps after the user clicks the gallery application icon on the system desktop to open the gallery application.

[0115] When the grid diagram displays pictures and / or videos stored in the electronic device, it can be presented in a variety of ways, for example, Figure 2 The method shown in (a) is to display the pictures and videos in the photo group in reverse chronological order; in actual applications, the gallery application can also set multiple object groups, for example, through Figure 2 The control 10 shown in (a) can view multiple groups in the album: image group, video group, screenshot and recording group, etc. The user can also display pictures through the image group in the gallery application, display videos through the video group in the gallery application, and display screenshots and screen recording videos through the screenshot and recording group in the gallery application. Each group can also display the corresponding pictures and / or videos in the form of the grid diagram displayed above.

[0116] The embodiment of the present application does not limit the number of steps of user operation required to display the grid diagram of the gallery application from the system desktop, and similarly, does not limit the way in which the grid diagram displays pictures and videos.

[0117] Reference Figure 2 (b) in the figure indicates that the user clicks Figure 2 After the cover of video A in the grid diagram shown in (a) is displayed, the electronic device displays a video playback interface of video A in response to the operation (the interface can be used as an example of the first interface). The video playback interface includes a video playback area 11 (the area can be used as an example of the first area) and a progress display area 12.

[0118] Among them, the video playback area 11 is used to display the video screen during the video playback process. After entering the video playback interface from the grid interface, the control 111 and the video cover (which can be used as an example of the first video frame) are displayed in the video playback area. The user clicks the control 111, and the electronic device responds to the click operation on the control 111, and the video playback area 11 starts playing the video screen from the 0th second; when the video is paused, the control 111 is also displayed in the video playback area 11. The user can click the control 111, and the electronic device responds to the click operation on the control 111, and the video playback area 11 starts playing the video screen from the current pause moment.

[0119] The progress display area 12 is used to display the video playback progress. The progress display area 12 includes a progress display frame 121 (which can be used as an example of the first control) and a progress indicator line 122. The position of the progress indicator line 122 can remain unchanged. The user can drag the progress display frame 121 located below the progress indicator line 122 so that the progress display frame 121 moves left and right below the progress indicator line without moving the progress indicator line 122. The position of the progress indicator line 122 above the progress display frame on the progress display frame indicates the playback progress of the current video.

[0120] In this example, the progress display frame and the progress indicator line are used as examples to display the video playback progress, and the progress display frame can be used as an example of the first control. In actual applications, other progress display methods can also be used, such as a progress bar and a progress ring. If the progress bar displays the video playback progress, the first control is an indicator on the progress bar that is used to display the playback progress and can be moved on the progress bar.

[0121] As an example, the total playback time of the current video is 15 seconds, and the total length of the progress display frame of the current video is the length represented by L pixels. Based on these two parameters, we can get: the playback time corresponding to the unit length (or the length corresponding to the unit time). For example, divide the playback time 15 by the total length L. Therefore, if the leftmost edge position of the progress display frame (the position corresponding to the 1st pixel) is set to correspond to the 0th second of the video, then different positions of the progress indicator line on the progress display frame correspond to different playback times. For example, the distance between the position of the progress indicator line on the progress display frame and the leftmost edge position of the set progress display frame is X, then the playback time corresponding to the progress indicator line is: X*15 / L.

[0122] For the convenience of subsequent description, the embodiment of the present application is described in a manner corresponding to "the moment corresponding to the progress indicator line". Of course, it can also be the playback progress represented by the position of the first control or the playback progress of the video represented when the first control is at a certain position.

[0123] Due to the space limitation of the progress display area 12, a portion of the video frame or a portion of the video frame in the progress display frame 121 may be displayed in the progress display area 12. Figure 2 As shown in (b) in FIG. 1 , the progress display frame 121 displayed in the progress display area 12 includes: thumbnails of 3 complete video frames and a partial area of ​​a thumbnail of a video frame.

[0124] Typically, the picture played in the video playing area 11 is a picture of a video frame corresponding to the moment corresponding to the progress indication line 122 in the progress display area 12 .

[0125] In a specific implementation, there is a situation that the time corresponding to the progress indicator line 122 is the time between the playback times corresponding to two adjacent video frames, that is, there is no video frame corresponding to the time of the progress indicator line 122 in the video file.

[0126] As an example, among the video frames arranged in ascending order according to the playback time (recorded as timestamp), the playback time of the i-th video frame is ti, and the playback time of the i+1-th video frame is t(i+1); the time t corresponding to the progress indicator line 122 is greater than ti and less than t(i+1). In this case, the video frame corresponding to the timestamp with the shortest time interval with the time t corresponding to the progress indicator line 122 can be used as the video frame at the time corresponding to the progress indicator line 122. Therefore, it can also be understood that the video frame corresponding to a certain time is the one whose absolute value of the difference between the timestamp and the time is less than or equal to the first value (half of the difference between the timestamps of two adjacent video frames in the video).

[0127] Of course, in actual applications, if there is no video frame in the video file with the same timestamp as the time corresponding to the progress indicator line, the previous video frame (the video frame corresponding to ti) of the time t corresponding to the progress indicator line 122 can also be used as the video frame at the time corresponding to the progress indicator line 122; or the next video frame (the video frame corresponding to tj) of the time t corresponding to the progress indicator line 122 can be used as the video frame at the time corresponding to the progress indicator line 122. In actual applications, there can be many ways to display the playback progress in the progress display area 12. For example, the video playback progress can be displayed in the form of a progress bar, or in the form of a progress ring. The present application provides two playback modes for video playback: ordinary playback and frame playback.

[0128] The embodiments of this application are only Figure 2 The progress display area shown in (b) (in actual application, it can also be other forms of progress display areas) is used as an example to describe the difference between normal playback and frame playback.

[0129] Reference Figure 2 (c) in the figure indicates that the user clicks Figure 2After the control 111 (the control can also be referred to as a play control) in the interface shown in (b) of FIG. 1 is pressed, an interface diagram of the electronic device playing video A in response to the operation is displayed. The interface diagram indicates that the video A has been played to the 5th second, and the video screen corresponding to the 5th second is displayed in the video playback area 11; at the same time, the progress display frame 121 of the progress display area 12 moves so that the time corresponding to the progress indicator line 122 is the 5th second. Figure 2 (b) to Figure 2 The playback process shown in (c) is the normal playback process of the video.

[0130] A video file includes multiple video frames, each of which has a timestamp. The timestamp records the playback time of the video frame during the playback process of the video file. As an example, according to the timestamps from small to large, the video frames in the video file are arranged as follows: video frame 0 (timestamp t0), video frame 1 (timestamp t1), video frame 2 (timestamp t2), video frame 3 (timestamp t3), video frame 4 (timestamp t4), video frame 5 (timestamp t5), video frame 6 (timestamp t6), video frame 7 (timestamp t7)... Normal playback is: according to the timestamps from small to large, each video frame is displayed in the video playback area in sequence according to the playback time corresponding to the timestamp.

[0131] As an example of a normal playback process, video frame 0 is displayed in video playback area 11 at time t0 when playback starts, video frame 1 is displayed in video playback area 11 at time t1 after playback starts, video frame 2 is displayed in video playback area 11 at time t2 after playback starts, video frame 3 is displayed in video playback area 11 at time t3 after playback starts, video frame 4 is displayed in video playback area 11 at time t4 after playback starts, video frame 5 is displayed in video playback area 11 at time t5 after playback starts, video frame 6 is displayed in video playback area 11 at time t6 after playback starts, video frame 7 is displayed in video playback area 11 at time t7 after playback starts….

[0132] Combination Figure 2 (b) to Figure 2 (c) in the figure shows the normal playback process of video A: starting from the video frame at the 0th second of video A, each video frame is displayed in sequence according to the playback time indicated by the timestamp; Figure 2 (c) is a schematic diagram of the interface showing the video frame at the 5th second.

[0133] Reference Figure 2 In (d), the user drags Figure 2 The progress display frame of the interface shown in (c).

[0134] In response to Figure 2In the drag operation shown in (d) (which can also be recorded as a sliding operation), the electronic device displays a screen corresponding to the drag operation in the video playback area.

[0135] As an example, see Figure 2 In (e), when the user drags the progress display frame so that the time corresponding to the progress indicator line is the 14th second, the video playback area displays the video frame corresponding to the 14th second. The video frame corresponding to the 14th second can be the video frame with a timestamp of the 14th second in video A; it can also be a video frame with a timestamp closest to the 14th second; it can also be a video frame with a timestamp closest to the 14th second before the 14th second; or a video frame with a timestamp closest to the 14th second after the 14th second.

[0136] It should be noted that in the subsequent embodiments, the specific meaning of the description of "the video frame corresponding to a certain moment" can refer to the detailed explanation of "the video frame corresponding to the 14th second". Figure 2 (d) to Figure 2 The playback process shown in (e) is the frame playback process of the video.

[0137] When the user drags the progress display frame 121 , a relative position is formed between the progress indication line 122 and the progress display frame 121 , and the position of the progress indication line 122 on the progress display frame 121 is related to the time corresponding to the progress indication line 122 .

[0138] In the specific implementation process, it is assumed that before the user moves in the area where the progress display frame 121 of the screen is located, the time indicated by the progress indicator line 122 is t1. The user moves left or right in the area where the progress display frame of the screen is located, and the electronic device can detect the user's real-time touch position at a certain time interval; the electronic device determines the user's moving direction and moving distance based on the continuous real-time touch positions. The electronic device determines the time change △t based on the user's moving distance; the calculation method for calculating the real-time time t2 is determined based on the user's moving direction. For example, if the moving direction is to the left, the starting time t1 is subtracted from the time change △t to obtain the real-time time t2; if the moving direction is to the right, the starting time t1 is added to the time change △t to obtain the real-time time t2. After obtaining the real-time time t2, the electronic device searches for the video frame corresponding to the real-time time t2, and displays the video frame corresponding to t2 in the video playback area.

[0139] On the other hand, the electronic device obtains the moving direction of the progress display frame according to the moving direction determined by the real-time contact position, and determines the moving distance of the progress display frame according to the moving distance of the real-time contact position; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2. As another example, the electronic device also determines the moving direction (if t2 is greater than t1, it moves to the left, and if t2 is less than t1, it moves to the right) and moving distance of the progress display frame according to the starting time t1 and the real-time time t2; the electronic device moves the progress display frame in the progress display area according to the moving direction and moving distance of the progress display frame, so that the time corresponding to the progress indicator line is t2.

[0140] It should be noted that the above process is only an example. In actual applications, there are multiple ways to determine the time corresponding to the progress indicator line 122, the position of the progress display frame 121 and the video frame displayed in the video playback area 11 when the user drags the progress display frame 121.

[0141] Based on the above frame playback process, it can be understood that the process of the user dragging the progress display frame is an uncertain process. It may move left or right, may be dragged faster or slower. Accordingly, the video frame picture displayed in the video playback area 11 is random.

[0142] As an example, before the user moves in the area where the progress display frame of the screen is located, the starting time indicated by the progress indicator line is t1; while the user is dragging the progress display frame, the electronic device detects the user's real-time touch position at certain time intervals, thereby obtaining real-time times corresponding to multiple consecutive time periods: t2, t3, t7, t4, and t3. In the video playback area, the video frames are played in the following order: starting from the video frame corresponding to time t1 as the starting video frame, the video frames corresponding to time t2, the video frames corresponding to time t3, the video frames corresponding to time t7, the video frames corresponding to time t4, and the video frames corresponding to time t3 are played in sequence.

[0143] It can be understood that the process of the electronic device playing the video in the video playback area triggered by the user clicking the control 111 is normal playback; the process of the electronic device triggering the user to drag the progress display frame to display the video frame at the moment corresponding to the progress indicator line in the video playback area is frame playback.

[0144] The various operations shown in the embodiments of the present application are examples. In actual applications, other operations may be set according to specific circumstances. For example, some single-click operations may also be double-click operations or other air gestures. Some drag operations may also be air gestures. Similarly, examples of some operations in subsequent embodiments may also be other operations.

[0145] In the above embodiment, whether it is ordinary playback or frame playback, before displaying the video frame in the video playback area, the video frame needs to be decoded, and the video frame decoding is usually performed in groups. A group of video frames constitutes a video frame group, and each group of video frames includes a key frame (I frame) and at least one forward reference frame (P frame); of course, in actual applications, each video frame can also include at least one bidirectional reference frame (B frame).

[0146] Reference Figure 3 , is a schematic diagram of the playback order and decoding order of a group of video frames provided in an embodiment of the present application. A group of video frames can also be recorded as a group of pictures (GOP) group or a video frame group.

[0147] Each group of video frames includes, in the order of playback: I-frame, B-frame, B-frame, P-frame, B-frame, B-frame and P-frame. Among them, I-frame as a key frame can be decoded to get a complete picture, and the decoding speed is slow; P-frame as a non-key frame records the changes relative to the previous frame (non-B-frame), and the picture of P-frame can be decoded with the information of the previous frame. Of course, the picture of B-frame can be decoded with the information of the previous P frame (which may also be an I-frame) and the information of the subsequent P frame. P-frame and B-frame can improve the smoothness of the picture while reducing the amount of data in the video file. Among them, P-frame and B-frame are both non-key frames.

[0148] Whether I-frame, P-frame or B-frame, they are all video frames. When decoding a video frame at a certain moment, it is not to directly decode the video frame at that moment, but to decode the video frame group in which the video frame at that moment belongs. When decoding the video frames in a video frame group, they are decoded one by one in the decoding order.

[0149] As an example, if a video frame at time t1 needs to be decoded, and the video frame at time t1 is the sixth video frame in the playback order of a group of video frames: a B frame; the decoding order is:

[0150] The first decoding: the playback order is the first video frame: I frame;

[0151] The second decoding: the playback order is the 4th video frame: P frame;

[0152] The third decoding: the playback order is the second video frame: B frame;

[0153] The fourth decoding: the playback order is the third video frame: B frame;

[0154] The fifth decoding: the video frame with the seventh playback order: P frame;

[0155] The sixth decoding: the playback order is the 5th video frame: B frame;

[0156] The seventh decoding: the playback order is the sixth video frame: B frame;

[0157] Therefore, when decoding the video frame at time t1, it is necessary to perform the seventh decoding to decode the video frame at time t1.

[0158] At present, in the common playback scenario, the video frames in each video frame group are parsed, decoded, rendered, and displayed in the playback order of the video frame groups. Since the common playback is played in the order of the video frame groups, the decoding cache is performed in advance; of course, when decoding the video frames in each video frame group, the video frames in each video frame group are displayed in the order of the video frame groups. Figure 3 The decoding order shown decodes each video frame in the video frame group in turn. Therefore, in a normal playback scenario, each video frame can be displayed according to the playback time corresponding to the time stamp of the video frame in the video frame group, and the video screen is relatively smooth during the video playback process.

[0159] In the frame playback scenario, as mentioned above, the process of the user dragging the progress display frame is an uncertain process. The progress display frame may be dragged to the left or to the right, and the drag may be faster or slower. Accordingly, the video frame displayed in the video playback area 11 is random. Therefore, in the process of the user dragging the progress display frame, the electronic device needs to determine the moment when the progress indicator line points to based on the real-time detection of the contact position, and then parse, decode, render, and display the video frame at the moment when the progress indicator line points to; as mentioned above, before decoding the video frame at that moment, it may be necessary to decode other video frames in the same video frame group in order to decode the video frame at that moment; therefore, the frame playback scenario is prone to freezes and is not smooth.

[0160] In view of this, an embodiment of the present application provides a video playback method. The electronic device sets at least two decoders for the frame playback process, and uses the at least two set decoders to decode the frame playback process, thereby improving the decoding capability, thereby playing the video frame more smoothly.

[0161] Reference Figure 4 , which is a video playback process decoded by at least two decoders provided in an embodiment of the present application.

[0162] When the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 1 (which is different from the position of the progress display frame on the display screen, and position 1 is an abstract position corresponding to the video frame), the decoder 1 (which can also be recorded as the first decoder) in the electronic device starts decoding the video frame group 1 (which can also be used as an example of the first frame group) where the video frame corresponding to position 1 is located, and caches the decoded video frame (the video frame in the video frame group 1) in the first cache space corresponding to decoder 1; when it is determined that the video frame group 1 where position 1 is decoded by decoder 1, decoder 2 (which can be recorded as the second decoder) starts decoding the next video frame group 2 (which can be used as an example of the second frame group) where the video frame corresponding to position 1 is located, and caches the decoded video frame (the video frame in the video frame group 2) in the second cache space corresponding to decoder 2. Among them, the first cache space is the cache space for storing the video frame decoded by decoder 1; the second cache space is the cache space for storing the video frame decoded by decoder 2. Generally, when a decoder decodes a video frame in a video frame group, it decodes the video frame in the decoding order, and each time a video frame is decoded, the decoded video frame obtained by decoding can be cached in the corresponding cache space.

[0163] In this example, the relationship between position 1 and the position of the first control on the display screen is: when the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 1 (which is different from the position of the progress display frame on the display screen, and position 1 is an abstract position corresponding to the video frame), the first control will be moved to the second position. The position of the first control can be the position of the point in the upper left corner of the area where the first control is located, or the position of the point in the upper right corner of the area where the first control is located, or even the position of any pixel point in the first control.

[0164] As an example of the first control, the progress display frame can move along the first track as the user slides. The position of the first control before moving as the user slides can be recorded as the first position (i.e., the initial position), that is, as the user slides, the first control can start to move from the first position.

[0165] As the user slides, the first control moves to a certain position, and the play progress (also understood as the play time) indicated by the first control at this position corresponds to the video frame displayed in the first area. For example, the video frame displayed in the first area is the video frame corresponding to the play time.

[0166] When the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 2, the video frame group 1 where the video frame corresponding to position 2 is located has been assigned to decoder 1 for decoding. In actual applications, the video frame corresponding to position 2 may have been cached in the first cache space. Therefore, the electronic device can obtain the decoded video frame corresponding to position 2 from the first cache space and perform subsequent rendering and display. Since the decoded video frame can be directly obtained from the first cache space and then rendered and displayed, the smoothness of the picture can be improved and the jamming can be reduced. Of course, since the next video frame group 2 of the video frame group 1 corresponding to position 2 has been assigned to the decoder for decoding, there is no need for repeated decoding.

[0167] When the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 3, the first control will be moved to the third position (the difference between the play progress indicated when the first control is at the third position and the timestamp of the video frame corresponding to position 3 is less than or equal to the first duration), and the video frame corresponding to position 3 has been assigned to decoder 2 for decoding (it has been assigned to decoder 2 for decoding when the first control is moved to the second position as the user slides). In actual applications, the video frame corresponding to position 3 may have been cached in the second cache space, so the electronic device can directly obtain the decoded video frame corresponding to position 3 from the second cache space, and perform subsequent rendering and display. In addition, when the electronic device is triggered to display the video frame corresponding to position 3, the electronic device will trigger decoder 1 to start decoding the next video frame group 3 (which can be recorded as the third frame group) of video frame group 2 where the video frame corresponding to position 3 is located, and cache the decoded video frame in video frame group 3 in the first cache space.

[0168] When the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 4, the first control will be moved to the fourth position (the difference between the play progress indicated when the first control is at the fourth position and the timestamp of the video frame corresponding to position 4 is less than or equal to the first duration), and the video frame corresponding to position 4 has been assigned to decoder 1 for decoding (it has been assigned to decoder 1 for decoding when the first control is moved to the third position as the user slides), and decoder 1 has cached the decoded video frames in video frame group 3 (as an example of the fourth frame group) in the first cache space, and the electronic device can directly obtain the video frame corresponding to position 4 (as an example of the fourth video frame) from the first cache space, and perform subsequent rendering and display. Of course, the decoding task of video frame group 4 also needs to be assigned to decoder 2.

[0169] Of course, as the user slides, when the user drags the progress display frame to trigger the electronic device to display the video frame corresponding to position 5, the electronic device starts to decode video frame group k through the first decoder, decodes video frame group k+1 through the second decoder, and displays the video frames in video frame group k (for example, the video frame corresponding to position 5) at the same time.

[0170] In an embodiment of the present application, when the playback progress determined by two consecutive contact positions respectively corresponds to the playback progress corresponding to position 4 and position 5, since the video frame group k corresponding to position 5 and the video frame group 3 corresponding to position 4 are not two adjacent video frame groups, when it is determined according to the contact position that the video frame corresponding to position 5 needs to be displayed, the video frame group K may not have been decoded in advance. In this case, the parser can assign video frame K to other decoders other than the decoder assigned when the decoding task was assigned last time.

[0171] For example, the last time the decoding task was assigned, the decoding task of video frame group 4 was assigned, and the decoding task of video frame group 4 was assigned to decoder 2. Then, when the decoding task is assigned this time, the task of decoding video frame group k is assigned to decoder 1, and the task of decoding video frame group k+1 is assigned to decoder 2. Figure 4 In the example shown, K is a natural number greater than 5.

[0172] Based on the above process, it can be understood that when the drag operation triggers the electronic device to display the video frames in video frame group j, another decoder (a decoder different from the decoder corresponding to video frame group j) will decode video frame group j+1; when the drag operation triggers the electronic device to display the video frames in video frame group j+1, the video frames in video frame group j+1 have been decoded when the display of the video frames in video frame group j is triggered, and cached in the corresponding cache space. The electronic device can directly obtain the decoded video frames in video frame group j+1 from the cache space for rendering and display, thereby improving the picture smoothness during frame playback.

[0173] Of course, when the dragging operation triggers the electronic device to display the video frames in video frame group j+1, there will also be a decoder decoding the video frames in video frame group j+2; in this way, there will always be a decoder to perform decoding work in advance. When the user actually drags to the corresponding position to trigger the display of a certain video frame, the corresponding video frame at that moment may have been decoded and cached, thereby improving the smoothness of the picture during frame playback.

[0174] In addition, in actual applications, only one decoder can be set for the frame playback scenario, and each video frame group is decoded by the decoder. However, if the decoding capability of the decoder is weak (the decoding speed is slow, the decoding of the previous video frame group has not yet been completed, and the decoding task of the next video frame group has arrived), two or even more decoders are set to take turns decoding.

[0175] As an example, when three decoders are set, if the user's drag operation triggers the electronic device to continuously display video frames in video frame group j, video frame j+1, video frame group j+2 and video frame group j+3, video frame group j is decoded by the first decoder, video frame group j+1 is decoded by the second decoder, video frame group j+2 is decoded by the third decoder, and video frame group j+3 is decoded by the first decoder.

[0176] Of course, the user's dragging operations include dragging to the left and dragging to the right; therefore, correspondingly, when the dragging process triggers the display of a video frame with a smaller timestamp (smaller than the timestamp of the last displayed video frame), for example, when the dragging operation triggers the electronic device to display the video frames in video frame group j, another decoder (a decoder different from the decoder corresponding to video frame group i) will decode video frame group j-1; when the dragging operation triggers the electronic device to display the video frames in video frame group j-1, the video frames in video frame group j-1 may have been decoded and cached in the cache space, and the electronic device can directly obtain the decoded video frames from the cache space for rendering and display, thereby improving the smoothness of the picture during frame playback.

[0177] Of course, when the dragging operation triggers the electronic device to display the video frames in video frame group j-1, there will also be a decoder decoding the video frames in video frame group j-2; in this way, there will always be a decoder to perform decoding work in advance. When the user actually drags to the corresponding position to trigger the display of a certain video frame, the video frame may have been decoded and cached, thereby improving the smoothness of the picture during frame playback.

[0178] In the above example, if the sliding operation triggers the electronic device to display a video frame at a certain moment, a video frame whose difference between the timestamp and the moment is less than a first value (half of the difference between the timestamps of two adjacent video frames in the video) can be obtained from the cache space. That is, among the video frames of the video file, the video frame whose difference between the timestamp and the moment is the smallest.

[0179] In addition, in actual applications, since it takes time for the decoder to decode a group of video frames, when the user drags faster and triggers the electronic device to display the video frame at a certain moment, the decoder may not have decoded the video frame corresponding to that moment in the group of video frames. Therefore, it can also be set: when the user drags faster, if the video frame to be displayed has not been successfully decoded, a decoded video frame with the most recent timestamp (smallest difference) of the video frame corresponding to the dragging operation can be obtained from the cache space for rendering and display; when the user drags to a certain speed, only the key frames in a group of video frames may be decoded and rendered for display.

[0180] The above embodiments all take the example of two decoders performing decoding alternately and one decoder always performing decoding of one video frame group in advance; in actual applications, more than two decoders can be set to perform decoding alternately, for example, decoding is performed in the order of decoder one, decoder two and decoder three, decoder one, decoder two and decoder three, while at least one decoder is always kept to perform decoding of at least one video frame group in advance (for example, two decoders can also be set to always perform decoding of two video frame groups in advance), and the embodiments of the present application will not give examples one by one.

[0181] In order to make the video playback method provided by the present application more clear, the video playback method provided by the embodiment of the present application is described in detail below through a timing diagram.

[0182] Figure 5 A timing diagram of the preparation work before frame playback provided in the embodiment of the present application. The process includes the following steps:

[0183] S101, the gallery application displays a grid image, which includes a cover of video A (as an example of a first video).

[0184] The grid image displayed by the gallery application can include not only the cover of video A, but also the covers of other videos, and of course, thumbnails of pictures. Figure 2 The grid diagram displayed in the interface shown in (a) can be used as an example of the grid diagram in this step.

[0185] S102: The gallery application receives a click operation on the cover of video A.

[0186] In the embodiment of the present application, the electronic device displays Figure 2 In the interface shown in (a) in the figure, the user clicks Figure 2 In the interface shown in (a), the electronic device displays the cover of any video displayed in the grid image in the interface, and in response to the operation, displays the video playback interface of the video through the gallery application. Figure 2 A click operation of a user's finger in the interface shown in (a).

[0187] It should be noted that in actual applications, other modules of the electronic device may receive the click operation and transmit the click operation to the gallery application. For the sake of simplicity of description, the embodiment of the present application omits the intermediate steps and takes the example that the gallery application can receive the click operation on the cover of video A.

[0188] S103, after receiving a click operation on the cover of video A, the gallery application sends the file path of video A to the frame playback controller.

[0189] In an embodiment of the present application, the gallery application of the electronic device stores the file paths of various pictures and video files in the electronic device, and the gallery application can display pictures and play videos based on the file paths of various picture and video files.

[0190] S104: After receiving the file path of video A, the frame playback controller sends the file path of video A to the frame playback scheduler.

[0191] S105, after receiving the file path of video A, the frame playback scheduler sends the file path of video A to the parser.

[0192] S201, after sending the file path of video A to the frame playback controller, the gallery application sends a frame playback start request to the frame playback controller.

[0193] In the embodiment of the present application, S201 is executed after step S103, but there is no strict order of execution between S201 and S104 and S105.

[0194] S202: After receiving the frame play start request, the frame play controller sends the frame play start request to the frame play scheduler.

[0195] S203: After receiving the frame play start request, the frame play scheduler sends a parsing thread start request to the parser.

[0196] S204, after receiving the parsing thread start request, the parser starts the parsing thread.

[0197] S205, after successfully starting the parsing thread, the parser sends information indicating that the parsing thread has been successfully started to the frame playback scheduler.

[0198] S206: The frame playback scheduler sends a decoding thread one start request to decoder one.

[0199] S207, after receiving the request to start decoding thread 1, decoder 1 starts decoding thread 1.

[0200] S208, after successfully starting decoding thread one, decoder one sends information indicating that decoding thread one has been successfully started to the frame playback scheduler.

[0201] S209, the frame playback scheduler sends a decoding thread two start request to decoder two.

[0202] S210, after receiving the request to start the decoding thread 2, the decoder 2 starts the decoding thread 2.

[0203] S211, after successfully starting the decoding thread 2, the decoder 2 sends information indicating that the decoding thread 2 has been successfully started to the frame playback scheduler.

[0204] In the embodiment of the present application, more decoders may be provided to improve the decoding capability of the electronic device. If more decoders are provided for frame playback, they may be recorded in order as decoder three...etc.

[0205] S212: The frame playback scheduler sends a rendering thread start request to the rendering module.

[0206] S213: After receiving the rendering thread start request, the rendering module starts the rendering thread.

[0207] S214, after successfully starting the rendering thread, the rendering module sends information indicating that the rendering thread has been successfully started to the frame playback scheduler.

[0208] S215, after receiving the click operation on the cover of video A, the gallery application displays the video playback interface of video A.

[0209] The video playback interface of video A can be referred to Figure 2 In the interface shown in (b), the video playback interface of video A includes a video playback area and a progress display area for displaying the playback progress.

[0210] It should be noted that in response to the click operation, the electronic device is also executing S103 to S214, and is also executing the process of displaying the video playback interface of video A (S215). Therefore, there is no strict order between step S103 (and subsequent steps to S214) and step S215.

[0211] After this step, the preparation for frame playback is completed. After receiving the operation for frame playback, the electronic device will perform frame playback of the video.

[0212] The embodiments of the present application mainly describe the frame playback process of a video to illustrate that the picture smoothness of the frame playback process can be improved by using at least two decoders.

[0213] Reference Figure 6 , Figure 6 The timing diagram of the video frame corresponding to the video playback process display position 1 provided in the embodiment of the present application. S301, the gallery application receives a drag operation on the progress display frame.

[0214] For an electronic device including a touch screen, a touch driver may sample the user's touch operation according to a period T. The above period, also called a control period or a reporting period, is a period in which the touch driver samples the user's touch operation.

[0215] For example, the touch driver collects data 240 times per second. The period T = 1 / 240 = 4.16ms, that is, the touch driver collects touch data every 4.16ms. If the user presses the touch screen with one or two fingers in a certain period, the touch driver can determine the touch position of the single or double fingers based on voltage changes.

[0216] For example, when the user wants to change the video frame displayed in the video playback area by dragging the position of the progress display frame, the touch driver receives the user's click operation corresponding to the progress display frame and passes the down event to the application processor. The application processor passes the down event to the gallery application. After receiving the click event, the touch driver continues to receive the user's continuous drag operation corresponding to the progress display frame and passes the move event to the application processor. The application processor passes the move event to the gallery application.

[0217] S302, the gallery application determines the time ti (as an example of the first time) for playing the frame of video A according to the dragging position.

[0218] The time ti of frame playback is not the current time, but the video frame corresponding to the time ti in the video file is played. The explanation of the subsequent time ti+1, ti+2, etc. will not be repeated.

[0219] In the embodiment of the present application, a continuous drag operation includes: the user's finger touches the progress display frame (down event), the user's finger moves (move event), and the user's finger is released from the progress display frame (up event). The time ti can be the playback progress corresponding to the first contact position in the move event, or it can be the playback progress corresponding to any contact position in the move event.

[0220] After the gallery application determines the time ti of the frame playback of video A according to the dragging position, it needs to move the position of the progress display frame to display the video frame corresponding to the time ti in the video playback area.

[0221] As an example, and Figure 4 The video frame corresponding to the example shown is Figure 4 The video frame corresponding to position 1 in .

[0222] S303, the gallery application sends a frame playback request to the frame playback controller, carrying the time ti of video A.

[0223] S304: After receiving the frame playback request, the frame playback controller sends the frame playback request to the frame playback scheduler, carrying the time ti of video A.

[0224] S306: After receiving the frame playback request, the frame playback scheduler sends a parsing request to the parser, carrying the time ti of video A.

[0225] S307: After receiving the frame playback request, the frame playback scheduler sends the time ti of video A to the rendering module.

[0226] S309: After receiving the parsing request, the parser parses the video A through the parsing thread to obtain the video frame group j (with Figure 4 The example shown corresponds to the encoded video frames in the video frame group 1), and records the start time ts and end time te corresponding to the video frame group j.

[0227] The timestamp of the first video frame in the video frame group j may be used as the start time ts, and the timestamp of the last video frame in the video frame group j may be used as the end time te.

[0228] In practical applications, ts-1 / 2*T2 and te+1 / 2*T2 may also be recorded as the start time and the end time, where T2 is the difference between the timestamps of two adjacent video frames.

[0229] S310, the parser sends a decoding request to decoder 1, carrying the encoded video frames in video frame group j (which may be all the encoded video frames in video frame group j).

[0230] When determining that the encoded video frame in video frame group j is decoded by decoder one, the parsing module determines the video frame in the next video frame group j+1 of video frame group j decoded by another decoder (decoder two), that is, the encoded video frame in video frame group j+1 is decoded by decoder two to obtain a decoded video frame.

[0231] In the specific implementation, it is also necessary to distinguish between forward playback and reverse playback. As an example, if ti is greater than ti-1, it is forward playback, and the playback progress corresponding to the subsequent drag operation is likely to be greater than ti, so the next video frame group j+1 of the video frame group j where ti is located can be decoded in advance. The next video frame group is adjacent to the video frame group j where the video frame corresponding to the time ti is located, and the timestamp is greater than the video frame group j.

[0232] Of course, if ti is less than ti-1, it is reverse playback, and the playback progress corresponding to the subsequent dragging operation is likely to be less than ti. Therefore, the previous video frame group j-1 of the video frame group j where ti is located can be decoded in advance. The previous video frame group is adjacent to the video frame group j where the video frame corresponding to the moment ti is located, and the time stamp is less than the video frame group j.

[0233] The above-mentioned step of determining forward playback or reverse playback can be performed by the gallery application, and then the playback mode (forward playback or reverse playback) is transmitted to the parser. The parser determines the previous video frame group (when playing backwards) or the next video frame group (when playing forward) of the video frame group corresponding to the moment ti according to the playback mode.

[0234] As another example, the frame playback controller, frame playback scheduler or parser may also determine the playback mode according to the relationship between the current received time and the last received time. This application does not cite examples one by one through diagrams.

[0235] S311, the parser sends a decoding request to decoder 2, carrying the encoded video frame in video frame group j+1.

[0236] and Figure 4 Corresponding to the embodiment shown, the video frame group j+1 can be understood as Figure 4 The video frame group 2 in .

[0237] S312, after receiving the decoding request, decoder 1 decodes the encoded video frames in video frame group j through decoding thread 1 to obtain decoded video frames.

[0238] In an embodiment of the present application, decoder 1 may cache the decoded video frames in the first cache space in a decoding order.

[0239] S313, after receiving the decoding request, decoder 2 decodes the encoded video frame in video frame group j+1 through decoding thread 2 to obtain a decoded video frame.

[0240] In an embodiment of the present application, decoder 2 may cache the decoded video frames in a second cache space in a decoding order.

[0241] S314, the rendering module determines that the decoder of the video frame group j to which the video frame corresponding to the decoding time ti belongs is decoder one.

[0242] In an embodiment of the present application, the rendering module may determine the decoder of the video frame group j where the video frame corresponding to ti is located by means of round-robin training.

[0243] As an example, the rendering thread searches for the first cached video frame (the first video frame is a key frame in the video frame group) from the first cache space corresponding to the decoder one. If the difference between the timestamp of ti and the first video frame in the first cache space is less than the time span T1 of a video frame group (the difference between the timestamps of the key frames in two adjacent video frame groups), it means that the video frame group j corresponding to the time ti is decoded by the decoder one.

[0244] In this example, ti is subtracted from the timestamp of the first video frame in the first cache space to obtain a second difference; if the second difference is in the range of [0, T1), it is determined that the video frame group j to which the video frame corresponding to time ti belongs is decoded by decoder one.

[0245] That is, [0, T1) can be taken as an example of the first time range.

[0246] In addition, [0-1 / 2*T2, T1-1 / 2*T2) may be set as the first time range, and T2 is the difference between the timestamps of two adjacent video frames.

[0247] Correspondingly, if the difference between ti and the timestamp of the first video frame in the first cache space is not less than the time span of a video frame group (the difference between the timestamps of the key frames in two adjacent video frame groups), the first cached video frame (the first video frame is the key frame in the video frame group) is searched from the second cache space corresponding to decoder two. If the difference between ti and the timestamp of the first video frame in the second cache space is less than the time span of a video frame group (the difference between the timestamps of the key frames in two adjacent video frame groups), it means that the video frame group j to which the video frame corresponding to time ti belongs is decoded by decoder two. In this way, the decoder corresponding to the video frame group j to which the video frame corresponding to time ti belongs can be determined.

[0248] S315, the rendering module obtains the video frame corresponding to the time ti from the decoder 1.

[0249] The embodiment of the present application takes the decoder of the video frame group j where the video frame corresponding to the time ti is located as decoder 1 as an example. After determining that the decoder where the video frame group j is located is decoder 1, a video frame whose difference between the timestamp and ti is less than or equal to 1 / 2*T2 (the difference between the timestamps of two adjacent video frames) is obtained from decoder 1 as the video frame corresponding to the time ti. This is because ti is a moment determined by the user's drag operation. In actual applications, there may be no video frame with a timestamp of ti. In this case, a video frame with a timestamp closest to ti in the group of video frames is obtained as the video frame corresponding to the time ti.

[0250] S316, the rendering module renders the video frame corresponding to time ti through the rendering thread to obtain the picture corresponding to time ti.

[0251] S317, the rendering module sends the picture corresponding to time ti to the frame playback scheduler.

[0252] S318, after receiving the picture corresponding to time ti, the frame playback scheduler sends the picture corresponding to time ti to the frame playback controller.

[0253] S319, after receiving the picture corresponding to time ti, the frame playback controller sends the picture corresponding to time ti to the gallery application.

[0254] S320, after receiving the picture corresponding to time ti, the gallery application displays the picture corresponding to time ti in the video playback area (as an example of the picture corresponding to the second video frame).

[0255] In actual applications, after S302, the gallery application also needs to determine the position of the first control according to the real-time contact position of the sliding operation, so as to move the first control in the video playback interface so that the first control moves to a specific position, and the playback progress represented by the first control at the specific position is the moment ti.

[0256] Of course, taking a user's sliding operation as an example (including down event, move event and up event), this sliding operation will trigger the first control to move along the first track. When the down event is detected, the position of the first control at that time is the first position, and the first control moves on the fixed first track. The move event in the sliding operation will trigger the first control to move from the first position along the first track.

[0257] During the process of the user dragging the progress display frame, the contact position of the user's finger detected by the electronic device may change. Accordingly, the positional relationship between the progress indicator line and the progress display frame is also changing, and the time corresponding to the progress indicator line is also changing. If the time corresponding to the two consecutive contact positions detected is ti and ti+1 respectively, the process of the electronic device displaying the video frame corresponding to ti+1 can refer to Figure 7 The frame playback process shown in FIG. 1 is a process in which the video frame group corresponding to the time ti+1 is the same as the video frame group corresponding to the time ti. Figure 4 The video frame corresponding to position 2 in . Figure 7 The frame playback process shown includes the following steps:

[0258] S401: The gallery application receives a drag operation on a progress display frame.

[0259] It should be noted that the dragging operation in step S401 and step S301 is the same continuous operation, that is, the user's dragging operation is not released between step S401 and step S301, but the time corresponding to the progress indicator line is changed due to the dragging operation.

[0260] S402, the gallery application determines the time ti+1 (which can be used as an example of the second time) for playing the frame of video A according to the dragging position.

[0261] As an example of the present application, the video frame with the smallest difference between the timestamp and the time ti can be Figure 4 The video frame corresponding to position 2 in .

[0262] S403, the gallery application sends a frame play request to the frame play controller, carrying the time ti+1 of video A.

[0263] S404: After receiving the frame playback request, the frame playback controller sends a frame playback request to the frame playback scheduler, carrying the time ti+1 of video A.

[0264] S405, after receiving the frame play request, the frame play scheduler calculates the difference between the time ti+1 in the frame play request received this time and the time ti in the frame play request received last time.

[0265] When the difference between ti+1 and ti is less than 16.7 ms, ti+1 is discarded, and S406 and subsequent steps are no longer executed.

[0266] S406, when the difference between ti+1 and ti is greater than or equal to 16.7 ms, the frame playback scheduler sends a parsing request to the parser, carrying the time ti+1 of video A, and records the time ti+1.

[0267] Usually, for video playback with a frame rate of 30fps or above, the human eye cannot perceive the difference; only when the frame rate is lower than 30fps can the human eye perceive the difference. Therefore, the frame rate of the video can be filtered to a suitable range. For example, if it is filtered to 30fps, the interval between video frames is 1000ms / 30=33.3ms, and 33.3ms is the difference threshold. Of course, in actual applications, if the frame rate is 120fps, the frame rate can also be filtered to 60fps, then the interval between video frames is 1000ms / 60=16.7ms, and 16.7ms is the difference threshold.

[0268] Or, as a compromise example, filtering the frame rate to between 30fps and 60fps can also reduce the number of video frames in the video, which can greatly reduce the power consumption of the decoder.

[0269] Based on the above principle, 16.6ms and 33.3ms are only examples of difference thresholds. In actual applications, it can also be a value around 16.6ms, for example, 15ms, 17ms, 18ms, 20ms, etc., or a value around 33.3ms; of course, it can also be any value between 16.6ms and 33.3ms.

[0270] Reference Figure 8, is the process of video filtering, taking filtering to 60fps as an example, the interval of video frames is 1000ms / 60=16.7ms. The difference threshold is 16.7ms.

[0271] As the user drags the progress display frame, the electronic device determines the various moments according to the continuous contact positions:

[0272] 11:43:46.646-3013000, 11:43:46.659-3784000, 11:43:46.676-4758000, 11:43:46.693-5528000, 11:43:46.710-6389000;

[0273] The time at which the first contact point is determined is 11:43:46.646-3013000; this time is retained for subsequent decoding, rendering, and display;

[0274] The time at which the second contact position is determined is 11:43:46.659-3784000, and the difference with the last retained time (11:43:46.646) is: 659-646=13ms, which is less than 16.7ms, so the time is discarded;

[0275] The time at which the third contact position is determined is 11:43:46.676-4758000, and the difference from the last retained time (11:43:46.659) is: 676-659=17ms, which is greater than 16.7ms. This time is retained for subsequent decoding, rendering and display;

[0276] The time at which the fourth contact position is determined is 11:43:46.693-5528000, and the difference from the last retained time (11:43:46.676) is: 693-676=17ms, which is greater than 16.7ms. This time is retained for subsequent decoding, rendering, and display;

[0277] The time at which the fifth contact position is determined is 11:43:46.710-6389000, and the difference between the time at which the last touch point is retained (11:43:46.693) is: 710-693=17ms, which is greater than 16.7ms. The time is retained for subsequent decoding, rendering, and display.

[0278] S407, when the difference between ti+1 and ti is greater than or equal to 16.7 ms, the frame playback scheduler sends the time ti+1 of video A to the rendering module.

[0279] S408: After receiving the parsing request, the parser determines that ti+1 is within the start time ts and the end time te of the record.

[0280] In the present application, the timestamp ts of the first video frame and the timestamp te of the last video frame in the video frame group where the last obtained time ti is located have been recorded in S309, and it can be determined whether ti+1 is greater than or equal to ts and less than or equal to te; if ti+1 is greater than or equal to ts and less than or equal to te, it means that the video frame corresponding to ti+1 is in video frame group j (the video frame group where the video frame corresponding to ti is located); if ti+1 is less than ts or greater than te, it means that the video frame corresponding to ti+1 is not in video frame group j.

[0281] After the parser determines that the video frame corresponding to timestamp ti+1 is in video frame group j, since video frame group j has been handed over to decoder 1 for decoding, video A is no longer parsed repeatedly, and the decoding instruction of video frame group j is no longer sent to the decoding module, because the video frame group j where the ti+1 video frame is located has been parsed before, and the decoder has also decoded the video frame group j where the video frame corresponding to ti+1 is located in sequence.

[0282] In the embodiment of the present application, Figure 4 The video frame corresponding to position 2 in can be understood as the video frame whose timestamp difference between video frame j and ti+1 is the smallest.

[0283] S414, the rendering module determines that the video frame group j to which the video frame corresponding to the time ti+1 belongs is decoded by the decoder one.

[0284] The process in which the rendering module determines that the video frame group j to which the video frame corresponding to the time ti+1 belongs is decoded by the decoder 1 can refer to the description of the above embodiment, which will not be repeated here.

[0285] S415, the rendering module obtains the video frame corresponding to time ti+1 from decoder 1.

[0286] For example, a video frame whose difference between a timestamp and ti+1 is less than a first value is obtained from a first cache space corresponding to decoder 1, where the first value is half of the difference between the timestamps of two adjacent video frames.

[0287] S416, the rendering module renders the video frame corresponding to the time ti+1 through the rendering thread to obtain the picture corresponding to the time ti+1.

[0288] S417, the rendering module sends the picture corresponding to time ti+1 to the frame playback scheduler.

[0289] S418, after receiving the picture corresponding to time ti+1, the frame playback scheduler sends the picture corresponding to time ti+1 to the frame playback controller.

[0290] S419, after receiving the picture corresponding to time ti+1, the frame playback controller sends the picture corresponding to time ti+1 to the gallery application.

[0291] S420, after receiving the picture corresponding to time ti+1, the gallery application displays the picture corresponding to time ti+1 in the video playback area.

[0292] Similarly, the gallery application also needs to determine the position of the first control according to the real-time contact position of the sliding operation, so as to move the first control in the video playback interface so that the first control moves to a specific position. The playback progress represented by the first control at the specific position is the moment ti+1.

[0293] Below Fig. 9 Describes the video playback process performed by the electronic device when the drag operation triggers the electronic device to display the video frame corresponding to position 3 (the difference between the timestamp and ti+2 is less than the video frame corresponding to the first value).

[0294] S501: The gallery application receives a drag operation on a progress display frame.

[0295] It should be noted that the dragging operation in step S501, step S401 and step S301 is the same continuous operation, that is, the user's dragging operation is not released between step S501 and step S301, but the time corresponding to the progress indicator line is changed due to the dragging operation.

[0296] S502, the gallery application determines the time ti+2 (which can be used as an example of the second time) at which the frame of video A is played according to the dragging position.

[0297] S503, the gallery application sends a frame play request to the frame play controller, carrying the time ti+2 of video A.

[0298] S504: After receiving the frame playback request, the frame playback controller sends a frame playback request to the frame playback scheduler, carrying the time ti+2 of video A.

[0299] S505, after receiving the frame playback request, the frame playback scheduler calculates the difference between the time ti+2 in the currently received parsing request and the time ti+1 carried in the last received parsing request, and determines that the difference is greater than 16.7 ms.

[0300] S506: When the difference between ti+2 and ti+1 is greater than or equal to 16.7 ms, the frame playback scheduler sends a parsing request to the parser, carrying the time ti+2 of video A, and records the time ti+2.

[0301] In practical applications, if the times corresponding to the consecutive contact positions are ti and ti+2, then the difference between ti+2 and ti needs to be calculated. In the embodiment of the present application, the times corresponding to the consecutive contact positions are ti, ti+1 and ti+2 as an example.

[0302] S507: The frame playback scheduler sends a rendering request to the rendering module, carrying the time ti+2.

[0303] S508: After receiving the parsing request sent by the frame playback scheduler, the parser determines that the time ti+2 is not within the range of the recorded start time ts and te.

[0304] S509, the parser parses the video A, obtains the video frame group j+1 (video frame group 2) where the video frame corresponding to ti+2 is located, and records the start time ts and te of the video frame group j+1.

[0305] In the present application, after each new video frame group is obtained by parsing video A, ts and te of the latest video frame group are recorded, and the previously recorded ts and te are overwritten.

[0306] S510, the parser queries whether the video frame group j+1 has been decoded by the decoder 2.

[0307] In this case, there is no need to repeatedly decode the video frame group j+1 by the decoder.

[0308] S511, the parser sends a decoding request to decoder 1, carrying video frame group j+2 (equivalent to Figure 4 The encoded video frames in the video frame group 3).

[0309] S513, decoder 1 decodes the encoded video frame in video frame group j+2 through decoding thread 1 to obtain a decoded video frame.

[0310] S514, the rendering module determines that the video frame group j+1 to which the video frame corresponding to ti+2 belongs is decoded by the second decoder.

[0311] Among them, the rendering module determines the process of the decoder corresponding to the video frame group j+1 where the video frame corresponding to ti+2 is located: first query the timestamp of the first key frame cached in decoder one (or in the first cache space corresponding to decoder one), and determine that the decoder of video frame group j+1 is not decoder one according to the relationship between the timestamp and ti+2 (the second difference between ti+2 and the queried timestamp is not within the first time range); if it is determined that it is not decoder one, query the timestamp of the first key frame cached in decoder two (or the second cache space corresponding to decoder two), and determine that the decoder of video frame group j+1 is decoder two according to the relationship between the timestamp and ti+2 (the second difference between ti+2 and the queried timestamp is within the first time range). The specific process of determining the decoder can also refer to the description in the above embodiment, which will not be repeated here.

[0312] S515, the rendering module obtains the video frame corresponding to ti+2 from the decoder 2. As another embodiment of the present application, in actual application, when the user drags the video frame quickly, the following situation may occur:

[0313] Decoder 2 needs to follow Figure 3 The decoding order shown is decoded, and if the difference between the timestamp and ti+2 is less than the first value (half of the difference between the timestamps of two adjacent video frames), the video frame is the 7th decoded video frame (B frame). When the user's dragging operation triggers the electronic device to display the video frame corresponding to ti+2 (the video frame corresponding to position 3), due to the fast dragging operation, although the decoder 2 decodes the group of video frames in advance, after the electronic device executes S514, the decoder 2 has not yet decoded the video frame corresponding to position 3 (the 7th video frame). In this case, it can also be set that the rendering module obtains a video frame with the smallest difference between the timestamp and ti+2 from the second cache space (the cache space corresponding to the decoder 2), and the video frame is used as the video frame corresponding to the ti+2 moment. That is, the video frame with the smallest difference between the currently successfully decoded video frame and ti+2 is used as the video frame corresponding to the ti+2 moment for subsequent rendering and display, so that the video frame can be obtained as soon as possible for rendering and display, reducing the screen jamming.

[0314] As an example, the rendering module may sequentially calculate the difference between the timestamp of each video frame cached in the second cache space and ti+2, and select a video frame with the smallest difference as the video frame corresponding to time ti+2.

[0315] S516, the rendering module renders the video frame corresponding to time ti+2 through the rendering thread to obtain the picture corresponding to time ti+2.

[0316] S517, the rendering module sends the picture corresponding to time ti+2 to the frame playback scheduler.

[0317] S518, after receiving the picture corresponding to time ti+2, the frame playback scheduler sends the picture corresponding to time ti+2 to the frame playback controller.

[0318] S519, after receiving the picture corresponding to time ti+2, the frame playback controller sends the picture corresponding to time ti+2 to the gallery application.

[0319] S520, after receiving the picture corresponding to time ti+2, the gallery application displays the picture corresponding to time ti+2 in the video playback area (as an example of the picture corresponding to the third video frame).

[0320] In actual applications, the user's dragging operation may be very fast. In this case, in order to enable the electronic device to quickly display the decoded picture, it can be set that each decoder only decodes the I frame in a group of video frames. Accordingly, when the rendering thread obtains the video frame corresponding to a certain moment, it can obtain the I frame from the cache space of a determined decoder (the decoder used to decode the video frame group to which the video frame corresponding to the moment belongs), and the obtained I frame is used as the video frame corresponding to the moment.

[0321] As an example, when the drag operation triggers the electronic device to display the video frame corresponding to the time tk (for example, Figure 4 ), where position 5 and position 4 are video frames pointed to by the playback moments corresponding to two consecutive contact positions detected by the electronic device. For example, the playback moment corresponding to the last contact position detected by the electronic device is ti+3, and position 4 is a video frame whose difference between the timestamp and ti+3 is less than the first value. In theory, the electronic device should display the video frame corresponding to position 4; the playback moment corresponding to the next contact position detected by the electronic device is ti+4, and position 5 is a video frame whose difference between the timestamp and ti+4 is less than the first value. In theory, the electronic device should display the video frame corresponding to position 5. However, due to the fast dragging operation of the user, the following situation may occur:

[0322] After the electronic device executes the decoder of the video frame group where ti+4 is determined to be decoded, if the decoder has not yet decoded the video frame corresponding to position 5, it can obtain a video frame with the smallest difference between the timestamp and ti+4 from the cache space of the determined decoder as the video frame corresponding to the time ti+4 for rendering and display; or directly obtain the first video frame (key frame) from the cache space of the determined decoder as the video frame corresponding to the time ti+4 for rendering and display.

[0323] As an example, the rendering thread can calculate the difference between the currently received time ti+4 and the last received time ti+3. When the difference is large, the first video frame (key frame) is directly obtained from the cache space of the determined decoder as the video frame corresponding to the time ti+4 for rendering and display; when the difference is small, the video frame with the smallest difference between the timestamp and ti+4 is obtained from the cache space of the determined decoder as the video frame corresponding to the time ti+4 for rendering and display.

[0324] As another example, when the difference is large, since the key frame is directly obtained, in order to avoid the decoder from decoding non-key frames (which will not be used later, but will cause resource waste), the rendering module can notify the decoder to only decode the key frames in a group of video frames when the difference is large. When the difference is small, the decoder is notified to decode the video frames in a group of video frames in the decoding order. This process can be detailed in Fig.10 and Fig.11 The embodiment shown.

[0325] The process executed by the electronic device is similar to the process in the above embodiment, and the differences include:

[0326] S601, the rendering module receives time ti+4.

[0327] S602, the rendering module calculates the difference between the currently received ti+4 and the last received ti+3, and determines that the difference is greater than 500ms (an example of the second duration).

[0328] 500ms is just an example. In actual applications, it can be 400ms, 600ms, etc. This value can be determined according to the decoding speed of the decoder and experience to achieve a balance between decoding capability and video picture smoothness. The second duration is used to determine whether to decode only key frames.

[0329] S603: When determining that the difference is greater than 500 ms, the rendering module sends an instruction to the decoder 1 to instruct the decoder 1 to decode only the I frame in the video frame group.

[0330] S604: When determining that the difference is greater than 500 ms, the rendering module sends an instruction to the second decoder to instruct the second decoder to decode only the I frame in the video frame group.

[0331] S605 , the rendering module determines that the decoder of the video frame group k to which the video frame corresponding to ti+4 belongs is decoder one.

[0332] S606: The rendering module obtains the video frame whose timestamp has the smallest difference with time t i+4 from the cache space where decoder 1 is located, as the video frame corresponding to time t i+4.

[0333] In the case where the decoder decodes only the I frame in the video frame group, the video frame whose time stamp obtained from the cache space has the smallest difference with t i+4 may be a key frame (I frame).

[0334] S607: The rendering module renders the decoded video frame at time t i+4 through the rendering thread to obtain the picture at time t i+4.

[0335] Subsequently, the rendering module sends the image at time t i+4 to the gallery application for display.

[0336] Of course, in actual applications, there is no strict order of execution for the process from S602 to S604 and the process from S605 to S607. The process from S602 to S604 may be executed first, and then the process from S605 to S607; or the process from S605 to S607 may be executed first, and then the process from S602 to S604; or the two processes may be executed simultaneously through two independent threads.

[0337] and Fig.10 Corresponding to the illustrated embodiment, when the user's dragging operation is not too fast, it is also necessary to restore the normal decoding logic (decoding the video frames in the video frame group according to the decoding order).

[0338] S701, the rendering module receives the time ti+5.

[0339] S702, the rendering module calculates the difference between the currently received ti+5 and the last received ti+4, and determines that the difference is not greater than 500ms.

[0340] S703: When determining that the difference is not greater than 500 ms, the rendering module sends an instruction to the decoder 1 to instruct the decoder 1 to cancel decoding of only the I frame.

[0341] S704: When determining that the difference is not greater than 500 ms, the rendering module sends an instruction to the second decoder to instruct the second decoder to cancel decoding of only the I frame.

[0342] S705 , the rendering module determines that the decoder of the video frame group k+1 where the video frame corresponding to ti+5 is located is decoder 2.

[0343] S706: The rendering module obtains the video frame whose timestamp has the smallest difference with time t i+5 from the cache space where the decoder 2 is located, as the video frame corresponding to time t i+5.

[0344] S707, the rendering module renders the decoded video frame at time t i+5 through the rendering thread to obtain the picture at time t i+5.

[0345] Subsequently, the rendering module sends the image at time t i+5 to the gallery application for display.

[0346] Similarly, in actual applications, there is no strict order of execution for the process from S702 to S704 and the process from S705 to S707. The process from S702 to S704 may be executed first, and then the process from S705 to S707; or the process from S705 to S707 may be executed first, and then the process from S702 to S704; or the two processes may be executed simultaneously through two independent threads.

[0347] certainly, Fig.10 and Fig.11 In the illustrated embodiment, it is also necessary to notify another decoder to decode the video frame in the next video frame group, but it is not shown in the figure.

[0348] In addition, in some extreme scenarios, for example, the dragging operation is very fast. During the process of displaying the video frame corresponding to the time ti+m as the dragging operation is performed (the video frame group in which it is located is the video frame group j+n), the video frame group in which the playback progress (playback time) corresponding to the next contact position is located is most likely not the adjacent video frame group of the video frame group j+n. Therefore, the decoder may not be able to decode in advance. However, since only key frames are decoded, the decoding time before rendering is greatly reduced, thereby reducing the freeze of the picture and improving the smoothness of the picture.

[0349] In the above embodiment, the two moments when the positions of any two consecutive touch points are determined can be used as examples of the first moment and the second moment; the position of the first control related to the corresponding first moment (or second moment), the determined video frame group, the displayed video frame, etc. can all be determined based on the internal logic corresponding to the above video playback process.

[0350] In the above embodiments, the forward playback scene is taken as an example. The process of the reverse playback scene is similar to the process of the forward playback scene, except that when determining the next video frame group, the previous video frame group is determined. The explanation of the next video frame group and the previous video frame group can refer to the description in the above embodiments in detail. In practical applications, there may also be a scene in which the video is played forward for a while and played backward for a while.

[0351] In the above embodiment, the first cache space can only cache video frames of one video frame group, and when the decoder decodes video frames in a new video frame group, the video frames of the new video frame group overwrite video frames of the old video frame group. Similarly, the second cache space can only cache video frames of one video frame group, and when the decoder decodes video frames in a new video frame group, the video frames of the new video frame group overwrite video frames of the old video frame group.

[0352] As another embodiment of the present application, the video playback method includes:

[0353] Displaying a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video;

[0354] Receiving a sliding operation on the first interface;

[0355] In response to the sliding operation, moving the first control from a first position along a first track;

[0356] During the process of moving the first control from the first position along the first track:

[0357] When the first control is moved from the first position to a second position on the first track, a video frame in a first frame group is decoded by a first decoder, a video frame in a second frame group is decoded by a second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress indicated by the first control at the second position is within the progress interval indicated by the first frame group;

[0358] When the first control is moved from the second position to the third position on the first trajectory, the video frames in the third frame group are decoded by the first decoder, and the decoded third video frames in the second frame group are displayed in the first area, wherein the playback progress represented by the first control at the third position is within the progress interval represented by the second frame group, and the first frame group, the second frame group and the third frame group are three temporally consecutive video frame groups in the first video.

[0359] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0360] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed on an electronic device, the steps in the above-mentioned method embodiments can be implemented.

[0361] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device or a wireless router, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0362] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0363] The embodiment of the present application further provides a chip, the chip includes a processor, the processor is coupled to a memory, and the processor calls a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip can be a single chip or a chip module composed of multiple chips.

[0364] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0365] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0366] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A video playback method, characterized in that: Applied to electronic equipment, the method comprises: Displaying a first interface, the first interface including a first area and a first control, the first area including a first video frame of a first video; Receiving a sliding operation on the first interface; In response to the sliding operation, moving the first control from a first position along a first track; During the process of moving the first control from the first position along the first track: When the first control is moved from the first position to a second position on the first track, a video frame in a first frame group is decoded by a first decoder, a video frame in a second frame group is decoded by a second decoder, and the decoded second video frame in the first frame group is displayed in the first area, wherein the playback progress indicated by the first control at the second position is within the progress interval indicated by the first frame group; When the first control is moved from the second position to the third position on the first trajectory, the video frames in the third frame group are decoded by the first decoder, and the decoded third video frames in the second frame group are displayed in the first area, wherein the playback progress represented by the first control at the third position is within the progress interval represented by the second frame group, and the first frame group, the second frame group and the third frame group are three temporally consecutive video frame groups in the first video.

2. The method according to claim 1, characterized in that During the process of moving the first control from the first position along the first track, the method further includes: When the first control is moved from the third position to the fourth position on the first trajectory, the video frames in the fourth frame group are decoded by the second decoder, and the decoded fourth video frames in the third frame group are displayed in the first area, wherein the playback progress represented by the first control when at the fourth position is within the progress interval represented by the third frame group, and the first frame group, the second frame group, the third frame group and the fourth frame group are four temporally continuous video frame groups.

3. The method according to claim 1 or 2, characterized in that Before decoding the video frames in the first frame group by the first decoder, the method further includes: Determining the playback progress of the first video as a first moment according to the first contact point position of the sliding operation; Parsing the first video, determining that a video frame group including a video frame corresponding to the first moment is the first frame group; Before decoding the video frames in the second frame group by the second decoder, the method further includes: Determining that the playback mode of the video frame corresponding to the first moment is forward playback; When the playback mode of the video frame corresponding to the first moment is forward playback, the next video frame group located in the first frame group in the first video is determined to be the second frame group.

4. The method according to claim 3, characterized in that After determining that the playback progress of the first video is a first moment according to the first contact point position of the sliding operation, the method further includes: Determine that the position of the first control when the playback progress of the first video is the first moment is the second position; The first control is moved along the first trajectory to the second position.

5. The method according to claim 3 or 4, characterized in that Before decoding the video frames in the third frame group by the first decoder, the method further includes: Determining the playback progress of the first video as a second moment according to the position of the second contact point of the sliding operation; Parsing the first video, determining that a video frame group including a video frame corresponding to the second moment is the second frame group; Determining that the second frame group has been assigned to the second decoder for decoding; Determining that the playback mode of the video frame corresponding to the second moment is forward playback; When the second frame group has been assigned to the second decoder for decoding and the playback mode of the video frame corresponding to the second moment is forward playback, the next song video frame group in the first video located in the second frame group is determined to be the third frame group.

6. The method according to claim 5, characterized in that After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes: Determine that the position of the first control when the playback progress of the first video is the second moment is the third position; The first control is moved along the first trajectory to the third position.

7. The method according to claim 5 or 6, characterized in that The determining that the playback mode of the video frame corresponding to the second moment is forward playback includes: Determining that the second moment is greater than the first moment; Correspondingly, when it is determined that the second moment is less than the first moment, it is determined that the playback mode of the video frame corresponding to the second moment is reverse playback.

8. The method according to any one of claims 5 to 7, characterized in that After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes: Calculate an absolute value of a first difference between the second moment and the first moment; It is determined that the absolute value of the first difference is greater than the first duration.

9. The method according to claim 8, characterized in that After calculating the absolute value of the first difference between the second moment and the first moment, the method further includes: If the absolute value of the first difference is not greater than the first duration, the video frame group where the second moment is located is no longer determined, and the video frame group next to the video frame group where the second moment is located is no longer determined.

10. The method according to any one of claims 5 to 9, characterized in that After determining that the video frame group to which the video frame corresponding to the first moment belongs is the first frame group, the method further includes: Recording the timestamp of the first video frame and the timestamp of the last video frame in the first frame group; After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, and before parsing the first video and determining that the video frame group to which the video frame corresponding to the second moment belongs is the second frame group, the method further includes: It is determined that the second moment is not within a time range between a timestamp of a first video frame and a timestamp of a last video frame in the first frame group.

11. The method according to claim 10, characterized in that After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method includes: If the second moment is within the range of the timestamp of the first video frame and the timestamp of the last video frame in the first frame group, the decoded video frame corresponding to the second moment is obtained from the first decoder that decodes the first frame group, and the decoded video frame corresponding to the second moment is displayed in the first area.

12. The method according to any one of claims 5 to 11, characterized in that After determining that the playback progress of the first video is a second moment according to the second contact point position of the sliding operation, the method further includes: Calculate an absolute value of a first difference between the second moment and the first moment; When the absolute value of the first difference is greater than the second duration, setting the first decoder and the second decoder not to decode non-key frames in the video frame group; When the absolute value of the first difference is not greater than the second duration, the first decoder and the second decoder are set to decode the video frames in the video frame group according to a decoding order.

13. The method according to any one of claims 3 to 12, characterized in that: Decoding the video frames in the first frame group by the first decoder comprises: decoding, by the first decoder, a plurality of video frames in the first frame group in a decoding order of the plurality of video frames in the first frame group; Decoding one video frame in the first frame group by the first decoder, and storing the decoded video frame in a first cache space; The video frame corresponding to the first moment includes: a video frame in the first cache space whose timestamp difference with the first moment is less than or equal to a first value, wherein the first value is half of the timestamp difference between two adjacent video frames; Or, a video frame having the smallest difference between a timestamp and the first moment among the decoded video frames cached in the first cache space; Alternatively, the key frames have been cached in the first cache space.

14. The method according to claim 13, characterized in that Before displaying the decoded second video frame in the first frame group in the first area, the method further includes: Determining that a decoder for decoding the first frame group is a first decoder; A video frame corresponding to the first moment is obtained from the first decoder, where the video frame corresponding to the first moment is the second video frame.

15. The method according to claim 14, characterized in that The determining that the decoder for decoding the first frame group is a first decoder comprises: Calculate a second difference between the first moment and the timestamp of the first video frame cached in the first cache space corresponding to the first decoder; When the second difference is within a first time range, it is determined that the decoder for decoding the first frame group is a first decoder.

16. The method according to claim 15, characterized in that After calculating the second difference between the first moment and the timestamp of the first video frame cached in the first cache space corresponding to the first decoder, the method further includes: If the second difference is not within the first time range, the third difference between the first time point and the timestamp of the first video frame cached in the second cache space corresponding to the second decoder is calculated to determine whether the decoder decoding the first frame group is the second decoder.

17. An electronic device, characterized in that: It comprises one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer programs, and when the one or more processors execute the computer programs, the electronic device executes the method as described in any one of claims 1-16.

18. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-16.

19. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 16.

Images