Video playback optimization method and system based on HLS protocol

By saving and using the end timestamp in the HLS protocol, the problems of live broadcast delay and recreating the player delay are solved, and a low-latency video playback experience is achieved.

CN113630657BActive Publication Date: 2024-11-01UNIONMANTECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110888274.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-03
Publication Date
2024-11-01
Estimated Expiration
2041-08-03

AI Technical Summary

Technical Problem

The HLS protocol cannot live broadcast delays less than 10 seconds, resulting in a long live broadcast delay, and may increase the delay when recreating the player, affecting the playback effect.

Method used

Save the current time node as the end time stamp when the playback is over, and find the corresponding playback time node based on the timestamp when recreating the player to avoid the increase in delay when recreating the player.

Benefits of technology

It effectively avoids the increase in playback delay, optimizes the video playback effect, and ensures a low-latency live broadcast experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113630657B_ABST
    Figure CN113630657B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for optimizing video playback based on the HLS protocol, the method comprising: creating a player, querying whether there is an end timestamp; if there is an end timestamp, the player searches for a playback time node corresponding to the end timestamp according to the end timestamp for playback; wherein the end timestamp is a timestamp recorded by the player created last time at the end of video playback; the playback time node corresponding to the end timestamp is a playback time node acquired and recorded as a timestamp by the player created last time during video playback. By saving the current time node as the end timestamp at the end of playback, the corresponding playback time node is found for playback according to the saved end timestamp when the player is recreated, thereby avoiding the increase of playback delay and optimizing the playback effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of video playback, and in particular to a video playback optimization method based on the HLS protocol and a video playback optimization system based on the HLS protocol. Background Art

[0002] HLS (HTTP Live Streaming) is an HTTP-based streaming media network transmission protocol proposed by Apple. This protocol is widely used for live broadcast and on-demand playback in today's network environment. Although the HLS protocol has many advantages, the delay time of the HLS protocol live broadcast cannot be reduced to less than 10 seconds. Compared with the RTMP protocol (Real Time Messaging Protocol), which can have a minimum delay of about 3 or 4 seconds, the HLS protocol has a longer delay. The delay of HLS is determined by the technical characteristics of the protocol itself. Using this protocol itself cannot avoid a 10-second delay or error in live broadcast.

[0003] The key to data transmission in the HLS protocol is the index file. The actual file name is a file with the suffix ".m3u8". This file will be requested during playback, but the fragmentation mechanism of the HLS protocol will divide the entire bitstream into fragments. Each fragment is usually 10 seconds long. The client plays the bitstream by downloading fragments. Therefore, it is necessary to record the bitstream for 10 seconds before a fragment is formed and data can be transmitted. This results in a 10-second delay in the HLS protocol. If there is a delay caused by the network or other factors, the delay will increase. Summary of the invention

[0004] The purpose of an embodiment of the present invention is to provide a video playback optimization method and system based on the HLS protocol. The method saves the current time node as the end timestamp when the playback ends, and finds the corresponding playback time node for playback according to the saved end timestamp when recreating the player, thereby avoiding the increase of playback delay and optimizing the playback effect.

[0005] In order to achieve the above object, an embodiment of the present invention provides a video playback optimization method based on the HLS protocol, which is executed on a client, and the method includes:

[0006] Create a player and check whether the end timestamp exists;

[0007] If there is an end timestamp, the player searches for a play time node corresponding to the end timestamp according to the end timestamp and plays the video;

[0008] Among them, the end timestamp is the timestamp recorded by the last created player at the end of the video playback; the playback time node corresponding to the end timestamp is the playback time node obtained and recorded as the timestamp by the last created player during the video playback process.

[0009] Optionally, the video playback end time is the time when the client exits the video playback.

[0010] Optionally, the client sends an end message to the server at the time when the player created last exits the playback, and the end message includes a timestamp of the time when the client exits the playback;

[0011] The server obtains the timestamp of the time when the client exits the playback from the received end message and stores it as the end timestamp; the server sends the end timestamp to the server when creating the player. The server saves the end timestamp, and the server sends the end timestamp to the client when recreating the player. The client finds the corresponding playback time node according to the end timestamp to play, which can effectively avoid the increase in delay caused by recreating the player when playing to the middle of the fragment.

[0012] Optionally, the client stores the timestamp of the moment when the client exits the video playback as the end timestamp. By saving the end timestamp on the client, when creating a player, if the existence of the end timestamp is found, the corresponding playback time node is found according to the end timestamp for playback, which can effectively avoid the increase in delay caused by re-creating the player when playing to the middle of the fragment.

[0013] Optionally, the step of creating a player and querying whether an end timestamp exists includes:

[0014] If the end timestamp does not exist, the server only responds to the client's request to create a player;

[0015] The player requests the current segment again and starts playing from the segment header.

[0016] Optionally, the end timestamp only stores a preset time.

[0017] Optionally, during live video broadcast, the preset time is 9-20 seconds; during video on demand, the preset time is customized according to the performance of the server. Setting the preset time to a shorter time can save server resources, and can also meet the storage requirements for the end time stamp of the action process of recreating the player such as exiting the full screen to a small video window for playback.

[0018] A second aspect of the present invention provides a video playback optimization system based on the HLS protocol, the system comprising:

[0019] A timestamp recording module is used to obtain the playback time node and record it as a timestamp;

[0020] An end timestamp storage module, used to store the timestamp of the moment when the client exits the video playback as the end timestamp;

[0021] The player is used to query whether there is an end timestamp when it is created, and to find the playback time node corresponding to the end timestamp according to the end timestamp for playback. The system sets an end timestamp storage module to save the current time node as the end timestamp when playback ends. When the player is recreated, it can find the corresponding playback time node according to the saved end timestamp for playback, thereby avoiding the increase of playback delay and optimizing the playback effect.

[0022] Optionally, the end timestamp storage module includes: a client end timestamp storage module and / or a server end timestamp storage module; the client end timestamp storage module is set on the client, and is used to store the timestamp of the moment when the client exits the video playback as the end timestamp; the server end timestamp storage module is set on the server, and is used to obtain the timestamp of the moment when the client exits the playback from the received end message and store it as the end timestamp. The end timestamp storage module can be set on the client or on the server to meet different user needs.

[0023] The present invention also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the video playback optimization method based on the HLS protocol described in this application.

[0024] Through the above technical scheme, the present invention provides a video playback optimization method based on the HLS protocol on the one hand. The method saves the current time node as the end timestamp at the end of playback, and finds the corresponding playback time node for playback according to the saved end timestamp when recreating the player, thereby avoiding the increase of playback delay and optimizing the playback effect.

[0025] On the other hand, the present invention provides a video playback optimization system based on the HLS protocol. The system is provided with an end timestamp storage module for saving the current time node as the end timestamp when the playback ends. When the player is recreated, it can find the corresponding playback time node according to the saved end timestamp for playback, thereby avoiding the increase of playback delay and optimizing the playback effect.

[0026] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0028] Figure 1 It is a flow chart of a video playback optimization method based on the HLS protocol provided by an embodiment of the present invention;

[0029] Figure 2 It is a flow chart of a video playback optimization method based on the HLS protocol provided by another embodiment of the present invention;

[0030] Figure 3 Implementation of the HLS protocol-based video playback optimization system block diagram provided by the present invention. DETAILED DESCRIPTION

[0031] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0032] In actual HLS live broadcast, except for the 10s delay at the start of the broadcast, in subsequent use, if the network is not disconnected, the HTTP connection is maintained, the network is unobstructed, and there is no problem with the code stream transmission itself, the delay will not increase in theory. However, there is a situation where if the client is playing the current segment (not played for 10s), and recreates the player within 10s (the actual scenario is to exit the full screen to play in a small screen window), the player will re-request the current segment and re-play from 0s, causing the delay to increase. For example, if the current segment is played for 5s, and the same source is requested again to start playing, then the playback point will be the picture 5s ago. A total of 15s of delay.

[0033] Similarly, in the process of using HLS on-demand, although there is no 10s delay in the start stage, because in HLS on-demand, the server records the order of the currently downloaded segments, if you quit playing before the segment ends within 10s, and then restart playing, there is a problem of replaying the current segment. That is, when playing to 5s, quit playing again, then the playback point will also be 0s of the segment, not 5s.

[0034] This shortcoming is generally due to the inability to record the client's playback time node, resulting in a playback error within 10 seconds.

[0035] In order to solve the above problems, the present invention provides a video playback optimization method based on the HLS protocol, the method comprising:

[0036] Create a player and check whether the end timestamp exists;

[0037] If there is an end timestamp, the player searches for a play time node corresponding to the end timestamp according to the end timestamp and plays the video;

[0038] Among them, the end timestamp is the timestamp recorded by the last created player at the end of the video playback; the playback time node corresponding to the end timestamp is the playback time node obtained and recorded as the timestamp by the last created player during the video playback process.

[0039] Embodiment 1

[0040] like Figure 1 This is a flow chart of a video playback optimization method based on the HLS protocol provided by an embodiment of the present invention. Figure 1 As shown, the video playback optimization method includes:

[0041] The player created last time obtains the playback time node during video playback and records it as a timestamp; obtaining the playback time node requires the player to cooperate and obtain the current playback time node and fill it into the timestamp timestamp.

[0042] The client exits the video playback, and saves the timestamp of the moment when the client exits the video playback as the end timestamp.

[0043] The client recreates the player and checks whether the end timestamp exists;

[0044] If there is an end timestamp, the player searches for a play time node corresponding to the end timestamp according to the end timestamp and plays the video;

[0045] If the end timestamp does not exist, the player re-requests the current segment and starts playing from the segment header. The end timestamp is saved by the client. When the player is re-created, if the end timestamp is found, the corresponding playback time node is found according to the end timestamp for playback. This can effectively avoid the increase in delay caused by re-creating the player when playing to the middle of the segment. In other words, if the client is playing the current segment, for example, 5 seconds into the current segment, and then exits from full screen to a small video window, the client will store the 5s timestamp of the current segment as the end timestamp. When the client re-creates the player, the player will query whether a timestamp is stored after re-requesting the current segment. If so, it will re-request the same source and find 5s into the current segment for playback. The delay is still close to 10s, and will not become 15s.

[0046] In this embodiment, the end timestamp only stores a preset time.

[0047] In this embodiment, during the live video broadcast, the preset time is 9-20s; during the video on demand, the preset time is customized according to the performance of the server. Preferably, it is 10s. Setting the preset time to 9-20s can save server resources, and at the same time, it can also meet the storage requirements of the end timestamp in the process of recreating the player such as exiting the full screen to the small screen window for playback. The end timestamp is only used in the switching process between the full screen and the small screen window. This switching process often does not need to last for a long time. Therefore, setting the storage time of the end timestamp to 9-20s can quickly release storage space on the one hand, and on the other hand, if the switching process is delayed for too long, for the live broadcast, it has already proceeded to the next segment, and the next segment can be played to fit the progress of the live broadcast; for on-demand, it is not necessary to fit the progress of the live broadcast, and the delay of repeated playback is not critical. Short-term repetition can help users recall the previous plot.

[0048] When recreating the player, if the time interval between the last end of playback and the re-creation of the player exceeds the preset time, the stored timestamp will be automatically deleted to save server resources for system operation. At this time, there is no end timestamp in the client, and the player re-requests the current segment and starts playing from the segment header.

[0049] Embodiment 2

[0050] Figure 2 It is a flow chart of a video playback optimization method based on the HLS protocol provided by another embodiment of the present invention. Figure 2 As shown, the video playback optimization method includes:

[0051] The player created last time obtains the playback time node during video playback and records it as a timestamp; obtaining the playback time node requires the player to cooperate and obtain the current playback time node and fill it into the timestamp timestamp.

[0052] The client exits the video playback and sends an end message to the server at the time when the player exits the playback created last time. The end message includes the timestamp of the time when the client exits the playback;

[0053] The server obtains the timestamp of the time when the client exits playing from the received end message and stores it as the end timestamp;

[0054] The client recreates the player;

[0055] The server sends the end timestamp to the server;

[0056] The client queries whether an end timestamp exists;

[0057] If there is an end timestamp, the player searches for a play time node corresponding to the end timestamp according to the end timestamp and plays the video;

[0058] If the end timestamp does not exist, the player re-requests the current segment and starts playing from the segment header.

[0059] The server saves the end timestamp and sends it to the client when recreating the player. The client finds the corresponding playback time node based on the end timestamp for playback, which can effectively avoid the increase in delay caused by recreating the player when playing to the middle of the fragment.

[0060] That is to say, if the client is playing the current segment, for example, 5 seconds into the current segment, and then exits from full screen to a small video window, the client will send an end message to the server when exiting playback. The end message will carry the timestamp 5s when playback ended. The server receives the end message, obtains the timestamp in the end message, and then stores it as the end timestamp. When the client requests to create a player again, the server sends the end timestamp to the client, creates a player request accordingly, and executes the action of creating a player. When the player requests the current segment again, it will query whether a timestamp is stored. If so, it will request the same source again, find the 5s point of the current segment for playback, and the delay will still be close to 10s, and will not become 15s.

[0061] In this embodiment, the end timestamp only stores a preset time.

[0062] In this embodiment, during the live video broadcast, the preset time is 9-20s; during the video on demand, the preset time is customized according to the performance of the server. Preferably, it is 10s. Setting the preset time to 9-20s can save server resources, and at the same time, it can also meet the storage requirements for the end timestamp of the action process of recreating the player such as exiting the full screen to the small screen window for playback. The end timestamp is only used in the switching process between the full screen and the small screen window. This switching process often does not need to last for a long time. Therefore, setting the storage time of the end timestamp to 9-20s can quickly release storage space on the one hand, and on the other hand, if the switching process is delayed for too long, for the live broadcast, it has already proceeded to the next segment, and the next segment can be played to fit the progress of the live broadcast; for on-demand, it is not necessary to fit the progress of the live broadcast, and the delay of repeated playback is not critical. Short-term repetition can help users recall the previous plot.

[0063] When recreating the player, if the time interval between the last end of playback and the re-creation of the player exceeds the preset time, the stored timestamp will be automatically deleted to save server resources for system operation. At this time, there is no end timestamp in the server. The server only responds to the client's request to create a player. The player re-requests the current segment and starts playing from the segment header.

[0064] To further optimize the playback, the method further includes:

[0065] Adjust the cache size of the shard cache according to the real-time memory changes of the system. Adjust the cache size according to the memory size to ensure that the normal operation of the system is not affected.

[0066] In some embodiments, adjusting the cache amount of the shard cache according to the real-time memory change of the system includes:

[0067] Adjust the number of slices in the slice cache according to the remaining memory size of the system. For systems with playback memory, adjust the amount of slice cache swaps according to the size of the playback memory.

[0068] In some embodiments, such as the HiSilicon platform system, memory information is obtained through commands (cat / proc / memdia-mem; cat / proc / meminfo; dumpsys meninfo; top; ps), and the information is filtered to obtain the current remaining memory size. Among these information, memdia-men is MMZ (media memory zone) information, which is the main media playback memory size. If the remaining space is insufficient, it cannot be played. The HiSilicon platform divides the memory into two parts: os (operating system) memory and MMZ memory. When the MMZ memory is initially set large, the os memory is small and is a fixed size at startup. Therefore, the MMZ information is obtained in the above text. If it is other platforms, the memory is not distinguished, and the remaining memory of the system can be directly obtained.

[0069] In some embodiments, when the remaining memory is less than 10%, one shard is cached; when the remaining memory is less than 30% and greater than 10%, three shards are cached; when the remaining memory is greater than 30%, five shards are cached. It should be noted that there is an upper limit on the cached shards to avoid affecting the space required by users and other processes.

[0070] In some other embodiments, adjusting the cache amount of the shard cache according to the real-time memory change of the system includes:

[0071] Change the buffer size according to the remaining system memory size, calculate the fragment size according to the decoded bitrate frame number and resolution, divide the buffer size by each fragment size and round it up to get the number of fragments that should be cached (some m3u8 files provide fragment sizes). For systems that set buffers, change the buffer size according to memory changes, and then calculate the number of fragments that can be cached.

[0072] The above method can achieve the purpose of real-time change of shard cache. The advantage is that when the memory is insufficient, the system operation is guaranteed to be normal and not stuck, and the video can be played normally without screen stuck. When the memory is sufficient, more data can be buffered to cope with more complex network environments, and the user experience is better. When the user operates the seek operation, there is no need to re-buffer and wait.

[0073] The second aspect of the present invention provides a video playback optimization system based on the HLS protocol, such as Figure 3 As shown, the system comprises:

[0074] A timestamp recording module is used to obtain the playback time node and record it as a timestamp;

[0075] An end timestamp storage module, used to store the timestamp of the moment when the client exits the video playback as the end timestamp;

[0076] The player is used to query whether there is an end timestamp when it is created, and to find the playback time node corresponding to the end timestamp according to the end timestamp for playback. The system sets an end timestamp storage module to save the current time node as the end timestamp when playback ends. When the player is recreated, it can find the corresponding playback time node according to the saved end timestamp for playback, thereby avoiding the increase of playback delay and optimizing the playback effect.

[0077] Optionally, the end timestamp storage module includes: a client end timestamp storage module and / or a server end timestamp storage module; the client end timestamp storage module is set on the client, and is used to store the timestamp of the moment when the client exits the video playback as the end timestamp, and the server end timestamp storage module is set on the server, and is used to obtain the timestamp of the moment when the client exits the playback from the received end message and store it as the end timestamp. The end timestamp storage module can be set on the client or on the server to meet different user needs.

[0078] Optionally, the system further includes: a cache capacity adjustment module, configured to adjust the cache capacity of the shard cache according to real-time memory changes of the system.

[0079] The present invention also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the video playback optimization method based on the HLS protocol described in this application.

[0080] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0081] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0082] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0083] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A video playback optimization method based on the HLS protocol, characterized in that: The method comprises: Create a player and check whether the end timestamp exists; If there is an end timestamp, the player searches for a play time node corresponding to the end timestamp according to the end timestamp and plays the video; The end timestamp is the timestamp recorded by the last created player at the end of the video playback; the playback time node corresponding to the end timestamp is the playback time node obtained and recorded as the timestamp by the last created player during the video playback process; The video playback end time is the time when the client exits the video playback; The client sends an end message to the server at the time when the player created last exits the playback, and the end message includes the timestamp of the time when the client exits the playback; The server obtains the timestamp of the time when the client exits the playback from the received end message and stores it as the end timestamp; the server sends the end timestamp to the client when creating the player; When recreating a player, if the time interval between the last end of playback and the re-creation of the player exceeds the preset time, the stored timestamp will be automatically deleted.

2. The video playback optimization method based on the HLS protocol according to claim 1 is characterized in that: The client stores the timestamp of the moment when the client exits the video playback as the end timestamp.

3. The video playback optimization method based on the HLS protocol according to claim 1 is characterized in that: The step of creating a player and querying whether an end timestamp exists includes: If the end timestamp does not exist, the server only responds to the client's request to create a player; The player requests the current segment again and starts playing from the segment header.

4. The video playback optimization method based on the HLS protocol according to claim 1 is characterized in that: During live video broadcast, the preset time is 9-20s; during video on demand, the preset time is customized according to the performance of the server.

5. A video playback optimization system based on the HLS protocol, characterized in that: The system comprises: A timestamp recording module is used to obtain the playback time node and record it as a timestamp; An end timestamp storage module, used to store the timestamp of the moment when the client exits the video playback as the end timestamp; The player is used to query whether there is an end timestamp when creating it, and find the play time node corresponding to the end timestamp according to the end timestamp to play it; The end timestamp storage module comprises: A client end timestamp storage module and / or a server end timestamp storage module; the client end timestamp storage module is arranged at the client, and is used to store the timestamp of the moment when the client exits the video playback as the end timestamp; the server end timestamp storage module is arranged at the server, and is used to obtain the timestamp of the moment when the client exits the playback from the received end message and store it as the end timestamp; The video playback end time is the time when the client exits the video playback; The client sends an end message to the server at the time when the player created last exits the playback, and the end message includes the timestamp of the time when the client exits the playback; The server obtains the timestamp of the time when the client exits the playback from the received end message and stores it as the end timestamp; the server sends the end timestamp to the client when creating the player; When recreating a player, if the time interval between the last end of playback and the re-creation of the player exceeds the preset time, the stored timestamp will be automatically deleted.

6. A machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the HLS protocol-based video playback optimization method described in any one of claims 1-4 of the present application.