A method for generating hls information

Abstract

The application relates to an HLS information generation method, belonging to the technical field of HLS streaming media, which comprises a processing method of an HLS server and a processing method of an HLS player when m3u8 acquisition is abnormal, data is lost or data is abnormal, and comprises the following steps: slicing streaming media audio and video data; acquiring and encapsulating related audio and video information of the slices into an MEPEG-TS format; encoding the related audio and video information into audio and video metadata and storing the audio and video metadata into an MEPEG-TS extension field; generating m3u8 information of HLS according to multiple slice-related audio and video information; and encoding m3u8 information of previous and subsequent slices to generate current slice link table data and storing the current slice link table data into the slice MEPEG-TS extension field. The application improves the fault tolerance of HLS when m3u8 is lost, quickly recovers lost m3u8, improves fault recovery efficiency and user experience, has strong universality and compatibility, and can be widely applied to various scenes to guarantee normal playing of HLS streaming media.

Description

Technical Field

[0001] This invention relates to the field of HLS streaming media technology, and in particular, to an HLS information generation method. Background Technology

[0002] HLS (HTTP Live Streaming) is a technology for streaming media transmission. The m3u8 file is an index file that lists the URLs of individual video segments and related information.

[0003] However, if the m3u8 file is lost in HLS, it will have a significant adverse impact on the normal transmission and playback of HLS streaming media, mainly including the following:

[0004] 1. Unable to play properly: The m3u8 file is an important component of HLS streaming media. It contains the URLs and related information of each video segment. Without the m3u8 file, the list and location information of the video segments cannot be obtained, thus preventing proper playback of the streaming media.

[0005] 2. Playback Interruption: HLS streaming media needs to continuously retrieve video segments from the server for playback. If the m3u8 file is missing, the latest list and position information of video segments cannot be obtained, causing playback to be interrupted.

[0006] 3. Inability to broadcast live: HLS is typically used for live streaming. The index information in the m3u8 file is updated in real time so that viewers can watch the latest live content. If the m3u8 file is lost, the latest live content cannot be obtained, and viewers will not be able to watch the latest live stream.

[0007] Therefore, ensuring the normal transmission of m3u8 files in HLS is crucial for the proper playback of HLS streaming media. Currently, there is an urgent need to develop a method that can promptly repair or regenerate m3u8 files if they are found to be missing in HLS, in order to solve the current difficulties and pain points caused by missing m3u8 files preventing normal playback in HLS. Summary of the Invention

[0008] Therefore, the purpose of this invention is to propose an HLS information generation method that can restore HLS streaming media playback through slice files when the m3u8 of HLS is abnormal. By encoding and storing the slice audio and video metadata and key m3u8 information into extended fields in the MEPEG-TS format, the m3u8 information can be obtained again by parsing the preserved slices when the m3u8 is abnormal, thereby realizing the restoration of HLS streaming media playback and effectively ensuring the normal playback of HLS streaming media.

[0009] This invention provides an HLS information generation method, including: an HLS server processing method and an HLS playback terminal processing method when m3u8 acquisition is abnormal, data is lost, or data is abnormal;

[0010] The HLS server-side processing method includes the following steps:

[0011] S11. Slice the streaming audio and video data into segments of fixed duration to obtain multiple audio and video segments;

[0012] S12. Obtain audio and video information related to the audio and video slice;

[0013] The audio and video information associated with each audio / video slice file is extracted and recorded. This information mainly includes video resolution, bitrate, frame rate, video keyframe index, audio sampling rate, and audio / video encoder information. This information will be used to generate relevant data in subsequent steps. This embodiment of the invention uses the FFprobe tool to parse and obtain audio and video related information. FFprobe is commonly used to analyze audio and video container formats, audio and video stream information, audio and video packets, and audio and video frames. The FFprobe tool can provide significant assistance when performing transcoding and fault analysis on audio and video.

[0014] S13. Encapsulate the audio and video data of the audio and video slices into MEPEG-TS format;

[0015] To ensure the stability and compatibility of audio and video data during transmission and storage, this invention encapsulates each audio and video segment file in MPEG-TS (MPEG Transport Stream) format. MPEG-TS is a widely used encapsulation format in digital television, satellite broadcasting, and online streaming media transmission, offering good transmission efficiency and compatibility. Preferably, the third-party open-source tool FFmpeg can be used for MPEG-TS format encapsulation.

[0016] S14. Encode the relevant audio and video information of the audio and video slice into audio and video metadata of the audio and video slice according to the set format, and store the audio and video metadata in the MEPEG-TS extended field of the audio and video slice.

[0017] S15. Generate HLS m3u8 information based on the audio and video information related to multiple audio and video slices; the form of generating HLS m3u8 information includes: generating an m3u8 playlist according to the RFC 8216 standard document on HLS with a set window size of the m3u8 playlist.

[0018] HLS (HTTP Live Streaming) is an HTTP-based adaptive bitrate video streaming protocol. Introduced by Apple in 2009, HLS describes a set of tools and processes for transmitting video and audio over the internet. HLS divides video into multiple small segments and organizes these segments into a playlist as m3u8 files. The player can dynamically select the most suitable segment to play based on network conditions and device performance to achieve a smooth video viewing experience.

[0019] M3U8 files are an HTTP-based streaming media file format primarily used to describe URLs containing multiple audio and video data segments. M3U8 divides audio and video data into multiple segments, allowing users to download and play them as needed. Therefore, M3U8 is more suitable for scenarios that do not require real-time transmission, such as video-on-demand and short videos. The characteristics of M3U8 files are flexibility and scalability; they can be optimized for different network environments and devices, and support encrypted playback. M3U8 files are commonly used for transmitting and playing audio and video content over the network. Unlike traditional download-and-play methods, M3U8 files can be played directly on browsers or mobile devices.

[0020] S16. Encode the m3u8 information of the preceding and following audio / video slices to generate the current slice linked list data, and store the slice linked list data in the MEPEG-TS extended field of the preceding and following audio / video slices.

[0021] Specifically, the information of one or more slices before and after the current slice is used as linked list data. The MEPEG-TS extended fields can still be stored in a dedicated segment. At the same time, the dedicated segment table_id of the linked list data needs to be distinguished from the table_id of the dedicated segment of audio and video metadata.

[0022] Furthermore, the handling method for m3u8 acquisition errors, data loss, and data anomalies in the HLS playback terminal includes the following steps:

[0023] S21. Parse the MEPEG-TS extended fields of the acquired audio and video slices;

[0024] S22. Extract the slice list data from the MEPEG-TS extended field to determine the associated slices of the playlist;

[0025] Specifically, the information of the associated slices is parsed according to the dedicated segment syntax of the slice linked list data. Based on the number of m3u8 associated slices that need to be recovered, the linked list data in the slices at the head or tail of the linked list can be parsed again, and their linked lists can be merged into the current linked list until the number of slices in the overall associated linked list matches the number of m3u8s that need to be recovered.

[0026] S23. Obtain the audio and video metadata of the associated slice;

[0027] Extract the dedicated segment data of the audio and video metadata of the audio and video slices, and parse the audio and video information of the audio and video slices according to the dedicated segment syntax of the audio and video metadata.

[0028] S24. Based on the playlist and the audio and video metadata of each associated slice, generate the relevant tags for the required m3u8 file until the complete m3u8 file is restored.

[0029] Furthermore, the method for generating HLS m3u8 information based on audio and video information related to multiple audio and video slices in step S15 includes the following steps:

[0030] S151. Based on the specified m3u8 version, generate the basic tags of m3u8, wherein the basic tags of m3u8 include: EXTM3U and EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8;

[0031] S152. Generate m3u8 media segment tags based on the audio and video information related to the audio and video segments; the m3u8 media segment tags include: EXTINF, EXT-X-BYTERANGE, and EXT-X-PROGRAM-DATE-TIME; wherein, the EXTINF tag indicates the duration of the media segment specified in the URL following it; the duration information of the audio and video segments can be filled into this field; the URL following EXTINF is generated based on the specific storage location of the segments in the HLS service, and the number of segments contained in the playlist is set according to the window size. For example, if the window size provided by the live HLS service is 5, then the m3u8 playlist will contain the 5 segments with EXTINF tags in the most recent time.

[0032] The EXT-X-BYTERANGE tag identifies the size of the relevant data range of the media slice specified in the URL that follows; this tag can be added when playing an m3u8 playlist and it is necessary to indicate the position of video keyframes in the slice. This tag can be generated using the video keyframe index information obtained in step two.

[0033] The EXT-X-PROGRAM-DATE-TIME label uses an absolute date / time to indicate the sampling time of the first sample fragment; it can be generated using the slice timestamp recorded in step one.

[0034] S153. Generate playlist tags, including: EXT-X-STREAM-INF, EXT-X-TARGETDURATION, and EXT-X-MEDIA-SEQUENCE;

[0035] The EXT-X-STREAM-INF tag represents the attribute information of the media stream, including: BANDWIDTH (bandwidth), RESOLUTION (resolution), and CODECS (encoder); where BANDWIDTH is the audio / video bitrate in the audio / video information related to the audio / video slice; RESOLUTION is the resolution width and height pixel values ​​in the audio / video information related to the audio / video slice; and CODECS is the encoder information in the audio / video information related to the audio / video slice.

[0036] The EXT-X-TARGETDURATION tag indicates the maximum duration (in seconds) of a slice in the playlist. This tag value is generated by selecting the maximum slice duration value from all slice information in the playlist.

[0037] The EXT-X-MEDIA-SEQUENCE tag indicates the sequence number of the first URL segment file in the playlist; each media segment URL has a unique integer sequence number.

[0038] Furthermore, the method for parsing the MPEG-TS extended field of the acquired audio / video slice in step S21 includes the following steps:

[0039] S211. Parse the TS data packets from the MEPEG-TS extension field of the audio and video slices. Each TS data packet is 188 bytes long and contains a header and payload.

[0040] S212. Obtain the PAT (Program Association Table) and PMT (Program Map Table) from the TS data packet, wherein the PAT table contains the PID value of the dedicated segment;

[0041] S213. Identify TS data packets with PID values ​​of dedicated segments, extract and merge the payloads of one or more TS data packets to obtain MEPEG-TS extended field data.

[0042] Furthermore, the method for generating the required m3u8 related tags in step S24 includes the following steps:

[0043] S241, based on the specified m3u8 version, restore the basic tags of m3u8; the basic tags of m3u8 include: EXTM3U, EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8;

[0044] S242. Recover the m3u8 media segment tags based on the audio and video metadata of the audio and video segments; the m3u8 media segment tags include: EXTINF, EXT-X-BYTERANGE, and EXT-X-PROGRAM-DATE-TIME;

[0045] The EXTINF tag identifies the duration of the media segment specified in the subsequent URL; this field can be recovered from the "duration" field in the segment's audio and video metadata. The URL following EXTINF is generated based on the specific segment storage location of the HLS service, and this tag can be recovered from the "segment_info_length" and "segment_info_byte" fields in the segment list data.

[0046] The EXT-X-BYTERANGE tag identifies the size of the data range of the media slice specified in the URL. This tag can be added when the m3u8 playlist needs to indicate the position of video keyframes in the slice. The tag can be recovered from the audio and video metadata of the audio and video slice, specifically the "keyframe_count", "keyframe_position", and "keyframe_size".

[0047] The EXT-X-PROGRAM-DATE-TIME tag uses an absolute date / time to indicate the sampling time of the first sample segment; this tag can be recovered from the "date_time" in the audio / video metadata of the audio / video slice.

[0048] S243. Restore playlist tags, which include:

[0049] The EXT-X-STREAM-INF tag represents the attribute information of the media stream. Its BANDWIDTH content can be recovered from the "video_data_rate" and "audio_sample_rate" of the segment audio and video metadata. Its RESOLUTION content can be recovered from the "height" and "width" of the segment audio and video metadata. Its CODECS content can be recovered from the "codecs_info_length" and "codecs_info_byte" of the segment audio and video metadata.

[0050] The EXT-X-TARGETDURATION tag indicates the maximum duration (in seconds) of a slice in the playlist. This tag is restored by selecting the maximum "duration" value from all the audio and video metadata of the slices in the playlist.

[0051] The EXT-X-MEDIA-SEQUENCE tag indicates the sequence number of the first URL segment file in the playlist. This tag is recovered by selecting the audio / video metadata "sequence" of the first segment of the playlist.

[0052] Furthermore, the packet header in step S211 includes synchronization bytes, PID (Packet Identifier), and TSC (Transport Scrambling Control) information, and the payload includes audio and video data and PSI / SI table data.

[0053] Furthermore, the MPEG-TS extension fields in step S14 are defined by the MPEG-TS standard (MPEG-TS is a standardized audio and video transmission and storage format). The fields used by the MPEG-TS extension include: dedicated segments and descriptors; the dedicated segments are used to transmit user-defined dedicated data, and the descriptors are used to describe the characteristics and parameters of audio and video streams and other data.

[0054] MPEG-TS (MPEG Transport Stream) is a standard data container format used for transmitting and storing audio, video, program, and system information protocol data. It is widely used in digital broadcasting systems such as DVB (Digital Video Broadcasting), ATSC (Advanced Television Systems Committee), and IPTV (Internet Protocol Television). MPEG-TS is defined in MPEG-2 Part 1 and is also known as ISO / IEC Standard 13818-1 or ITU-T Recommendation.

[0055] In MPEG-TS, private sections are a mechanism for transmitting user-defined, proprietary data. In the standard, a private section is defined as a special segment used to transmit user-defined data. These private sections can carry application-specific or vendor-defined additional information, such as specific metadata, additional descriptive information, etc. The use of private sections allows users to add custom data to MPEG-TS to meet specific application needs. This custom data can be transmitted through private sections and parsed and processed at the receiving end. By using private sections, users can flexibly extend the functionality of MPEG-TS to adapt to different application scenarios.

[0056] In MPEG-TS, descriptors are used to describe the characteristics and parameters of audio and video streams and other data. In some cases, custom descriptors can be added to extend the functionality of MPEG-TS, such as adding specific encoding parameters or identifiers.

[0057] The use of custom extended fields should comply with the specifications of the MPEG-TS standard, and compatibility issues may exist between different applications and devices. Therefore, when using custom extended fields, it is necessary to ensure that the relevant devices and applications can correctly parse and process these fields.

[0058] Preferably, this embodiment of the invention uses a dedicated segment in MPEG-TS to store the audio and video metadata of the segments. The dedicated segment consists of an 8-bit `table_id` field, a 1-bit `section_syntax_indicator` field, a 12-bit `section_length` field, and some other fields. The `section_length` field indicates the total length of the dedicated segment. The private data indicator (`private_data_indicator`) in the dedicated segment is used to identify whether the segment contains private data. A value of 1 for the private data indicator indicates that the segment contains private data. The specific format and content of the private data (`private_data_byte`) are user-defined.

[0059] Furthermore, the method for slicing streaming audio and video data into fixed durations in step S11 includes:

[0060] The streaming audio and video data is segmented into multiple segments according to a preset duration of 10 seconds or 20 seconds. Each segment is stored independently as an audio / video slice for subsequent processing. During slicing, information such as slice number, slice start timestamp, and slice duration is recorded for use in generating and using relevant fields in subsequent steps. To ensure compatibility with different streaming audio and video formats, the slicing process can utilize third-party open-source tools such as Libav, Gstreamer, and OpenCV.

[0061] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the HLS information generation method as described above.

[0062] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the HLS information generation method as described above.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] The HLS information generation method provided by this invention improves the fault tolerance of HLS streaming media when m3u8 is lost, lowers the threshold for users to watch videos, and improves the user experience; by parsing the segment file, it realizes the rapid recovery of lost m3u8, improving the efficiency of fault recovery; the relevant information of HLS streaming media is saved and transmitted through the MEPEG-TS extended field, which has strong versatility and compatibility, can be widely used in various scenarios, and effectively ensures the normal playback of HLS streaming media. Attached Figure Description

[0065] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0066] In the attached diagram:

[0067] Figure 1 This is a schematic diagram of streaming media data slicing and encapsulation according to an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of slice extended data according to an embodiment of the present invention;

[0069] Figure 3 This is a syntax diagram of a dedicated MPEG-TS segment in an embodiment of the present invention;

[0070] Figure 4This is a schematic diagram illustrating the restoration of associated slices in an m3u8 playlist according to an embodiment of the present invention;

[0071] Figure 5 This is a flowchart of the HLS server processing method according to an embodiment of the present invention;

[0072] Figure 6 This is a flowchart of the handling method for m3u8 acquisition errors, data loss, and data anomalies in the HLS playback terminal according to an embodiment of the present invention.

[0073] Figure 7 This is a flowchart of a method for generating HLS m3u8 information based on audio and video information related to multiple audio and video slices, according to an embodiment of the present invention.

[0074] Figure 8 This is a flowchart illustrating the method for parsing the MEPEG-TS extended field of an acquired audio / video slice according to an embodiment of the present invention.

[0075] Figure 9 A flowchart illustrating the method for generating the required m3u8 tags in this embodiment of the invention;

[0076] Figure 10 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation

[0077] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and products consistent with some aspects of this disclosure as detailed in the appended claims.

[0078] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0079] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0080] The embodiments of the present invention will be described in further detail below.

[0081] This invention proposes an HLS information generation method to address the current problem that the loss of m3u8 files in HLS streams can negatively impact the normal transmission and playback of HLS media, causing HLS to malfunction.

[0082] Method Implementation Examples

[0083] This invention provides an HLS information generation method, including: an HLS server processing method and an HLS playback terminal processing method when m3u8 acquisition is abnormal, data is lost, or data is abnormal;

[0084] The HLS server-side processing method includes the following steps (see...). Figure 5 As shown):

[0085] S11. Slice the streaming audio and video data into segments of fixed duration to obtain multiple audio and video segments;

[0086] The method for slicing streaming audio and video data into segments of fixed duration includes:

[0087] The streaming audio and video data is segmented into multiple segments according to a preset duration of 10 seconds or 20 seconds. Each segment is stored independently as an audio / video slice for subsequent processing. During slicing, information such as slice number, slice start timestamp, and slice duration is recorded for use in generating and using relevant fields in subsequent steps. To ensure compatibility with different streaming audio and video formats, the slicing process can utilize third-party open-source tools such as Libav, Gstreamer, and OpenCV.

[0088] S12. Obtain audio and video information related to the audio and video slice;

[0089] The audio and video information associated with each audio / video slice file is extracted and recorded. This information mainly includes video resolution, bitrate, frame rate, video keyframe index, audio sampling rate, and audio / video encoder information. This information will be used to generate relevant data in subsequent steps. This embodiment of the invention uses the FFprobe tool to parse and obtain audio and video related information. FFprobe is commonly used to analyze audio and video container formats, audio and video stream information, audio and video packets, and audio and video frames. The FFprobe tool can provide significant assistance when performing transcoding and fault analysis on audio and video.

[0090] S13. Encapsulate the audio and video data of the audio and video slices into MEPEG-TS format;

[0091] To ensure the stability and compatibility of audio and video data during transmission and storage, this embodiment refers to... Figure 1 As shown, each audio / video segment file is encapsulated in MPEG-TS (MPEG Transport Stream) format. MPEG-TS is a widely used encapsulation format for digital television, satellite broadcasting, and online streaming media transmission, offering good transmission efficiency and compatibility. Preferably, the third-party open-source tool FFmpeg can be used for MPEG-TS encapsulation (see [link]). Figure 1 (As shown).

[0092] S14. Encode the relevant audio and video information of the audio and video slice into audio and video metadata of the audio and video slice according to the set format, and store the audio and video metadata in the MEPEG-TS extended field of the audio and video slice.

[0093] The MPEG-TS extension fields are defined by the MPEG-TS standard (MPEG-TS is a standardized audio and video transmission and storage format). The fields used by MPEG-TS extensions include: dedicated segments and descriptors; the dedicated segments are used to transmit user-defined dedicated data, and the descriptors are used to describe the characteristics and parameters of audio and video streams and other data.

[0094] Private Section: In MPEG-TS, a private section is a mechanism for transmitting user-defined, proprietary data. In the standard, a private section is defined as a special segment used to transmit user-defined data. These private sections can carry application-specific or vendor-defined additional information, such as specific metadata, additional descriptive information, etc. The use of private sections allows users to add custom data to MPEG-TS to meet specific application needs. This custom data can be transmitted through the private section and parsed and processed at the receiving end. By using private sections, users can flexibly extend the functionality of MPEG-TS to adapt to different application scenarios (see [link to standard MPEG-TS documentation]). Figure 3 (As shown).

[0095] Descriptor: In MPEG-TS, descriptors are used to describe the characteristics and parameters of audio and video streams and other data. In some cases, custom descriptors can be added to extend the functionality of MPEG-TS, such as adding specific encoding parameters, identifiers, etc.

[0096] The use of custom extended fields should comply with the specifications of the MPEG-TS standard, and compatibility issues may exist between different applications and devices. Therefore, when using custom extended fields, it is necessary to ensure that the relevant devices and applications can correctly parse and process these fields.

[0097] Preferably, this embodiment of the invention uses a dedicated segment in MPEG-TS to store the audio and video metadata of the segments. The dedicated segment consists of an 8-bit `table_id` field, a 1-bit `section_syntax_indicator` field, a 12-bit `section_length` field, and some other fields. The `section_length` field indicates the total length of the dedicated segment. The private data indicator (`private_data_indicator`) in the dedicated segment is used to identify whether the segment contains private data. A value of 1 for the private data indicator indicates that the segment contains private data. The specific format and content of the private data (`private_data_byte`) are user-defined. According to the format standard requirements of dedicated segments, the format of the segment audio and video metadata in MPEG-TS in this embodiment is shown in Table 1:

[0098] Table 1

[0099]

[0100]

[0101] The definitions of each field in Table 1 are as follows:

[0102] sequence: This field identifies the slice sequence number, which is the slice sequence number recorded in step 11. The sequence number value starts from 0 and increments continuously. When it exceeds the maximum value, it can start again.

[0103] date_time: This field identifies the time the slice was generated. It uses the slice timestamp recorded in step one and can be in 10-digit Unix timestamp format or string format.

[0104] duration: This field identifies the duration of the slice, using the slice duration information recorded in step one, in seconds.

[0105] frame_rate: This field identifies the video frame rate of the slice. Using the slice video information obtained in step two, it indicates how many frames of video are in one second.

[0106] video_data_rate: This field identifies the bitrate of the video slice. It uses the video slice information obtained in step two, and the unit is kb / s, which indicates the size of the video data in kb per second.

[0107] audio_sample_rate: This field identifies the bitrate of the slice audio frequency. It uses the slice audio information obtained in step two, and the unit is kb / s, which indicates how many kb of video data per second.

[0108] heigh: This field identifies the resolution height of the video slice, using the video slice information obtained in step two, in pixels, indicating how many pixels the video resolution height is.

[0109] width: This field identifies the resolution width of the video slice. It uses the video slice information obtained in step two, and the unit is pixels, indicating how many pixels the video resolution width is.

[0110] codecs_info_length: This field identifies the length of the media audio / video encoder information, using the sliced ​​audio / video information obtained in step two, in bytes.

[0111] codecs_info_byte: This field identifies the specific content of the media audio and video encoder information. It uses the sliced ​​audio and video information obtained in step two. The content can be represented by a string with a length of codecs_info_length.

[0112] keyframe_count: This field identifies the number of video keyframes within the slice, using the slice video information obtained in step two.

[0113] keyframe_position: This field identifies the offset position of each video keyframe within the slice. It uses the slice video information obtained in step two, and the unit is bytes. The number of this field depends on keyframe_count.

[0114] keyframe_size: This field identifies the size of each video keyframe within the slice. It uses the slice video information obtained in step two, and the unit is bytes. The number of keyframes in this field depends on keyframe_count.

[0115] S15. Generate HLS m3u8 information based on the audio and video information related to multiple audio and video slices; the form of generating HLS m3u8 information includes: generating an m3u8 playlist according to the RFC 8216 standard document on HLS with a set window size of the m3u8 playlist.

[0116] The method for generating HLS m3u8 information based on audio and video information related to multiple audio and video slices is described in [reference needed]. Figure 7 As shown, it includes the following steps:

[0117] S151. Based on the specified m3u8 version, generate the basic tags of m3u8, wherein the basic tags of m3u8 include: EXTM3U and EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8;

[0118] S152. Generate m3u8 media segment tags based on the audio and video information related to the audio and video segments; the m3u8 media segment tags include: EXTINF, EXT-X-BYTERANGE, and EXT-X-PROGRAM-DATE-TIME; wherein, the EXTINF tag indicates the duration of the media segment specified in the URL following it; the duration information of the audio and video segments can be filled into this field; the URL following EXTINF is generated based on the specific storage location of the segments in the HLS service, and the number of segments contained in the playlist is set according to the window size. For example, if the window size provided by the live HLS service is 5, then the m3u8 playlist will contain the 5 segments with EXTINF tags in the most recent time.

[0119] The EXT-X-BYTERANGE tag identifies the size of the relevant data range of the media slice specified in the URL that follows; this tag can be added when playing an m3u8 playlist and it is necessary to indicate the position of video keyframes in the slice. This tag can be generated using the video keyframe index information obtained in step two.

[0120] The EXT-X-PROGRAM-DATE-TIME label uses an absolute date / time to indicate the sampling time of the first sample fragment; it can be generated using the slice timestamp recorded in step one.

[0121] S153. Generate playlist tags, including: EXT-X-STREAM-INF, EXT-X-TARGETDURATION, and EXT-X-MEDIA-SEQUENCE;

[0122] The EXT-X-STREAM-INF tag represents the attribute information of the media stream, including: BANDWIDTH, RESOLUTION, and CODECS; where BANDWIDTH is the audio / video bitrate in the audio / video information related to the audio / video slice; RESOLUTION is the resolution width and height pixel values ​​in the audio / video information related to the audio / video slice; and CODECS is the encoder information in the audio / video information related to the audio / video slice.

[0123] The EXT-X-TARGETDURATION tag indicates the maximum duration (in seconds) of a slice in the playlist. This tag value is generated by selecting the maximum slice duration value from all slice information in the playlist.

[0124] The EXT-X-MEDIA-SEQUENCE tag indicates the sequence number of the first URL segment file in the playlist; each media segment URL has a unique integer sequence number.

[0125] S16. Encode the m3u8 information of the preceding and following audio / video slices to generate the current slice linked list data, and store the slice linked list data in the MEPEG-TS extended field of the preceding and following audio / video slices (see...). Figure 2 (As shown).

[0126] Specifically, information from one or more slices before and after the current slice is used as linked list data. MPEG-TS extended fields can still be stored using dedicated segments, and the dedicated segment's `table_id` needs to be distinguished from the `table_id` of the audio / video metadata's dedicated segment. According to the dedicated segment format standard requirements, the dedicated segment format of the slice linked list data in MPEG-TS in this embodiment is shown in Table 2:

[0127] Table 2

[0128]

[0129]

[0130] The definitions of each field in Table 2 are as follows:

[0131] The `pre_segment_count` field indicates the number of preceding slices in the linked list, i.e., the number of slices preceding the current slice in the linked list. For example, if the current slice number is 10 and `pre_segment_count` is 2, it means that the current slice linked list contains slices with numbers 8 and 9. The preceding slice information in the linked list is sorted in ascending order of slice number.

[0132] The `next_segment_count` field indicates the number of subsequent slices in the linked list, that is, the number of slice information following the current slice in the linked list. For example, if the current slice number is 10 and `next_segment_count` is 2, it means that the current slice linked list contains slice information with numbers 11 and 12. The subsequent slice information in the linked list is sorted according to the slice number in ascending order.

[0133] The segment_info_length field indicates the length of the subsequent slice information content, in bytes.

[0134] The segment_info_byte field identifies the specific content of the segment information. The segment information should ensure that the segment can be located and obtained. For example, it can directly store the segment's URL download address, the segment's file name, etc., and its length is segment_info_length.

[0135] The handling methods for m3u8 acquisition errors, data loss, and data anomalies in the HLS playback terminal are described in [reference needed]. Figure 6 As shown, it includes the following steps:

[0136] S21. Parse the MEPEG-TS extended fields of the acquired audio and video slices;

[0137] The method for parsing the MEPEG-TS extended fields of the acquired audio and video slices is described in [reference needed]. Figure 8 As shown, it includes the following steps:

[0138] S211. Parse the TS data packets from the MEPEG-TS extension field of the audio and video slices. Each TS data packet is 188 bytes long and contains a header and payload.

[0139] The packet header contains synchronization bytes, PID (Packet Identifier), and TSC (Transport Scrambling Control) information, while the payload contains audio and video data and PSI / SI table data.

[0140] S212. Obtain the PAT (Program Association Table) and PMT (Program Map Table) from the TS data packet, wherein the PAT table contains the PID value of the dedicated segment;

[0141] S213. Identify TS data packets with PID values ​​of dedicated segments, extract and merge the payloads of one or more TS data packets to obtain MEPEG-TS extended field data.

[0142] S22. Extract the slice list data from the MEPEG-TS extended field to determine the associated slices of the playlist;

[0143] Specifically, the associated slice information is parsed according to the dedicated segment syntax for slice linked list data. Based on the number of associated m3u8 slices to be recovered, the linked list data in the slices at the head or tail of the linked list can be parsed again, and their linked lists merged into the current linked list, until the total number of slices in the associated linked list matches the number of m3u8 slices to be recovered (see [link to relevant documentation]). Figure 4 (As shown).

[0144] S23. Obtain the audio and video metadata of the associated slice;

[0145] Extract the dedicated segment data of the audio and video metadata of the audio and video slices, and parse the audio and video information of the audio and video slices according to the dedicated segment syntax of the audio and video metadata.

[0146] S24. Based on the playlist and the audio and video metadata of each associated slice, generate the relevant tags for the required m3u8 file until the complete m3u8 file is restored.

[0147] The method for generating the required m3u8 tags is described in [reference needed]. Figure 9 As shown, it includes the following steps:

[0148] S241, based on the specified m3u8 version, restore the basic tags of m3u8; the basic tags of m3u8 include: EXTM3U, EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8;

[0149] S242. Recover the m3u8 media segment tags based on the audio and video metadata of the audio and video segments; the m3u8 media segment tags include: EXTINF, EXT-X-BYTERANGE, and EXT-X-PROGRAM-DATE-TIME;

[0150] The EXTINF tag identifies the duration of the media segment specified in the subsequent URL; this field can be recovered from the "duration" field in the segment's audio and video metadata. The URL following EXTINF is generated based on the specific segment storage location of the HLS service, and this tag can be recovered from the "segment_info_length" and "segment_info_byte" fields in the segment list data.

[0151] The EXT-X-BYTERANGE tag identifies the size of the data range of the media slice specified in the URL. This tag can be added when the m3u8 playlist needs to indicate the position of video keyframes in the slice. The tag can be recovered from the audio and video metadata of the audio and video slice, specifically the "keyframe_count", "keyframe_position", and "keyframe_size".

[0152] The EXT-X-PROGRAM-DATE-TIME tag uses an absolute date / time to indicate the sampling time of the first sample segment; this tag can be recovered from the "date_time" in the audio / video metadata of the audio / video slice.

[0153] S243. Restore playlist tags, which include:

[0154] EXT-X-STREAM-INF represents the attribute information of the media stream. Its BANDWIDTH content can be recovered from the "video_data_rate" and "audio_sample_rate" of the segment audio and video metadata. Its RESOLUTION content can be recovered from the "height" and "width" of the segment audio and video metadata. Its CODECS content can be recovered from the "codecs_info_length" and "codecs_info_byte" of the segment audio and video metadata.

[0155] EXT-X-TARGETDURATION indicates the maximum duration (in seconds) of a slice in the playlist. This tag is restored by selecting the maximum "duration" value from all the audio and video metadata of the slices in the playlist.

[0156] EXT-X-MEDIA-SEQUENCE represents the sequence number of the first URL segment file in the playlist. This tag is recovered by selecting the audio / video metadata "sequence" of the first segment of the playlist.

[0157] This invention also provides a computer device. Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 10 As shown, the computer device includes: an input system 23, an output system 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the HLS information generation method provided in the above embodiments; wherein the input system 23, the output system 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0158] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the HLS information generation method described in this embodiment of the invention. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] The input system 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; the output system 24 may include display devices such as a display screen.

[0160] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the HLS information generation method described above.

[0161] The computer equipment provided above can be used to execute the HLS information generation method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0162] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the HLS information generation method provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape systems; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0163] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the HLS information generation method described in the above embodiments, but can also perform related operations in the HLS information generation method provided in any embodiment of the present invention.

[0164] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating HLS information, characterized in that, include: The HLS server handles the following issues: handling problems such as m3u8 acquisition errors, data loss, and data anomalies on the HLS playback end. The HLS server-side processing method includes the following steps: S11. Slice the streaming audio and video data into segments of fixed duration to obtain multiple audio and video segments; S12. Obtain audio and video information related to the audio and video slice; S13. Encapsulate the audio and video data of the audio and video slices into MEPEG-TS format; S14. Encode the relevant audio and video information of the audio and video slice into audio and video metadata of the audio and video slice according to the set format, and store the audio and video metadata in the MEPEG-TS extended field of the audio and video slice. S15. Generate HLS m3u8 information based on the audio and video information related to multiple audio and video slices; the form of generating HLS m3u8 information includes: generating an m3u8 playlist according to the RFC 8216 standard document on HLS with a set window size of the m3u8 playlist. S16. Encode the m3u8 information of the preceding and following audio and video slices to generate the current slice linked list data, and store the slice linked list data in the MEPEG-TS extended field of the preceding and following audio and video slices. The handling method for m3u8 acquisition errors, data loss, and data anomalies in the HLS playback terminal includes the following steps: S21. Parse the MEPEG-TS extended fields of the acquired audio and video slices; S22. Extract the slice list data from the MEPEG-TS extended field to determine the associated slices of the playlist; S23. Obtain the audio and video metadata of the associated slice; S24. Based on the playlist and the audio and video metadata of each associated slice, generate the relevant tags for the required m3u8 file until the complete m3u8 file is restored; The method for parsing the MPEG-TS extended field of the acquired audio and video slices in step S21 includes the following steps: S211. Parse the TS data packets from the MEPEG-TS extension field of the audio and video slices. Each TS data packet is 188 bytes long and contains a header and payload. S212. Obtain the PAT table and PMT table from the TS data packet, wherein the PAT table contains the PID value of the dedicated segment; S213. Identify TS data packets with PID values ​​of dedicated segments, extract and merge the payloads of one or more TS data packets to obtain MEPEG-TS extended field data.

2. The HLS information generation method according to claim 1, characterized in that, The method for generating HLS m3u8 information based on audio and video information related to multiple audio and video slices in step S15 includes the following steps: S151. Generate basic tags for m3u8 based on the specified m3u8 version; the basic tags for m3u8 include: EXTM3U and EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8; S152. Generate m3u8 media slice class tags based on the audio and video information related to the audio and video slices; the m3u8 media slice class tags include: EXTINF, EXT-X-BYTERANGE, and EXT-X-PROGRAM-DATE-TIME; wherein, the EXTINF tag indicates the duration of the media slice specified in the URL; the EXT-X-BYTERANGE tag indicates the size of the relevant data range of the media slice specified in the URL; the EXT-X-PROGRAM-DATE-TIME tag uses an absolute date / time to indicate the sampling time of the first sample segment; S153. Generate playlist-related tags; the playlist-related tags include: EXT-X-STREAM-INF、EXT-X-TARGETDURATION、 EXT-X-MEDIA-SEQUENCE; The EXT-X-STREAM-INF tag represents the attribute information of the media stream, including: BANDWIDTH, RESOLUTION, and CODECS; where BANDWIDTH is the audio / video bitrate in the audio / video information related to the audio / video slice; RESOLUTION is the resolution width and height pixel values ​​in the audio / video information related to the audio / video slice; and CODECS is the encoder information in the audio / video information related to the audio / video slice. The EXT-X-TARGETDURATION tag indicates the maximum duration of a slice in the playlist, in seconds. This tag value is generated by selecting the maximum slice duration value from all slice information in the playlist. The EXT-X-MEDIA-SEQUENCE tag indicates the sequence number of the first URL segment file in the playlist; each media segment URL has a unique integer sequence number.

3. The HLS information generation method according to claim 1, characterized in that, The method for generating the required m3u8 tags in step S24 includes the following steps: S241, based on the specified m3u8 version, restore the basic tags of m3u8; the basic tags of m3u8 include: EXTM3U, EXT-X-VERSION; wherein, EXTM3U is a fixed m3u8 identifier, and the EXT-X-VERSION tag is used to indicate the current version of m3u8; S242. Recover the m3u8 media slice class tags based on the audio and video metadata of the audio and video slices; the m3u8 media slice class tags include: EXTINF, EXT-X-BYTERANGE, EXT-X-PROGRAM-DATE-TIME; The EXTINF tag indicates the duration of the media slice specified in the URL that follows; The EXT-X-BYTERANGE tag indicates the size of the data range associated with the media slice specified in the subsequent URL; The EXT-X-PROGRAM-DATE-TIME label uses an absolute date / time to indicate the sampling time of the first sample fragment; S243. Restore playlist related tags, wherein the playlist related tags include: EXT-X-STREAM-INF represents the attribute information of the media stream; EXT-X-TARGETDURATION indicates the maximum duration of a slice in the playlist; EXT-X-MEDIA-SEQUENCE represents the sequence number of the first URL segment file in the playlist.

4. The HLS information generation method according to claim 1, characterized in that, The packet header in step S211 contains synchronization bytes, PID, and TSC information, and the payload contains audio and video data and PSI / SI table data.

5. The HLS information generation method according to claim 1, characterized in that, The MPEG-TS extension fields in step S14 are defined by the MPEG-TS standard. The fields used by MPEG-TS extensions include: dedicated segments and descriptors. The dedicated segments are used to transmit user-defined dedicated data, and the descriptors are used to describe the characteristics and parameters of audio and video streams and other data.

6. The HLS information generation method according to claim 4, characterized in that, The method for slicing streaming audio and video data into segments of fixed duration in step S11 includes: The streaming audio and video data is segmented by time according to any preset duration, including 10 seconds or 20 seconds, and the streaming audio and video data is divided into multiple segments. Each segment is stored independently as an audio and video slice data for subsequent processing. At the same time as slicing, information such as slice number, slice start timestamp, and slice duration are recorded for use in the generation and use of relevant fields in subsequent steps.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the HLS information generation method according to any one of claims 1-6.

8. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the HLS information generation method as described in any one of claims 1-6.

Images