Reference clock design method supporting multi-terminal synchronous playback

Abstract

The present invention discloses a reference clock design method supporting multi-terminal synchronous playback, comprising the following steps: setting an initial synchronization reference timestamp, which is used to obtain a data source code stream and specify an initial value of the synchronization timestamp when a live broadcast or on-demand task is started; decapsulating the data source code stream to obtain a code stream audio and video frame sequence, and performing synchronization timestamp calculation; calibrating the synchronization timestamp in the encapsulation layer or encoding layer of the code stream file by adding private data provided by the specification; re-encapsulating or encoding the audio and video data frames with custom synchronization timestamp information added according to the file format of the source code stream; different terminals obtain the re-encapsulated or encoded code stream, parse it according to the write rule, and obtain the timestamp value corresponding to the current audio and video frame data as the basis for synchronous playback.

Description

Technical Field

[0001] The invention belongs to streaming media production and playback, and in particular relates to a reference clock design method supporting multi-terminal synchronous playback. Background Art

[0002] When playing the same program source on different types of smart terminal devices, users often hope to achieve synchronous playback, that is, the images seen on different broadcasting devices at any time are synchronized.

[0003] In fact, since the transmission of streaming media often goes through many links (content production end-source station-content distribution-consumer terminal), each link will produce different degrees of delay; the timestamps of the audio and video frames initially carried by the stream from the data source are distributed through multiple links and are usually changed by the streaming media servers in the middle or edge nodes during the transmission process. In addition, the clocks on different devices cannot be completely synchronized, so there is a lack of a common reference for achieving synchronization. Summary of the invention

[0004] In view of the above problems, the present invention provides a reference clock design method that supports multi-terminal synchronous playback.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A reference clock design method supporting multi-terminal synchronous playback includes the following steps:

[0007] Set the initial synchronization reference timestamp, which is used to obtain the data source stream when the live broadcast or on-demand task is started, and specify the initial value of the synchronization timestamp ptsT1;

[0008] Decapsulate the data source code stream into the code stream audio and video frame sequence, and calculate the synchronization timestamp;

[0009] At the encapsulation layer or encoding layer of the bitstream file, the synchronization timestamp is calibrated by adding private data as provided by the specification;

[0010] Re-encapsulate or encode the audio and video data frames with custom synchronization timestamp information according to the file format of the source code stream;

[0011] Different terminals obtain the repackaged or encoded bitstream, parse it according to the written rules, and obtain the timestamp value corresponding to the current audio and video frame data as the basis for synchronous playback.

[0012] In one possible design, the data source code stream is decapsulated into the code stream audio and video frame sequence, and the synchronization timestamp calculation specifically includes:

[0013] Demultiplex the data source stream to parse out the timestamps corresponding to each audio frame or video frame. The timestamps corresponding to the audio frames are timestampA1, timestampA2, …, timestampAn; the timestamps corresponding to the video frames are timestampV1, timestampV2, …, timestampVn, where the subscripts 1 to n represent the frame sequence numbers.

[0014] For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the value of the synchronization timestamp of the first audio frame to ptsAT1 = ptsT1, and the value of the synchronization timestamp of the first video frame to ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa.

[0015] For subsequent audio frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous audio frame; for subsequent video frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous video frame.

[0016] In a possible design, the process of calibrating the synchronization timestamps by adding private data provided by the specification in the encapsulation layer of the stream file is as follows:

[0017] Based on the file encapsulation layer, for TS files, add a description of private_packet in the PMT; define the format of private_packet, add a description of private_date_byte, customize the descriptor information about the synchronization timestamps, and write the actual values of the synchronization timestamps of the audio and video frames as ptsAi and ptsVi respectively.

[0018] In a possible design, in the encoding layer of the stream file, the process of calibrating the synchronization timestamps by adding private data provided by the specification is as follows: For the encoding types of video frames being H264 / H265, in the Supplemental Enhancement Information (SEI) section, customize the payload type about the synchronization timestamps according to the specification, and write the actual value of the synchronization timestamp of the video frame, ptsVi, in the content part of the payload.

[0019] In one possible design, at the bitstream file encoding layer, the synchronization timestamp is calibrated by adding private data as provided by the specification. The process is as follows: for audio frames with AAC encoding type, the raw data is obtained after decoding, and the private data type definition of DSE is added according to the specification. The synchronization timestamp value ptsAi of the actual audio frame is written in the data content part, and then it is re-encoded into audio frame data with a custom synchronization timestamp added.

[0020] The present invention has the following beneficial effects:

[0021] (1) Through the reference clock design method provided in the embodiment of the present invention, the newly defined audio and video reference timestamp is solidified in the bitstream, eliminating the risk of the original timestamp being changed in the intermediate link, and ensuring that each playback end can obtain it and use it as a reference for synchronous playback;

[0022] (2) The embodiment of the present invention provides a variety of timestamp calibration methods based on the file encapsulation layer and the coding layer. For data source streams of different file formats and different coding methods, a technology for synchronous playback of multiple terminals can be applied based on a common timestamp reference.

[0023] (3) Even if the data source code stream is in a single specific format, a new code stream is formed by transpackaging or transcoding. If the timestamp reference design method of the embodiment of the present invention is used in the new code stream, multiple playback terminals can also perform synchronous playback based on this when playing different code streams formed after the conversion of the same original data source code stream.

[0024] (4) In addition, other different file formats and encoding formats can also be calibrated in a similar manner, which greatly expands the scope of application of the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flowchart of the steps of a reference clock design method for supporting multi-terminal synchronous playback according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1 , shown is a reference clock design method supporting multi-terminal synchronous playback according to an embodiment of the present invention, comprising the following steps:

[0028] S1, set the initial synchronization reference timestamp, which is used to obtain the data source stream when the live broadcast or on-demand task is started, and specify the initial value of the synchronization timestamp ptsT1;

[0029] S2, decapsulate the data source code stream to obtain the code stream audio and video frame sequence, and calculate the synchronization timestamp;

[0030] S3, at the encapsulation layer or encoding layer of the bitstream file, the synchronization timestamp is calibrated by adding private data as provided by the specification;

[0031] S4, re-encapsulating or encoding the audio and video data frames with the custom synchronization timestamp information according to the file format of the source code stream;

[0032] S5, different terminals obtain the re-encapsulated or encoded bit streams, parse them according to the written rules, and obtain the timestamp value corresponding to the current audio and video frame data as the basis for synchronous playback.

[0033] In one embodiment of the present invention, the initial value ptsT1 of the synchronization timestamp may be a random value, the current UTC (Universal Time Coordinated) time, or may be given in other appropriate ways, and the unit may be milliseconds, nanoseconds, or a specific unit.

[0034] In one embodiment of the present invention, decapsulating a data source code stream into a code stream audio and video frame sequence and performing synchronization timestamp calculation specifically includes:

[0035] Decapsulate the data source code stream and parse out the timestamps corresponding to each audio frame and video frame. The timestamps corresponding to the audio frames are timestampA1, timestampA2, …, timestampA n ; The timestamps corresponding to the video frames are timestampV1, timestampV2, ..., timestampV n , subscripts 1 to n represent frame sequence numbers;

[0036] For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the value of the synchronization timestamp of the first audio frame to ptsAT1 = ptsT1, and the value of the synchronization timestamp of the first video frame to ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa, that is, if timestampV1 < timestampA1, set the value of the synchronization timestamp of the first video frame to ptsVT1 = ptsT1, and the value of the synchronization timestamp of the first audio frame to ptsAT1 = (ptsT1 + timestampA1 - timestampV1)

[0037] For subsequent audio frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous audio frame; for subsequent video frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous video frame. For example, for the audio frame sequence timestampA1, timestampA2, …, timestampA i , timestampA i+1 , …, timestampA n ; the corresponding synchronization timestamp sequence is ptsA1, ptsA2, …, ptsA i , ptsA i+1 , …, timestampA n ; where ptsA2 = ptsA1 + timestampA2 - timestampA1, ptsA i+1 = timestampA i + timestampA i+1 - timestampA i .

[0038] In a specific application example, the above timestampA i , timestampV i are presented differently in different file encapsulation formats. For example, in a TS (Transport Stream: ISO / IEC 13818:1) file, they are 33-bit values with the unit of 1 / 90 milliseconds; while in an FLV file, they are 24 - 32-bit values with the unit of milliseconds. When these values are inconsistent with the unit of the set synchronization timestamp, a conversion is required. Assuming the unit of the synchronization timestamp is milliseconds, the timestamps in the TS file need to be divided by 90 first and then the above assignment process is performed.

[0039] In one embodiment of the present invention, the process of calibrating the synchronization timestamp in the encapsulation layer of the code stream file by adding private data as provided by the specification is as follows:

[0040] Based on the file encapsulation layer, for TS files, add a description of private_packet in PMT (Program Map Table); define the format of private_packet, add a description of private_date_byte, customize the descriptor information about the synchronization timestamp, and write the actual synchronization timestamp values ​​of the audio and video frames as ptsAi and ptsVi respectively.

[0041] In one embodiment of the present invention, at the bitstream file encoding layer, the synchronization timestamp is calibrated by adding private data as provided by the specification as follows: for video frames with an encoding type of H264 / H265, in the supplementary enhancement information SEI part, the payload type for the synchronization timestamp is customized according to the specification, and the actual value ptsVi of the synchronization timestamp of the video frame is written in the content part of the payload.

[0042] In one embodiment of the present invention, at the bitstream file encoding layer, the synchronization timestamp is calibrated by adding private data as provided in the specification. The process is as follows: for an audio frame with an encoding type of AAC, the raw data is obtained after decoding, and the private data type definition of DSE is added according to the specification. The synchronization timestamp value ptsAi of the actual audio frame is written in the data content part, and then re-encoded into audio frame data with a custom synchronization timestamp added.

[0043] In one embodiment of the present invention, for the live broadcast / on-demand task based on the same original code stream, transcoding or repackaging to form multiple new code stream tasks, that is, there are multiple new same-source code streams, or the new same-source code streams and the original code streams are distributed at the same time, steps S1 to S5 can be implemented in the new same-source code stream, and each terminal that finally obtains different code streams can use this as the basis for synchronous playback.

[0044] In a specific application example of the present invention, a reference clock design method supporting multi-terminal synchronous playback includes the following steps:

[0045] Start the on-demand task, obtain the original data source code stream, and set the initial reference timestamp value; decapsulate the original code stream, obtain the audio and video timestamp values ​​of the original code stream, calculate the increment relative to the original timestamp based on the initial reference timestamp, obtain the new reference audio and video timestamp sequence values, and write the newly calculated reference timestamp values ​​into the private fields of the audio and video frame data in the file encapsulation layer; re-encapsulate based on the file encapsulation format of the data source code stream, and the code stream enters the next processing link.

[0046] In a specific application example of the present invention, a reference clock design method supporting multi-terminal synchronous playback includes the following steps:

[0047] Start the live broadcast task, obtain the original data source code stream, set the initial reference timestamp value; decapsulate the original code stream, obtain the audio and video timestamp values ​​of the original code stream, calculate the increment relative to the original timestamp based on the initial timestamp, obtain the new reference audio and video timestamp sequence values, decode the original code stream, and write the newly calculated reference timestamp value into the private field of the audio and video frame data in the coding layer; re-encode based on the coding format of the data source code stream, re-encapsulate based on the file encapsulation format of the data source code stream, and the code stream enters the next processing link.

[0048] In a specific application example of the present invention, a reference clock design method supporting multi-terminal synchronous playback includes the following steps:

[0049] Start the on-demand task, obtain the original data source code stream, and set the initial reference timestamp value; start one or more transpackaging tasks; decapsulate the original code stream, obtain the audio and video timestamp values ​​of the original code stream, calculate the increment relative to the original timestamp based on the initial timestamp, and obtain the value of the new reference audio and video timestamp sequence. For the code stream generated by the transpackaging task, write the newly calculated reference timestamp value into the private field of the audio and video frame data at the file layer; the code stream of the transpackaging task with the new reference timestamp information added enters the next processing link.

[0050] In a specific application example of the present invention, a reference clock design method supporting multi-terminal synchronous playback includes the following steps:

[0051] Start the live broadcast task, obtain the original data source code stream, set the initial reference timestamp value; start one or more transcoding tasks; decapsulate the original code stream, obtain the audio and video timestamp values ​​of the original code stream, calculate the increment relative to the original timestamp based on the initial reference timestamp, obtain the value of the new reference audio and video timestamp sequence, and write the newly calculated reference timestamp value to the private field of the audio and video frame data at the file layer or encoding layer for the code stream generated by the transcoding task. The code stream of the transcoding task with the new reference timestamp information added enters the next processing link.

[0052] Through the reference clock design method for multi-terminal synchronous playback of the embodiment of the present invention described above, it can be known that by adopting the above method, for a specific live broadcast or on-demand task, the original data source code stream is obtained, and an initial timestamp based on the reference clock time axis is set; the original code stream is decapsulated, and the timestamp of the new audio and video sequence based on the reference clock time axis is calculated relative to the original timestamp of the data source code stream; the value of the newly defined audio and video timestamp is written in the form of private data; the code stream with the newly defined timestamp information added is re-encapsulated (re-encoding is required if the encoding is changed), and enters the next processing link, such as CDN (content distribution network) for distribution; different terminals obtain the re-encapsulated or encoded code stream, parse it according to the write rules, and obtain the newly defined timestamp value of the current audio and video frame data as the basis for synchronous playback.

[0053] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A reference clock design method supporting multi-terminal synchronous playback, characterized in that: It includes the following steps: Set an initial synchronization reference timestamp, which is used to obtain the data source stream when starting a live or on-demand task, and specify the initial value ptsT1 of the synchronization timestamp; Demultiplex the data source stream to obtain the stream audio and video frame sequences, and perform synchronization timestamp calculation; At the encapsulation layer or encoding layer of the stream file, calibrate the synchronization timestamp by adding private data provided by the specification; Repackage or encode the audio and video data frames with the custom synchronization timestamp information according to the file format of the source stream; Different terminals obtain the repackaged or encoded stream, parse it according to the writing rules, and obtain the timestamp value defined corresponding to the current audio and video frame data, which is used as the basis for synchronous playback.

2. The reference clock design method supporting multi-terminal synchronous playback according to claim 1, characterized in that: Demultiplexing the data source stream to obtain the stream audio and video frame sequences and performing synchronization timestamp calculation specifically includes: Decapsulate the data source code stream and parse out the timestamp corresponding to each audio frame or video frame. The timestamps corresponding to the audio frames are timestampA1, timestampA2, ..., timestampA n ; The timestamps corresponding to the video frames are timestampV1, timestampV2, ..., timestampV n , subscripts 1 to n represent frame sequence numbers; For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the synchronization timestamp value of the first audio frame to ptsAT1 = ptsT1, and the synchronization timestamp value of the first video frame to ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa, that is, if timestampV1 < timestampA1, set the synchronization timestamp value of the first video frame to ptsVT1 = ptsT1, and the synchronization timestamp value of the first audio frame to ptsAT1 = (ptsT1 + timestampA1 - timestampV1); For subsequent audio frames, the value of their synchronization timestamp is the increment relative to the synchronization timestamp of the previous audio frame; for subsequent video frames, the value of their synchronization timestamp is the increment relative to the synchronization timestamp of the previous video frame.

3. The reference clock design method supporting multi-terminal synchronous playback according to claim 1 or 2, characterized in that: The process of calibrating the synchronization timestamp by adding private data provided by the specification at the encapsulation layer of the stream file is as follows: Based on the file encapsulation layer, for TS files, add a description of private_packet in the PMT; define the format of private_packet, add a description of private_date_byte, customize the descriptor information about the synchronization timestamp, and write the actual synchronization timestamp values of the audio and video frames as ptsAi and ptsVi respectively.

4. The reference clock design method supporting multi-terminal synchronous playback according to claim 1 or 2, characterized in that: At the encoding layer of the stream file, the process of calibrating the synchronization timestamp by adding private data provided by the specification is as follows: for the encoding type of video frames being H264 / H265, in the supplementary enhancement information SEI part, customize the payload type about the synchronization timestamp according to the specification, and write the actual synchronization timestamp value ptsVi of the video frame in the content part of the payload.

5. The reference clock design method supporting multi-terminal synchronous playback according to claim 1 or 2, characterized in that: At the bitstream file encoding layer, the synchronization timestamp is calibrated by adding private data as provided by the specification. The process is as follows: for audio frames with the encoding type of Advanced Audio Coding AAC, the raw data is obtained after decoding, and the private data type definition of DSE is added according to the specification. The synchronization timestamp value ptsAi of the actual audio frame is written in the data content part, and then it is re-encoded into audio frame data with a custom synchronization timestamp added.

Images