Reference clock design method for synchronously playing pure audio streams on multiple terminals

Abstract

The present invention discloses a reference clock design method for synchronously playing a pure audio stream in multiple terminals, comprising the following steps: when a live broadcast or on-demand task is started, a data source code stream is obtained, and an initial value of a synchronization reference timestamp is specified; an output file encapsulation format is specified, a video stream encoding task is started, and a redundant video stream is synchronously generated; the data source code stream is decapsulated, and the parsed content includes an audio frame, and a synchronization timestamp is calculated; for the redundant video stream, in the video frame supplementary enhancement information SEI part, the payload type about the synchronization timestamp is customized according to the specification, and the synchronization timestamp value of the actual video frame is written in the content part of the payload; the audio frame contained in the data source code stream and the video data frame with the customized synchronization timestamp information are re-encapsulated according to the file format specified by the task; different terminals obtain the code stream to be synchronously played, decapsulate, and obtain the newly defined timestamp value from the video stream.

Description

Technical Field

[0001] The invention belongs to the technical field of stream media production and playback, and in particular relates to a reference clock design method for synchronously playing a pure audio stream by multiple terminals. Background Art

[0002] When playing the same program source on various different types of smart terminal devices, users often hope to achieve synchronous playback, that is, the images seen on different broadcasting devices at any time are synchronized. In fact, since the transmission of streaming media often goes through many links (content production end-source station-content distribution-consumer terminal), each link will produce different degrees of delay; the timestamps of the audio and video frames initially carried by the stream from the data source are distributed through multiple links, and are usually changed by the streaming media server at the intermediate or edge node during the transmission process. In addition, the clocks on different devices cannot be completely synchronized, so there is a lack of a common reference for synchronization. This common reference for synchronization can be defined as the timestamp information that does not change the data source code stream after various transmission and distribution links and corresponds to the current audio and video frame in the data source code stream.

[0003] Normally, what multiple terminals need to play synchronously are the data source code streams after various transmission and distribution links, which also contain audio and video encoding data. This common reference for synchronization can be added by customizing the encapsulation format or encoding data. However, for some specific application scenarios, some terminals need to play pure audio streams synchronously, the data source is pure audio or some terminals participating in synchronization can only play audio streams, and the encapsulation format and audio encoding format are specified. It is impossible to add the timestamp required for common reference for synchronous playback through the corresponding encapsulation format and encoding data. Summary of the invention

[0004] In view of the above problems, the present invention provides a reference clock design method for the synchronous playback of multiple terminals containing pure audio streams, aiming to provide a timestamp calibration mechanism without the need for precise timing synchronization of the terminals. When the terminals participating in the synchronous playback contain pure audio streams, users of different terminal devices can also obtain and adopt it as a common time reference as the basis for synchronous playback.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A reference clock design method for synchronously playing a pure audio stream by multiple terminals includes the following steps:

[0007] When the live broadcast or on-demand task is started, the data source stream is obtained and the initial value of the synchronization reference timestamp is specified;

[0008] Specify the output file encapsulation format, start a video stream encoding task, and synchronously generate a redundant video stream;

[0009] Decapsulate the data source code stream, parse out the content including audio frames, and calculate the synchronization timestamp;

[0010] For redundant video streams, in the SEI part of the video frame, customize the payload type for the synchronization timestamp according to the specification, and write the actual synchronization timestamp value of the video frame in the content part of the payload;

[0011] Repackage the audio frames contained in the data source code stream and the video data frames with custom synchronization timestamp information in the file format specified by the task;

[0012] Different terminals obtain the code stream to be played synchronously, decapsulate it, and obtain the newly defined timestamp value from the video stream as the basis for synchronous playback.

[0013] In a possible design, the redundant video stream is used as a carrier of the synchronization timestamp and does not contain truly meaningful picture data; the redundant video stream complies with the encoding specification.

[0014] In a possible design, the redundant video stream encoding parameters occupy as little transmission bandwidth as possible, and set the resolution and transmission bit rate as low as possible.

[0015] In a possible design, the data source bitstream is decapsulated, each audio frame is parsed, and synchronization timestamp calculation is performed, specifically including: calculating the timestamp information of the corresponding video frame sequence according to the frame rate information specified by the output redundant bitstream.

[0016] In one possible design, the data source code stream is decapsulated, and the parsed content includes video frames, and the synchronization timestamps of the audio frames and video frames are calculated.

[0017] In a possible design, the synchronization timestamp calculation of the audio frame and the video frame includes:

[0018] The timestamps corresponding to the audio frames are timestampA1, timestampA2, ..., timestampA n ; The timestamps corresponding to the video frames are timestampV1, timestampV2, ..., timestampV n , subscripts 1 to n represent frame sequence numbers;

[0019] For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the value of the synchronization timestamp of the first audio frame to ptsAT1 = ptsT1, and the value of the synchronization timestamp of the first video frame to ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa;

[0020] For subsequent audio frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous audio frame; for subsequent video frames, the value of their synchronization timestamps is the increment relative to the synchronization timestamp of the previous video frame.

[0021] Adopting the present invention has the following beneficial effects: The reference clock design method provided by the embodiments of the present invention, for the scenario of multi-terminal synchronous playback and when the data source or the stream to be played by some terminals only contains an audio stream, by newly adding a video stream with a small amount of data and solidifying the newly defined reference timestamp in the code stream, the risk of the original timestamp being changed in the intermediate link is eliminated; for the case where the output code stream specifies the encapsulation format and the audio coding format, a general processing method is provided to ensure that the intermediate transmission link and the final terminal will not process abnormally, and after the terminal processes according to the defined rules, it can be used as a benchmark for synchronous playback. Description of the Drawings

[0022] Figure 1 It is a flowchart of the steps of the reference clock design method for multi-terminal synchronous playback including a pure audio stream in the embodiments of the present invention. Detailed Embodiments

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0024] See Figure 1 , which shows a flowchart of the steps of the reference clock design method for multi-terminal synchronous playback including a pure audio stream in the embodiments of the present invention, including the following steps:

[0025] S1. When the live broadcast or on-demand task is started, obtain the data source code stream and specify the initial value ptsT1 of the synchronization reference timestamp;

[0026] S2. Start a task for video stream encoding and synchronously generate a redundant video stream;

[0027] S3, decapsulate the data source code stream, parse out the content including audio frames, and calculate the synchronization timestamp;

[0028] S4: For redundant video streams, in the SEI part of the video frame, customize the payload type of the synchronization timestamp according to the specification, and write the actual synchronization timestamp value of the video frame in the content part of the payload;

[0029] S5, repackaging the audio frames contained in the data source code stream and the video data frames with custom synchronization timestamp information added according to the file format specified by the task;

[0030] S6, different terminals obtain the code stream to be played synchronously, decapsulate it, and obtain the newly defined timestamp value from the video stream as the basis for synchronous playback.

[0031] In one embodiment of the present invention, the initial value ptsT1 of the synchronization timestamp may be a random value, the current UTC time, or given in other appropriate ways, and the unit may be milliseconds, nanoseconds, or a specific unit.

[0032] In one embodiment of the present invention, the redundant video stream is used as a carrier of the synchronization timestamp, does not contain truly meaningful picture data, and can be a black screen or other image data that does not affect the playback. The redundant video stream complies with the encoding specifications and will not cause abnormal processing in the midway distribution link or terminal, resulting in the synchronization timestamp information carried being discarded, or abnormal terminal playback, etc. The redundant video stream encoding parameters ensure that the transmission bandwidth is occupied as little as possible, that is, the resolution and transmission bit rate are set as low as possible. For example, the bit rate of a video that is usually guaranteed to be clearly viewed is at the level of 300Kbit per second. The resolution of the redundant bit stream can be set to 8*8 pixels and the bit rate can be set to 8Kbit per second, which can greatly reduce the transmission bandwidth.

[0033] In one embodiment of the present invention, when the data source code stream does not contain video, the timestamps timestampA1, timestampA2, ..., timestampA corresponding to each audio frame are parsed. n ; Subscripts 1 to n represent frame sequence numbers. The timestamp information of the corresponding video frame sequence is calculated according to the frame rate information specified by the output redundant code stream. For example, when the audio frame interval is 20 milliseconds, the redundant video stream frame rate is 25 frames, and the frame interval is 40 milliseconds, timestampA1 = 1 millisecond, timestampA2 = 21 milliseconds, timestampA3 = 41 milliseconds, timestampA4 = 61 milliseconds, timestampA5 = 81 milliseconds, ..., the corresponding video frame sequence timestamp values ​​are timestampV1 = 0 milliseconds, timestampV2 = 40 milliseconds, timestampV3 = 80 milliseconds, and so on.

[0034] In one embodiment of the present invention, for the data source stream demultiplexing, parsing out each audio frame or video frame, and performing the synchronous timestamp calculation specifically includes:

[0035] Demultiplex the data source stream, and parse out the timestamps corresponding to each audio frame or video frame. The timestamps corresponding to the audio frames are timestampA1, timestampA2,..., timestampA n ; The timestamps corresponding to the video frames are timestampV1, timestampV2,..., timestampV n , where the subscripts from 1 to n represent the frame sequence numbers;

[0036] For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the synchronous timestamp value of the first audio frame as ptsAT1 = ptsT1, and the synchronous timestamp value of the first video frame as ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa. That is, if timestampV1 < timestampA1, set the synchronous timestamp value of the first video frame as ptsVT1 = ptsT1, and the synchronous timestamp value of the first audio frame as ptsAT1 = (ptsT1 + timestampA1 - timestampV1); for subsequent audio frames, the value of its synchronous timestamp is the increment relative to the synchronous timestamp of the previous audio frame. For example, for the audio frame sequence timestampA1, timestampA2,..., timestampA i , timestampA i+1 , …, timestampA n ; the corresponding synchronous timestamp sequence is ptsA1, ptsA2,..., ptsA i , ptsA i+1 , …, timestampA n ; where ptsA2 = ptsA1 + timestampA2 - timestampA1, ptsA i+1＝ timestampA i +timestampA i+1 -timestampA i . Similarly, for subsequent video frames, the value of its synchronous timestamp is the increment relative to the synchronous timestamp of the previous video frame.

[0037] In the specific application example, the above timestampA i , timestampV i Different file encapsulation formats present different values. For example, in TS (Transport Stream: ISO / IEC 13818:1) files, the values ​​are 33 bits, and the unit is 1 / 90 milliseconds; while in FLV files, the values ​​are 24 to 32 bits, and the unit is milliseconds. When these values ​​are inconsistent with the units of the set synchronization timestamps, a conversion is required. Assuming that the unit of the synchronization timestamp is milliseconds, the timestamps in the TS files need to be divided by 90 before the above assignment process.

[0038] In a specific application example, a reference clock design method for synchronously playing a pure audio stream on multiple terminals, where the data source stream also contains audio and video data, includes the following steps:

[0039] Start the on-demand task, obtain the original data source stream, and set the initial reference timestamp value;

[0040] Set the file format of the new code stream output, such as TS, FLV or others. Start a new video stream encoding task and generate a redundant video stream as a carrier for the reference timestamp;

[0041] Decapsulate the original bitstream, obtain the audio and video timestamp values ​​of the original bitstream, calculate the increment relative to the original timestamp based on the initial timestamp, and obtain the value of the new reference video timestamp sequence;

[0042] Write the calculated new reference video timestamp value into the SEI part of each frame of the new redundant video stream;

[0043] According to the encapsulation format specified by the output, the audio stream obtained after decapsulating the data source code stream (re-encoded as needed) and the redundant video stream generated by the new encoding are encapsulated into a new code stream, and the code stream enters the next processing link.

[0044] In another specific application example, a reference clock design method for synchronously playing a pure audio stream on multiple terminals, where the data source stream only contains audio data, includes the following steps:

[0045] Start the live broadcast task, obtain the original data source stream, and set the initial reference timestamp value.

[0046] Set the file format of the new code stream output, such as TS, FLV or others. Start a new video stream encoding task and generate a redundant video stream as a carrier for the reference timestamp.

[0047] Decapsulate the original bitstream, obtain the audio stream and the timestamp value corresponding to the audio stream, calculate the increment relative to the original timestamp based on the initial timestamp, and obtain the value of the new reference audio timestamp sequence. Calculate the value of the converted video frame timestamp sequence according to the encoding frame rate set by the redundant video bitstream.

[0048] The calculated reference timestamp value is written into the SEI portion of each frame of the new redundant video stream.

[0049] According to the encapsulation format specified by the output, the audio stream obtained after decapsulating the data source code stream (re-encoded as needed) and the redundant video stream generated by the new encoding are encapsulated into a new code stream, and the code stream enters the next processing link.

[0050] The reference clock design method for synchronous playback of multiple terminals containing pure audio streams provided by the embodiment of the present invention, for the scenario where multiple terminals are synchronously playing and the data source or the stream to be played by some terminals only contains audio streams, a newly defined reference timestamp is solidified in the bit stream by adding a new video stream with a small amount of data, thereby eliminating the risk of the original timestamp being changed in the intermediate link; for the situation where the output bit stream specifies the encapsulation format and audio encoding format, a general processing method is provided to ensure that the intermediate transmission link and the final terminal will not process exceptions, and the terminal can be used as a benchmark for synchronous playback after processing according to the defined rules.

[0051] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention are described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A reference clock design method for synchronously playing a pure audio stream on multiple terminals, characterized in that: It includes the following steps: When starting a live or on-demand task, obtain the data source code stream and specify the initial value ptsT1 of the synchronization reference timestamp; Specify the output file encapsulation format, start a video stream encoding task, and synchronously generate a redundant video stream; Demultiplex the data source code stream, and the parsed content includes audio frames for synchronous timestamp calculation; For the redundant video stream, in the supplementary enhancement information SEI part of the video frame, customize the payload type regarding the synchronous timestamp according to the specification, and write the value of the synchronous timestamp of the actual video frame in the content part of the payload; Repackage the audio frames included in the data source code stream and the video data frames with the customized synchronous timestamp information according to the file format specified by the task; Different terminals obtain the bitstreams to be synchronously played, demultiplex them, and obtain the newly defined timestamp value from the video stream as the basis for synchronous playback.

2. The reference clock design method for synchronously playing a pure audio stream by multiple terminals as claimed in claim 1, characterized in that: The redundant video stream is used as a carrier for carrying the synchronous timestamp and does not contain truly meaningful picture data; the redundant video stream complies with the encoding specification.

3. The reference clock design method for synchronously playing a pure audio stream by multiple terminals as claimed in claim 1, characterized in that: The encoding parameters of the redundant video stream occupy the smallest possible transmission bandwidth, and the lowest possible resolution and transmission bit rate are set.

4. The reference clock design method for synchronously playing a pure audio stream by multiple terminals as claimed in claim 1, characterized in that: Demultiplex the data source code stream, parse each audio frame, and the specific synchronous timestamp calculation includes: calculating the timestamp information of the corresponding video frame sequence according to the frame rate information specified by the output redundant bitstream.

5. The reference clock design method for synchronously playing a pure audio stream by multiple terminals as described in any one of claims 1 to 4, characterized in that: Demultiplex the data source code stream, and the parsed content includes video frames for synchronous timestamp calculation of audio frames and video frames.

6. The reference clock design method for synchronously playing a pure audio stream by multiple terminals as claimed in claim 5, characterized in that: The synchronous timestamp calculation of audio frames and video frames includes: The timestamps corresponding to the audio frames are timestampA1, timestampA2, ..., timestampA n ; The timestamps corresponding to the video frames are timestampV1, timestampV2, ..., timestampV n , subscripts 1 to n represent frame sequence numbers; For the timestamp values timestampA1 and timestampV1 of the first audio frame or the first video frame, select the smaller one. If timestampA1 < timestampV1, set the synchronous timestamp value of the first audio frame to ptsAT1 = ptsT1, and the synchronous timestamp value of the first video frame to ptsVT1 = (ptsT1 + timestampV1 - timestampA1); vice versa, that is, if timestampV1 < timestampA1, set the synchronous timestamp value of the first video frame to ptsVT1 = ptsT1, and the synchronous timestamp value of the first audio frame to ptsAT1 = (ptsT1 + timestampA1 - timestampV1); For subsequent audio frames, the value of their synchronous timestamp is the increment relative to the synchronous timestamp of the previous audio frame; for subsequent video frames, the value of their synchronous timestamp is the increment relative to the synchronous timestamp of the previous video frame.

Images