Multi-camera end-to-end frame synchronization method based on multiplexing

Abstract

The invention discloses a multi-camera end-to-end frame synchronization method based on multiplexing. The method comprises the following steps: S10, carrying out clock synchronization configuration among a camera, an encoder and a multiplexer; s20, on the coding side, audio and video signals are collected through multiple cameras, and the display timestamp PTS of the collected audio and video streams is local UTC time; carrying out audio and video coding; tS single program packaging is carried out, and an original PTS is maintained; each path of program generates a multi-program stream at a multiplexer, a unique program reference clock PCR is generated according to a local clock for multiplexing of each path of program, and then the multi-program stream is sent to a demultiplexing dispatcher at a decoding side through a special line or a network; and S30, on a decoding side, the demultiplexing distributor carries out MPTS splitting and carries out rectification and alignment on audio and video code streams, the decoding multiplexer carries out rectification on a multi-program stream and then distributes the multi-program stream to a decoding terminal in a single-program stream form, and the decoding terminal carries out decoding, playing and displaying on a decoder player according to a built-in decoding synchronous playing strategy.

Description

Technical Field

[0001] The invention belongs to the technical field of audio and video encoding and transcoding, and in particular relates to a multi-camera end-to-end frame synchronization method based on multiplexing. Background Art

[0002] In live broadcast business scenarios such as sports events and cultural performances, in order to fully record the wonderful moments from all angles and meet the personalized needs of the audience and the director's switching needs, it is necessary to shoot the scene from multiple angles and transmit the video back. In the back-end playback or monitoring link, in order to achieve smooth switching and multi-screen broadcast effects, the video needs to be frame synchronized. In the existing solution, the acquisition encoder and the player link generally perform synchronization control based on the synchronization information embedded in the encoder, and the playback end performs synchronization control based on the synchronization information. The advantage of this solution is that each return stream is encoded and transmitted independently, supports multiple media encapsulation protocols, records the synchronization time at the encoding acquisition end, and displays and plays according to the synchronization time at the playback end, with less modification to the original system. The disadvantage is that each encoded code stream is transmitted independently, and the playback terminal receives it independently. Each player needs to implement the playback function based on the synchronization information, and the display delay needs to be preset according to the actual network jitter status of each link. Summary of the invention

[0003] In view of the above problems, the present invention provides a multi-camera end-to-end frame synchronization method based on multiplexing. By adding multiplexing and demultiplexing links, the front-end encoding and back-end decoding side systems are decoupled, so that the decoding terminal does not need to cooperate with the synchronization strategy of the encoding side and the front-end and back-end network status to achieve synchronous playback of each program frame.

[0004] The abbreviations of technical terms used in this application are as follows:

[0005] TS (Transport Stream), an audio and video packaging format, is called MPEG2-TS.

[0006] MPTS (Multi-Program Transport Stream) refers to a TS stream that contains multiple audio and video programs.

[0007] PCR (Program Clock Reference) is a basic clock in the DVB transport stream that the player can use.

[0008] DTS (Decoding Time Stamp), the significance of this timestamp is to tell the player when to decode the data of this frame.

[0009] PTS (Presentation Time Stamp), this timestamp is used to tell the player when to display the data of this frame.

[0010] UTC (Coordinated Universal Time) is the global unified time standard.

[0011] CBR (Constant Bitrate) means that the bit stream is transmitted at a constant bit rate.

[0012] SDI (Serial Digital Interface).

[0013] PTP (Precision Time Protocol) is a protocol used for time synchronization, and its time accuracy can usually reach microseconds or nanoseconds.

[0014] NTP (Network Time Protocol) is also a protocol used for time synchronization, and the time synchronization accuracy it provides is generally at the millisecond level.

[0015] I frame (Intra-coded picture), I frame indicates a key frame, which is referenced by other frames but can be decoded by itself without referring to other frames.

[0016] P frame (Predictive-coded Picture), P frame represents the difference between this frame and the previous key frame (or P frame). During decoding, the difference defined by this frame needs to be superimposed with the previously cached picture to generate the final picture.

[0017] B frame (Bidirectionally predicted picture), B frame is a bidirectional difference frame, that is, B frame records the difference between the current frame and the previous and next frames. To decode a B frame, it is necessary not only to obtain the previous cached picture, but also to decode the subsequent picture, and obtain the final picture by superimposing the previous and next pictures with the current frame data.

[0018] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0019] A multi-camera end-to-end frame synchronization method based on multiplexing comprises the following steps:

[0020] Perform clock synchronization configuration between cameras, encoders and multiplexers, including: Perform clock synchronization configuration between cameras, encoders and multiplexers, including: Setting multiple front-end cameras to achieve clock synchronization and same-frequency refresh acquisition through the camera synchronous phase-lock interface; Setting the live encoder corresponding to each front-end camera to collect camera signals through SDI; Setting the live encoder and multiplexer to perform clock synchronization through NTP / PTP. After synchronization, the local clocks of each device are synchronized with the same UTC clock source;

[0021] On the encoding side, audio and video signals are collected through multiple cameras, and the display timestamp PTS of the collected audio and video streams is the local UTC time; audio and video encoding is performed, and PTS is kept unchanged; TS single program encapsulation is performed, and the original PTS is kept; each program generates a multi-program stream in the multiplexer, and a unique program reference clock PCR is generated according to the local clock for multiplexing of each program, and then the multi-program stream is sent to the demultiplexer distributor on the decoding side through a dedicated line or network;

[0022] On the decoding side, the demultiplexer splits the multi-program transport stream MPTS and aligns the audio and video code streams. The decoder multiplexer rectifies the multi-program streams and distributes them to the decoding terminal as a single program stream. The decoding terminal decodes, plays and displays them on the decoder player according to the built-in decoding synchronization playback strategy.

[0023] In a possible implementation, setting the live encoder to collect camera signals via SDI includes: the encoder collects the camera audio and video SDI signals, records the arrival time of each SDI data frame as PTS_Capture, and uses the local UTC clock to record as the display timestamp PTS of each frame of audio and video.

[0024] In a possible implementation manner, each channel of the program generates a multi-program stream in the multiplexer, and generates a unique program reference clock PCR according to the local clock for multiplexing of each channel of the program, including: generating a new PCR as a common PCR for all programs through the local clock and the preset coding delay, counting the preset coding delay as V_delay, UTC_local as the local UTC time, setting the initial PCR_base=UTC_local+V_delay, MPTS encapsulation is sent in CBR mode, the total output bit rate of the multiple programs is determined by multiple single-channel encodings, multiplying a coefficient mux_coef on the basis of the sum of the multiple single-channel encoding bit rates, setting the sum of the audio and video encoding bit rates of the front-end encoder to TotalAVBitate, then the output encapsulation bit rate MPTSBitrate=mux_coef*TotalAVBitate, or the audio and video encoding bit rate of the single-channel encoder can be set by first determining the target total output bit rate;

[0025] During the multiplexing process of multiple program streams, PCR generates PCR_now=PCR_base+packet_cout*188 / MPTSBitrate in real time through PCR_base and the number of packets packet_count. The TS packet length is 188, and an independent PID is used to identify the PCR data packet. At the same time, datagrams are sent and empty packets are filled according to the standard process of the multiplexer, and the PSI / SI table is updated. The valid audio and video datagrams are directly used for the newly generated MPTS code stream.

[0026] In one possible implementation, the PSI / SI table includes a PAT table, a PMT table, and an SDT table. The PAT table includes the PIDs of all program table PMTs. The PMT table includes the PIDs of all audio and video streams in the program, audio and video description information, and program numbers. The SDT table includes the business description information of the program. The above information is used to extract corresponding audio and video and program description information data for playback or program arrangement processing during decoding and playback.

[0027] In one possible implementation, the demultiplexing distributor performs multi-program transport stream MPTS splitting including: splitting the multi-program stream into single-program streams according to the target output decoding terminal, wherein the audio and video valid data directly uses the original TS packets, and the PCR is distributed to each single-program channel by copying, and each single-program channel becomes a single-program TS code stream by updating the PSI / SI table.

[0028] In one possible implementation, the rectification and alignment of the audio and video streams includes: establishing a video stream cache for multiple channels, aligning the video frames frame by frame according to the timestamp PTS, and filling the gaps with TS packets with a PID consistent with the PID of the video stream, so that the video frame data sizes of all programs with the same PTS are also completely consistent. Since the audio is encoded at a fixed bit rate, the size of each frame itself is consistent. A sending queue is established for each audio and video channel, and the data queue head is aligned according to PTS. The audio and video frames with the same PTS are sent to the output stream at the same time, so that the split single program stream can achieve synchronous output and synchronous transmission of the same PTS frames when output, thereby achieving synchronous refresh of audio and video frames based on PTS.

[0029] The present invention has the following beneficial effects:

[0030] (1) A multiplexing and demultiplexing link is added to the traditional end-to-end encoding and decoding system. The multiplexing link is a separate software service processing module that can be deployed on the encoder or deployed separately. If deployed separately, clock synchronization with the encoder is required. The network jitter difference caused by independent encoding transmission of each channel can be eliminated by multiplexing multiple programs. The coupling relationship between the encoding end, decoding end and network status is released, and the independence of the front-end encoding system and the back-end decoding system is increased.

[0031] (2) Each encoding process uses UTC as the audio and video acquisition time and remains unchanged in the subsequent encoding, packaging and multiplexing processes. Independent PIDs are used in the single-channel and multiplexing processes to identify and store PCR information, making the audio and video TS packets independent. In the subsequent distribution process, there is no need to re-parse and encapsulate. PCR is generated through local UTC and video encoding cache.

[0032] (3) On the decoding side, a demultiplexing device is added as an independent device to split the multi-program streams. During the splitting process, the video is rectified to ensure that the output audio and video frames are refreshed at the same frequency and sent at the same time. Then, ASI or UDP unicast is used to send data so that the code streams received by each decoding terminal have the same frame data structure and size. Therefore, as long as each player uses the same playback strategy, the audio and video frames of each playback terminal can be played synchronously. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the system structure used in a multiplex-based end-to-end frame synchronization method for multiple cameras according to an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a flow chart of a method for end-to-end frame synchronization of multiple cameras based on multiplexing according to an embodiment of the present invention;

[0035] Figure 3 The present invention is a schematic diagram of video encoding frame synchronization rectification in a multiplexed multi-camera end-to-end frame synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Reference Figure 1 and Figure 2 ,in Figure 1A schematic diagram of the system structure used in a multiplex-based end-to-end frame synchronization method for multiple cameras according to an embodiment of the present invention; Figure 2 A flowchart of a multiplexed multi-camera end-to-end frame synchronization method implemented by the system using the above configuration includes the following steps:

[0038] S10, configure the clock synchronization between the camera, encoder and multiplexer, including: setting multiple front-end cameras to achieve clock synchronization and same-frequency refresh acquisition through the camera synchronous phase-lock interface; setting the live encoder corresponding to each front-end camera to collect camera signals through SDI; setting the live encoder and the multiplexer to perform clock synchronization through NTP / PTP, and after synchronization, the local clock of each device is synchronized with the same UTC clock source;

[0039] S20, on the encoding side, audio and video signals are collected through multiple cameras, and the display timestamp PTS of the collected audio and video streams is the local UTC time; audio and video encoding is performed, and the PTS is kept unchanged; TS single program encapsulation is performed, and the original PTS is kept; each program generates a multi-program stream in the multiplexer, and a unique program reference clock PCR is generated according to the local clock for multiplexing of each program, and then the multi-program stream is sent to the demultiplexer distributor on the decoding side through a dedicated line or network;

[0040] S30, on the decoding side, the demultiplexer distributor splits the multi-program transport stream MPTS, rectifies and aligns the audio and video code streams, and the decoder multiplexer rectifies the multi-program streams and distributes them to the decoding terminal as a single program stream. The decoding terminal decodes, plays and displays the decoder player according to the built-in decoding synchronization playback strategy.

[0041] In a specific application example, a multi-camera end-to-end frame synchronization method based on multiplexing is provided, in S10, a live encoder is set to collect camera signals through SDI, including: the encoder collects the camera audio and video SDI signals, records the arrival time of each SDI data frame as PTS_Capture, and uses the local UTC clock to record as the display timestamp PTS of each frame of audio and video.

[0042] Furthermore, in S10, audio and video encoding is performed, and the PTS is kept unchanged; TS single program encapsulation is performed, and the original PTS is kept, including: encoding and TS single program encapsulation are performed in a normal manner. During the encoding and multiplexing process, it is necessary to ensure that the PTS corresponding to each frame of audio and video remains unchanged during the audio and video encoding stage and the TS encapsulation stage, and support the GOP structure of each encoding channel, and have the same starting position, so as to keep the I, B, and P frame types of the video frames of each encoding channel at the same time consistent. In addition, in the TS encapsulation, the program reference clock PCR uses an independent PID identifier.

[0043] Since clock synchronization is achieved between the multiplexer and the encoder, multiple program streams can use the same PCR during the multiplexing stage. Further, in S10, each program generates a multi-program stream in the multiplexer, and generates a unique program reference clock PCR according to the local clock for multiplexing of each program, including: generating a new PCR as a common PCR for all programs through the local clock and the preset encoding delay, where the encoding delay includes the encoding delay itself and the transmission delay caused by the encoding bit rate jitter, the preset encoding delay is V_delay (generally 1-2 seconds), UTC_local is the local UTC time, and the initial PCR_base is set to UTC_local+V_delay, and the MPTS encapsulation is sent in CBR mode. The total output bit rate of multiple programs is determined by multiple single-channel encodings, and a coefficient mux_coef (the coefficient value is generally in the range of 1.05 to 1.2) is multiplied on the basis of the sum of the multiple single-channel encoding bit rates. The sum of the audio and video encoding bit rates of the front-end encoder is set to TotalAVBitate, then the output encapsulation bit rate MPTSBitrate=mux_coef*TotalAVBitate, or the audio and video encoding bit rate of the single-channel encoder can be set by first determining the target total output bit rate;

[0044] During the multiplexing process of multiple program streams, PCR generates PCR_now=PCR_base+packet_cout*188 / MPTSBitrate in real time through PCR_base and packet number packet_count. The TS packet length is 188, and an independent PID is used to identify the PCR data packet. At the same time, datagrams are sent and empty packets are filled according to the standard process of the multiplexer, and the PSI / SI table is updated. The audio and video valid data packets are directly used for the newly generated MPTS code stream. Since the PTS remains unchanged, the audio and video valid data packets do not need to be decapsulated here.

[0045] Furthermore, the PSI / SI table updated during the multiplexing process of multiple program streams includes a PAT table, a PMT table, and an SDT table. The PAT table includes the PIDs of all program tables PMT, the PMT table includes the PIDs of all audio and video streams in the program, audio and video description information, and program numbers, and the SDT table includes the business description information of the program (such as CCTV1). The above information is used to extract corresponding audio and video and program description information data for playback or program arrangement processing during decoding and playback.

[0046] A multi-camera end-to-end frame synchronization method based on multiplexing in a specific application example, the demultiplexing distributor in S20 performs multi-program transport stream MPTS splitting, including: splitting the multi-program stream into a single program stream according to the target output decoding terminal, wherein the audio and video valid data directly uses the original TS packet, and the PCR is distributed to each single program channel by copying, and each single program channel becomes a single program TS code stream by updating the PSI / SI table. The difference between the PSI / SI table here and the PSI / SI table at the front-end multiplexing end is that the front-end multiplexing end is a multi-program stream synthesized from multiple perspectives. The back-end demultiplexing end splits multiple programs (perspectives) into a single program stream, and the PSI table includes all the information of a program.

[0047] In order to enable the decoding terminal to achieve the ability to play video frames synchronously without any configuration, it is necessary to ensure that the code streams sent to each decoding terminal are consistent in terms of spatial structure and time synchronization. That is, each decoding terminal receives consistent data buffers, and each video frame occupies a consistent code stream size and arrives at the same time. Due to the IBP frame characteristics of the video code stream and the characteristics of video encoding, the size of each frame after encoding cannot be completely consistent, and the video code stream data frame needs to be synchronously rectified. Furthermore, the rectification and alignment of the audio and video code streams in S20 includes: Figure 3 As shown, video stream caches for multiple channels are established, and video frames are aligned frame by frame according to the timestamp PTS. The gaps are filled with TS packets with a PID consistent with the PID of the video stream, so that the data sizes of video frames with the same PTS for all programs are also completely consistent. Since the audio is encoded at a fixed bit rate, the size of each frame itself is consistent. A sending queue is established for each audio and video channel, and the data queue head is aligned according to PTS. Audio and video frames with the same PTS are sent to the output stream at the same time, so that the split single-program stream can achieve synchronous output and synchronous transmission of frames with the same PTS when output, thereby achieving synchronous refresh of audio and video frames based on PTS.

[0048] Specifically, see Figure 3 , classify the code stream into channels, namely video, audio and PCR. The video is padded so that the video frames with the same PTS are of the same size. Since the audio has a fixed bit rate, the size of each frame is the same. All audio and video are aligned according to PTS and sent from small to large. Frames with the same PTS are distributed to each program code stream at the same time and sent at the same time to achieve frame synchronization refresh. When encountering a PCR packet, the PCR packet is copied first and distributed to each split program code stream.

[0049] During the decoding and synchronous playback process, since the demultiplexer and the playback terminal belong to the same machine room, the playback equipment layout and the demultiplexer belong to the same switch, or the ASI cable can be used directly for code stream data transmission from the demultiplexer to the playback terminal, demultiplexing realizes the simultaneous sending of audio and video packets with the same timestamp and the video frame and audio frame sizes of each program are completely consistent. The received data frame can be refreshed synchronously at the decoding terminal, thereby achieving cache consistency of each decoder. The decoder decodes and plays according to its own decoding and playback strategy, and a synchronous playback effect can be obtained. If the decoder is output by a hardware interface such as SDI, the BB synchronization signal can be used to synchronize the clock frequency of the SDI output of each decoding terminal.

[0050] The above-set multi-camera end-to-end frame synchronization method based on multiplexing decouples the front-end encoding and back-end decoding systems by adding multiplexing and demultiplexing links, so that the decoding terminal does not need to cooperate with the synchronization strategy of the encoding side and the front-end and back-end network status to achieve synchronous playback of each program frame. In the entire front-end encoding and back-end decoding system construction and network transmission construction link, it can be independently constructed according to the set requirements, and the decoding back-end does not need to configure policy parameters to achieve synchronization, reducing operation and maintenance costs. At the same time, the code stream output by the front-end multi-program multiplexing has achieved intra-code stream synchronization, which can be recorded and directly used for playback synchronization. At the same time, this solution can be used for statistical multiplexing encoding systems, so that the multi-program streams generated by the statistical multiplexing encoder can be played synchronously at the back end.

[0051] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A multi-camera end-to-end frame synchronization method based on multiplexing, characterized in that: The following steps are involved: Perform clock synchronization configuration between cameras, encoders and multiplexers, including: Perform clock synchronization configuration between cameras, encoders and multiplexers, including: Setting multiple front-end cameras to achieve clock synchronization and same-frequency refresh acquisition through the camera synchronous phase-lock interface; Setting the live encoder corresponding to each front-end camera to collect camera signals through SDI; Setting the live encoder and multiplexer to perform clock synchronization through NTP / PTP. After synchronization, the local clocks of each device are synchronized with the same UTC clock source; On the encoding side, audio and video signals are collected through multiple cameras, and the display timestamp PTS of the collected audio and video streams is the local UTC time; audio and video encoding is performed, and PTS is kept unchanged; TS single program encapsulation is performed, and the original PTS is kept; each program generates a multi-program stream in the multiplexer, and a unique program reference clock PCR is generated according to the local clock for multiplexing of each program, and then the multi-program stream is sent to the demultiplexer distributor on the decoding side through a dedicated line or network; On the decoding side, the demultiplexer splits the multi-program transport stream MPTS and aligns the audio and video code streams. The decoder multiplexer rectifies the multi-program streams and distributes them to the decoding terminal as a single program stream. The decoding terminal decodes, plays and displays them on the decoder player according to the built-in decoding synchronization playback strategy.

2. The method for end-to-end frame synchronization of multiple cameras based on multiplexing as claimed in claim 1, characterized in that: The setting of the live encoder to collect camera signals through SDI includes: the encoder collects the camera audio and video SDI signals, records the arrival time of each SDI data frame as PTS_Capture, and uses the local UTC clock to record as the display timestamp PTS of each audio and video frame.

3. The method for end-to-end frame synchronization of multiple cameras based on multiplexing as claimed in claim 1, characterized in that: Each channel of the program generates a multi-program stream in the multiplexer, and generates a unique program reference clock PCR according to the local clock for multiplexing of each channel of the program, including: generating a new PCR as a common PCR for all programs through the local clock and the preset coding delay, counting the preset coding delay as V_delay, UTC_local as the local UTC time, setting the initial PCR_base=UTC_local+V_delay, MPTS encapsulation is sent in CBR mode, the total output bit rate of multiple programs is determined by multiple single-channel encodings, multiplying a coefficient mux_coef on the basis of the sum of the multiple single-channel encoding bit rates, setting the sum of the audio and video encoding bit rates of the front-end encoder to TotalAVBitate, then the output encapsulation bit rate MPTSBitrate=mux_coef*TotalAVBitate, or the audio and video encoding bit rate of the single-channel encoder can be set by first determining the target total output bit rate; During the multiplexing process of multiple program streams, PCR generates PCR_now=PCR_base+packet_cout*188 / MPTSBitrate in real time through PCR_base and the number of packets packet_count. The TS packet length is 188, and an independent PID is used to identify the PCR data packet. At the same time, datagrams are sent and empty packets are filled according to the standard process of the multiplexer, and the PSI / SI table is updated. The valid audio and video datagrams are directly used for the newly generated MPTS code stream.

4. The method for end-to-end frame synchronization of multiple cameras based on multiplexing as claimed in claim 3, characterized in that: The PSI / SI table includes a PAT table, a PMT table, and an SDT table. The PAT table includes the PIDs of all program tables PMT. The PMT table includes the PIDs of all audio and video streams in the program, audio and video description information, and program numbers. The SDT table includes the business description information of the program. The above information is used to extract corresponding audio and video and program description information data for playback or program arrangement processing during decoding and playback.

5. The method for end-to-end frame synchronization of multiple cameras based on multiplexing as claimed in claim 3, characterized in that: The demultiplexing distributor splits the multi-program transport stream MPTS into: splitting the multi-program stream into single-program streams according to the target output decoding terminal, wherein the audio and video valid data directly use the original TS packets, and the PCR is distributed to each single-program channel by copying, and each single-program channel becomes a single-program TS code stream by updating the PSI / SI table.

6. The method for end-to-end frame synchronization of multiple cameras based on multiplexing as claimed in claim 3, characterized in that: The rectification and alignment of the audio and video code streams includes: establishing video code stream caches for multiple channels, aligning video frames frame by frame according to the timestamp PTS, and filling the gaps with TS packets with a PID consistent with the PID of the video code stream, so that the data sizes of video frames with the same PTS for all programs are completely consistent. Since the audio is encoded at a fixed bit rate, the size of each frame itself is consistent, a sending queue is established for each audio and video channel, the data queue head is aligned according to the PTS, and the audio and video frames with the same PTS are sent to the output code stream at the same time. Therefore, the split single program code stream can be output synchronously with the same PTS frame and sent synchronously. Realize synchronous refresh of audio and video frames based on PTS.

Images