System for recovering presentation timestamps from transcoders

Abstract

A method for transcoding a digital video stream includes transcoding a video stream, including a presentation timestamp for the video stream, and an audio stream, including a presentation timestamp for the audio stream, using a transcoder in a manner that modifies the presentation timestamps of the video stream such that a plurality of first values for presentation timestamps of a first set of video frames of the video stream are modified to a plurality of second values for presentation timestamps of a second set of video frames. The audio stream includes a first value for presentation timestamps embedded at a first location. The method includes determining an offset for a second value of the second set of presentation timestamps of the transcoded video stream based on the first value of the set of presentation timestamps embedded in the audio stream from the transcoder. The method includes combining the transcoded video stream and the associated audio stream based on the offset. Preferably, the transcoder also modifies the audio timestamps. Preferably, the audio stream includes the embedded first value. Preferably, the offset is determined by taking into account the difference between the audio PTS in a PES and an embedded PTS in an audio packet (private data or embedded in an audio frame).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 016,496, filed April 28, 2020, the entire contents of which are incorporated herein by reference. Background Art

[0003] The subject matter of this application relates to a system for recovering presentation timestamps from a transcoder.

[0004] Video transcoding technology using a video transcoder is the process of converting a digital video signal having an initial set of characteristics into another digital video signal having a modified set of characteristics. For example, the modified characteristics of the resulting transcoded digital video signal may include, for example, a different bit rate of the video signal, a different video frame rate, a different video frame size, different color characteristics, a different set of video encoding parameters, a different lossy video compression technique, and / or a different lossless encoding.

[0005] In many applications, such as cable broadcast systems, a full-resolution main video file is stored as a mezzanine file, which is a compressed video file that, when rendered, is typically visually indistinguishable from a rendering of the full-resolution main video file. The mezzanine file format can be any suitable format, such as an MXF file format or an MOV file format. Mezzanine files stored in the mezzanine file format are typically modified to another file format, such as an H.264 video stream, an H.265 video stream, an FLV video stream, an MPEG-1 video stream, an MPEG-2 video stream, an MPEG-4 video stream, a VC-1 video stream, a WMV video stream, a TAPE video stream, a Pores video stream, a DNxHD video stream, or a Cineform video stream, when streamed to another device.

[0006] Typically, the modified file format is provided by a video distribution server that transcodes the compressed video stream or the originally encoded mezzanine file into a format suitable for distribution to a specific user or user group. For example, a programmer for a broadcast distribution system may transcode a video stream into a format and / or bit rate suitable for distribution by a satellite transmission system to one or more users or user groups having satellite receivers. For example, a head-end system for a cable distribution system may transcode a video stream into a format and / or bit rate suitable for distribution by an integrated cable management terminal system to one or more users or user groups. For example, a video distribution server may transcode a video stream into a format and / or bit rate suitable for distribution to one or more users or user groups via the Internet.

[0007] In some embodiments, with the proliferation of different video compression standards, such as H.261, H.263, H.264, MPEG-1, MPEG-2, and MPEG-4, the need for convertibility of video streams from one digital video compression type to another and / or bit rate has steadily increased. In embodiments where a source video stream is provided to multiple users, each user utilizing a different channel with varying capabilities, the video stream is transcoded into a digital video format and / or bit rate appropriate for that particular user. For example, video conferencing systems often transmit multiple video streams, many of which are transmitted over different data channels at varying respective bit rates.

[0008] An exemplary transcoder may include a decoder, a transport port, and an encoder output. The decoder may operate in synchronization with the encoder's timestamps, as shown below. The encoder includes a master oscillator and a counter that serve as a system clock (STC). The STC is a pre-programmed, master clock for the video and audio encoder programs.

[0009] Timestamps are used to synchronize different components with each other. When a video frame or audio block is input to the encoder, the encoder samples the STC from the video frame or audio frame. A constant indicating the delay between the encoder and decoder buffers is added to the sampled STC to form a presentation timestamp (PTS). The PTS is inserted into the header of the video frame or audio frame.

[0010] When reordering video frames, a decoding time stamp (DTS) indicating when each video frame will be decoded by the decoder is inserted into the video frame. The value of the DTS used for the frame reordering process can be the same as the value of their respective PTS, including I, P, and non-referenced B pictures, and the DTS and their respective PTS can be different for I, P, and referenced B pictures. Whenever the DTS is used, the PTS is used.

[0011] According to the Advanced Television Systems Committee (ATSC) standard, a PTS or DTS is inserted into the header of each picture. The encoder buffer outputs transport packets, each with a timestamp called a Program Clock Reference (PCR), or a Packetized Elementary Stream (PES), each with a timestamp called a System Clock Reference (SCR). For MPEG, PCRs are generated at 100 millisecond intervals, for ATSC, PCRs are generated at 40 millisecond intervals, and SCRs are generated at up to 700 millisecond intervals. PCRs or SCRs are used to synchronize the decoder's STC with the encoder's STC.

[0012] The program stream (PS) has SCR as its clock reference, and the transport stream (TS) has PCR as its clock reference. Therefore, each type of video stream or audio stream has a timestamp corresponding to STC in order to synchronize the STC of the decoder with the STC of the encoder.

[0013] MPEG-based streams include time information, such as PCR or SCR for synchronizing the encoder with the decoder, and STC, PTS, and DTS for synchronizing audio content with video content. The MPEG stream is reconstructed using a decoder, and the time information is discarded after being used to synchronize the decoder with the encoder and the audio content with the video content. Unfortunately, in some cases, the timestamps are modified in an unintended manner.

[0014] Therefore, what are needed are systems and methods for improving efficient timestamp management from transcoders. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a better understanding of the invention, and to show how it may be put into practice, reference will now be made, by way of example, to the accompanying drawings, in which:

[0016] Figure 1 A transcoding system is shown.

[0017] Figure 2A An MPEG video packetized elementary stream (PES) is shown.

[0018] Figure 2B An MPEG transport stream is shown.

[0019] Figure 3 A video transcoder and an audio transcoder are shown.

[0020] Figure 4 An exemplary transcoding system with PTS recovery is shown.

[0021] Figure 5 A set of I, B and P pictures is shown.

[0022] Figure 6 The ordering of sets of I, B, and P pictures is shown.

[0023] Figure 7 An exemplary transcoding system with PTS recovery and de-jittering is shown.

[0024] Figure 8 A 3:2 pull-down technique is shown.

[0025] Figure 9 Extended presentation timestamps for two video segments are shown. DETAILED DESCRIPTION

[0026] refer to Figure 1 , shows an exemplary transcoding system 90. The transcoding system 90 includes a transcoder 100. The transcoder 100 may include a timing synchronizer 110, a decoder 120, and an encoder 130. The transcoding system 90 may also include a demultiplexer 140 and a multiplexer 150.

[0027] The demultiplexer 140 receives an input transport stream (TS) or an input program stream (PS), extracts timing parameters from the input TS or PS, and transmits the extracted timing parameters to the timing synchronizer 110. The demultiplexer 140 extracts video data previously compressed in a predetermined manner from the input TS or PS, and transmits the extracted video data to the decoder 120. In many video encoding technologies, the timing parameters include a presentation time stamp (PTS), a decoding time stamp (DTS), and a program clock reference (PCR).

[0028] A presentation timestamp is a timestamp metadata field in MPEG transport streams and other transport streams that is used to synchronize a program's individual elementary streams (e.g., video stream, audio stream, subtitle stream, etc.) when presented to a viewer. Presentation timestamps are typically given in units relative to the program's overall clock reference, such as the Program Clock Reference (PCR) or System Clock Reference (SCR), which is also sent in the transport or program stream.

[0029] The presentation timestamp typically has a resolution of 90kHz, which is suitable for presentation synchronization tasks. The PCR or SCR typically has a resolution of 27MHz, which is suitable for synchronizing the decoder's master clock with the clock of a remote decoder.

[0030] A transport stream can contain multiple programs, and each program can have its own time base. The time bases of different programs within a transport stream can be different. Because PTSs apply to the decoding of individual elementary streams, they reside in the PES packet layer of transport and program streams. End-to-end synchronization occurs when the encoder saves timestamps at capture time, propagates them to the decoder along with the associated encoded data, and when the decoder uses these timestamps to schedule presentation.

[0031] Synchronization of the decoding system with the channel is achieved through the use of SCRs in the program stream and their analogs, PCRs in the transport stream. SCRs and PCRs are timestamps for the timing of the encoded bitstream itself and are derived from the same time base used for audio and video PTS values from the same program. Because each program has its own time base, a transport stream containing multiple programs will have a separate PCR field for each program. In some cases, programs may share PCR fields.

[0032] Timing synchronizer 110 maintains the timing parameters received from demultiplexer 140 intact, allowing them to remain synchronized with segment metadata even after the video data undergoes the transcoding process. It also sends the timing parameters to encoder 130 and multiplexer 150. Decoder 120 uses a predetermined decoding method to restore the compressed video data received from demultiplexer 140 into a video sequence and provides the video sequence to encoder 130. Encoder 130 compresses the video sequence received from decoder 120 according to predetermined conditions set by transcoding parameter controller 160, records the timing parameters received from timing synchronizer 110 in the compressed video sequence, and sends the resulting compressed video sequence to multiplexer 150. Transcoding parameter controller 160 can be configured based on user input determined by the system, such as from a GUI, and / or based on a specific application. Transcoding parameter controller 160 determines transcoding conditions appropriate for the end-user environment and provides these conditions to encoder 130 and timing synchronizer 110. Transcoding conditions include, for example, video quality, video resolution, bitrate, and video frame rate. The multiplexer 150 multiplexes the video sequences received from the encoder 130 to generate an output TS or PS. The multiplexer 150 records the timing parameters received from the timing synchronizer 110 in the header of the output TS or PS. The segment metadata may have been extracted from the input TS or PS by the demultiplexer 140, or may have been provided by another metadata provider. Similarly, other transcoders may be used as needed.

[0033] For convenience only, this discussion will be described using TS instead of PS as an example. For example, in the following paragraphs, only PCR will be described as a reference time indicator, but in the case where the stream input to or output from the transcoding system 200 is a PS, SCR can also be used as a reference time indicator. Even if PCR is input to the transcoding system 200 as a reference time indicator, SCR can be output from the transcoding system 200 as a reference time indicator, and vice versa.

[0034] refer to Figure 2A and Figure 2B , respectively showing MPEG-2 packetized elementary stream (PES) packets and MPEG-2 TS. Figure 2A , an MPEG-2 video stream compressed using an encoder is packetized into PES packets. Each PES packet includes an optional PES header and a PES packet data field. The optional PES header 200 includes an optional field 210. The optional field 210 includes a PTS field 220 and a DTS field 230. PTS information is recorded in the PTS field 220, and DTS information is recorded in the DTS field 230.

[0035] refer to Figure 2B, the TS formed by the multiplexing process is 188 bytes long and includes a header 240 and a payload 250. A PES packet or a program association table (PAT) or a program map table (PMT) is contained in the payload 250. The header 240, which begins with a synchronization byte, includes various fields such as an adaptation field 260. The adaptation field 260 includes an optional field 270, and the optional field 270 includes a PCR field 280. PCR information is recorded in the PCR field 280 as reference time information.

[0036] refer to Figure 3 In some cases, it is desirable to use a specific video transcoder 300 to transcode the video stream 330 of a digital video stream (e.g., a packetized elementary stream) 320, and to use a separate audio transcoder 310 to transcode the audio stream 340 of the digital video stream (e.g., a packetized elementary stream) 320. The video transcoder 300 typically also includes the ability to process the corresponding audio stream and maintain synchronization between the received video and audio streams. For example, the video transcoder 300 may have desired transcoding characteristics, such as very high-quality video encoding with a relatively low bit rate. For example, the video transcoder 300 may have other desired transcoding characteristics, such as software applications suitable for running on common off-the-shelf servers (e.g., servers in public and / or private data centers). For example, the audio transcoder 310 may have desired transcoding characteristics, such as a reduced bit rate or a reduced sampling rate. In this way, a video stream from the digital video stream can be provided to the video transcoder 300, and an audio stream from the digital video stream can be provided to the audio transcoder 310. The output of the video transcoder 300 and the output of the audio transcoder 310 can be combined 350, for example, into a packetized elementary stream 360. Unfortunately, due to the video transcoding process, the video transcoder often modifies the value of the presentation timestamp in such a way that the presentation timestamp associated with a group of video frames is different from the presentation timestamp associated with the same group of video frames generated by the video transcoder (e.g., the relative difference between the corresponding presentation timestamps is modified). Automatically resynchronizing an audio stream, such as that initially provided with a digital video stream or such as that transcoded by an audio transcoder, is problematic because the presentation timestamps of the video stream have been modified in an unknown manner, and thus synchronization between the audio and video streams has been lost. If the audio stream is not properly aligned with the video stream, the audio content will not match the corresponding video content. It should be noted that the PTS of the video stream and the corresponding PTS of the audio stream are typically different numbers, but the known difference between the different PTS numbers (e.g., video and audio streams) provides synchronization.

[0037] refer to Figure 4, it is determined that because the video transcoder 400 tends to modify the presentation timestamps encoded with the video stream from the presentation timestamps provided at its input to the modified presentation timestamps provided at its output, it is desirable to somehow recover the original presentation timestamps provided to the input of the video transcoder 400. Typically, the video transcoder 400 includes the ability to process an input audio stream 402 and an input video stream 404 (e.g., provided separately or otherwise encoded together as packetized elementary streams) while maintaining synchronization of the video stream 404 and the corresponding audio stream 402 at its output. Understanding that synchronization of the video stream 404 and the corresponding audio stream 402 is maintained by the video transcoder 400 during the transcoding process, it is determined that the presentation timestamps of the input video stream 404 can be encoded in a suitable manner within the input audio stream 402 that remains unchanged through the video transcoder 400. In the case where the presentation timestamp of the input video stream 404 is embedded within the audio stream 402, such that the presentation timestamp remains unchanged due to the transcoding process, it can be used to resynchronize the video stream 404 with the original audio stream 414 of the digital video stream 410, which includes both the video stream 412 and the audio stream 414. The video stream 412 is provided to the video transcoder 400 as the input video stream 404.

[0038] The video PTS extraction process 420 can process the video stream 412 of the digital video stream 410 to extract the presentation timestamp associated with the video stream 412. The recorded audio stream 430 is provided by the system to the presentation timestamp embedding process 432, which also receives the extracted presentation timestamp associated with the video stream 412. The presentation timestamp extracted from the video stream 412 is embedded within the recorded audio stream 430 by the presentation timestamp embedding process 432. The presentation timestamp embedding process 432 embeds the presentation timestamp within the recorded audio stream 430 in a manner that remains unchanged through the transcoding process of the video transcoder 400 and provides the input audio stream 402. The input audio stream 402 is synchronized with the input video stream 404 (e.g., provided separately or otherwise encoded together as a packetized elementary stream) and provided to the video transcoder 400 using the multiplexer 403. The video transcoder 400 provides an output transport stream 440 that includes both a transcoded output video stream 442 and an output audio stream 444. The output audio stream 444 includes an embedded presentation timestamp that remains unchanged through the transcoding process.

[0039] For example, the presentation timestamp embedded in the separate audio stream 430 may be encoded within the private data portion of the encoded data stream. The private data portion may include, for example, one or more of: (1) Transport Stream Packet Table 2-2; (2) Transport Stream Adaptation Field Table 2-6; (3) Packetized Elementary Stream Packet Table 2-17; (4) Packetized Elementary Stream Packet Header; (5) Packetized Elementary Stream Packet Data Bytes Field; (6) Descriptors within the program stream and / or transport stream; and (7) Private Segment Table 2-30.

[0040] Presentation timestamp offset determination process 450 receives transcoded output video stream 442 and output audio stream 444 and extracts presentation timestamps from the output video stream and embedded presentation timestamps and normal presentation timestamps from output audio stream 444. In this way, three different presentation timestamps can be extracted from the data obtained by video transcoder 400. Comparison of the presentation timestamp from recorded audio stream 404 with the presentation timestamp embedded within input audio stream 402 provides an offset 452 between the two presentation timestamps, which corresponds to the offset between the timestamps of transcoded output video stream 442 and video stream 412. Offset 452 is added to the presentation timestamp of transcoded output video stream 442 by PTS offset adjustment process 454 to provide a transcoded video stream having adjusted presentation timestamps 460. Offset 452 can also be used to adjust the program clock reference of transcoded output video stream 442. Offset 452 can also be used to adjust the decoding timestamp of transcoded output video stream 442. If desired, the output audio stream 444 after the extraction timestamp may be discarded.

[0041] An audio transcoder 470, if included, is used to transcode an audio stream 414 of a digital video stream (e.g., a packetized elementary stream) 410. The output of the audio transcoder 470 may be combined 472 with the transcoded video stream having the adjusted presentation timestamps 460, e.g., into a packetized elementary stream 474. Furthermore, the audio stream 414 may be passed 471 (which may include a buffer) to the combiner 472.

[0042] In another embodiment, recorded audio stream 430 may be replaced by audio stream 414 in which the presentation timestamps from video stream 412 are embedded in a manner that is not modified by the transcoding process of video transcoder 400. If desired, the audio stream from the transcoding process of video transcoder 400 may be discarded after the timestamps are extracted.

[0043] In another embodiment, the video transcoding process can include a 3:2 pulldown technique so that there is no one-to-one match between the video frames entering the video transcoder and the video frames leaving the video transcoder. The 3:2 pulldown technique converts 24 frames per second to 29.97 (or 30) frames per second. Typically, when converting 24 frames per second to 29.97, this results in every 4 frames being converted to 5 frames, plus a slight slowdown.

[0044] Preferably, the input video stream is not modified to include a presentation timestamp in the private data section of the video transcoder to reduce the possibility of introducing errors. Furthermore, there may not be available space to include the presentation timestamp in the input video stream. Furthermore, in some cases, the transcoder may discard the private data field when generating the output PES header. Alternatively, the input video stream may be modified to include a presentation timestamp in a private data section that is not modified by the video transcoder.

[0045] Preferably, the input audio stream is not modified to include a presentation timestamp in the private data section of the video transcoder to reduce the possibility of introducing errors. Furthermore, in some cases, the transcoder may discard the private data field when generating the output PES header. Furthermore, there may not be space available to include a presentation timestamp in the audio stream of the digital video stream.

[0046] refer to Figure 5 Many encoding schemes for video streams include intra frames (i.e., I-frames), which are pictures compressed based only on information within the frame. A video stream may also include predicted frames (i.e., P-frames), which are pictures predicted at least in part from a previous I or P-frame. A video stream may also include bidirectionally predicted frames (i.e., B-frames), which use past and future I and P-frames for motion compensation. Other types of frames may be used, depending on the type of encoding scheme used.

[0047] refer to Figure 6 In order for the decoder to reconstruct a B frame from the preceding I and following P frames, both the I frame and the following P frame must arrive before the B frame. Therefore, the order in which the frames are transmitted is different from the order in which they appear when presented. Using a decoding timestamp to tell the decoder when to decode a frame and a presentation timestamp to tell the decoder when to render a frame adapts the decoding and rendering timing of the frames.

[0048] The Advanced Television Systems Committee (ATSC) for MPEG-2 offers a resolution of 1920x1080 with progressive video at frame rates of 23.976, 24, 29.97, or 30 frames per second. ATSC offers MPEG-2 a resolution of 1920x1080 with interlaced video at frame rates of 29.97 frames per second (59.94 fields) or 30 frames per second (60 fields). ATSC offers MPEG-2 a resolution of 1280x720 with progressive video at frame rates of 23.976, 24, 29.97, 30, 59.94, or 60 frames per second. ATSC offers MPEG-2 a resolution of 704 / 858x480 with progressive video (SMPTE259M) at frame rates of 23.976, 24, 29.97, 30, 59.94, or 60 frames per second. ATSC provides MPEG-2 with a resolution of 704 / 858x480 and an interlaced video (SMPTE259M) frame rate of 29.97 frames per second (59.94 fields) or 30 frames per second (60 fields). ATSC provides MPEG-2 with a resolution of 640x480 and progressive video frame rates of 23.976, 24, 29.97, 30, 59.94, or 60 frames per second. ATSC provides MPEG-2 with a resolution of 640x480 and an interlaced video frame rate of 29.97 frames per second (59.94 fields) or 30 frames per second (60 fields). ATSC also supports other PAL frame rates and resolutions, and supports the H.264 video codec with other frame rates and resolutions.

[0049] For example, for MPEG-2 progressive video at a frame rate of 29.97 frames per second and a resolution of 1920×1080, the presentation timestamp increments by 3003 between frames (with a 90 kHz clock resolution) when incremented appropriately. Similarly, for H.264 interlaced video at a frame rate of 59.94 fields per second and a resolution of 1920×1080 with field-coded pictures, the presentation timestamp increments by 1501 / 1502 / 1501 / 1502 / ... (with a 90 kHz clock resolution) when incremented appropriately. Note that 1501+1502 (for two consecutive fields) equals 3003, the frame rate. Therefore, the presentation timestamp should increment uniformly and consistently between frames or fields.

[0050] A video transcoder modifies the presentation timestamp associated with a particular video frame of video content between its input and its output, tending to create a modified presentation timestamp with an offset of 1. This process of presentation times deviating from the preferred value tends to continue over time. Many rendering devices and associated decoders will tend to decode and render frames properly even if the presentation timestamp value is jittery. Unfortunately, when there is sufficiently large jitter in the presentation timestamp value, some rendering devices and associated decoders may tend to incorrectly decode and render frames properly. Furthermore, since the decoding timestamp is typically the same as the presentation timestamp for I and P frames, and is appropriately modified for B frames, if the presentation timestamp includes jitter in its value, the decoding timestamp will also include jitter in its value. The video transcoder may introduce jitter into the presentation timestamp, and / or the determination of the offset (described previously) may introduce jitter into the presentation timestamp. In either case, it is desirable to reduce the amount of jitter in the presentation timestamps (including the decoding timestamp) to reduce the likelihood of decoding and / or rendering failures of the video content.

[0051] refer to Figure 7 , it is determined that because the video transcoder 700 tends to modify the presentation timestamps encoded with the video stream in a manner that includes jitter in its value from the presentation timestamps provided at its input to the modified presentation timestamps provided at its output, it is desirable to remove or otherwise reduce the amount of jitter in the resulting presentation timestamps. Typically, the video transcoder 700 includes the ability to process an input audio stream 702 from a recorded audio stream 707 that uses the most recent video PTS for the audio PES (PTS) header and to place the most recent video PTS in audio frames (or private data) and input video stream 704 (e.g., provided separately or otherwise encoded together as a packetized elementary stream) using a multiplexer 703, while keeping the video stream 704 and the corresponding audio stream 702 synchronized at its output.

[0052] The digital video stream 710 includes both a video stream 712 and an audio stream 714. The video stream 712 of the digital video stream 710 is provided to the video transcoder 700 as the input video stream 704. The audio stream 714 of the digital video stream 710 is provided to the video transcoder 700 as the input audio stream 702.

[0053] Since the video transcoder modifies presentation timestamps, it is desirable to read presentation timestamps from the video stream 712 provided to the video transcoder 700 by the video PTS and DTS extraction process 720. Since the video transcoder 700 modifies decoding timestamps, it is also desirable to read decoding timestamps from the video stream 712 provided to the video transcoder 700 by the video PTS and DTS extraction process 720. A period of presentation timestamps and decoding timestamps is stored in table 730. Table 730 preferably includes a defined time window for retaining data, such as 10 seconds. In this way, when new presentation timestamps and decoding timestamps are added to table 730, older presentation timestamps and decoding timestamps are removed from table 730.

[0054] An input audio stream 702 is synchronized with a video stream 704 (e.g., provided separately or otherwise encoded together as a packetized elementary stream) and provided to a video transcoder 700. The video transcoder 700 provides an output transport stream 740, which includes both a transcoded output video stream 742 and an output audio stream 744. Generally speaking, jitter adjustment can be performed as follows (described in more detail below). For the output audio stream 744, it includes a PES header that includes both the original PTS and a jittered PTS from the video transcoder 700, the difference being the offset. The offset is subtracted from (or added based on a difference calculation method for) the video PTS / DTS / PCR. Excluding the jitter, the output PTS+offset corresponds to the input PTS in the table. The system determines the closest input PTS that matches the output PTS+offset. Note that the system also adds the offset to the PCR so that both the PTS / DTS and PCR are adjusted to the same offset.

[0055] Presentation timestamp jitter determination process 750 receives the transcoded output video stream 742 and output audio stream 744 and extracts jittered presentation timestamps from the PES headers of the audio and video, as well as the original video PTS embedded in the audio stream. Output audio stream 744 includes a PES header that includes both the original PTS and the jittered PTS from video transcoder 700, with the difference being the offset. Except for the jitter, the output video PTS + offset corresponds to the input video PTS in the table. Presentation timestamp jitter determination process 750 compares the video PTS + offset presentation timestamp with the extracted presentation timestamps included in PTS and DTS table 730. Based on a match between the video timestamp calculated using the offset generated from output audio stream 744 and the extracted presentation timestamps included in PTS and DTS table 730, presentation timestamp jitter determination process 750 determines the presentation timestamp from table 730 that most closely matches. Timestamp update process 760 modifies the presentation timestamp in transcoded output video stream 742 to the matching presentation timestamp in table 730 identified by presentation timestamp jitter determination process 750 .

[0056] The presentation timestamp jitter determination process 750 may also retrieve a matching decode timestamp from the table 730 based on the matching presentation timestamp. The timestamp update process 760 may also modify the decode timestamps in the transcoded output video stream 742 to the matching decode timestamps in the table 730 identified by the presentation timestamp jitter determination process 750.

[0057] An audio transcoder 770, if included, is used to transcode an audio stream 714 of a digital video stream (e.g., a packetized elementary stream) 710. The output of the audio transcoder 770 may be combined 772 with the transcoded video stream having adjusted presentation and decoding timestamps 762, e.g., into a packetized elementary stream 474. Furthermore, the audio stream 714 may be passed 771 (which may include a buffer) to the combiner 772.

[0058] As previously mentioned, the transcoded video stream from the video transcoder 700 often includes some jitter, especially when the video frames from the input and output do not have a one-to-one correlation. The lack of one-to-one correlation primarily occurs when video transcoding modifies the field rate and / or frame rate of the video stream.

[0059] As mentioned above, one of the frame rate conversions is the 3:2 pull-down technique, which converts 24 frames per second to 29.97 (or 30) frames per second. Figure 8 , shows a set of frames 800 representing 24 frames per second film and a set of fields 810 representing 30 frames per second (60 fields per second) video. Typically, the first frame 802 is transferred to three fields 812 in the set of fields 810. Typically, the second frame 804 is transferred to two fields 814 in the set of fields 810. Typically, the third frame 806 is transferred to three fields 816 in the set of fields 810. Typically, the fourth frame 808 is transferred to two fields 814 in the set of fields 810. In this way, the process can be repeated for the video transcoder 700. It should also be noted that the fields used for the frames alternate their field selection. For example, frame 802 includes field 1 / field 2 / field 1, while frame 806 includes field 2 / field 1 / field 2. For example, frame 804 includes field 2 / field 1, while frame 808 includes field 1 / field 2.

[0060] With video content at 23.98 frames per second, the presentation timestamps should differ by 3754 / 3754 / 3754 / 3753 between frames (at a clock resolution of 90kHz) when incremented appropriately. As a result of the 3:2 pulldown process, fields 810 should have presentation timestamps that are offset based on the presentation timestamps of each frame 800. For example, frame 802 should produce 3 fields 812, so the presentation timestamps of the 3 fields 812 should be offset by 1502 / 1501 / 1502. For example, frame 804 should produce 2 fields 814, so the presentation timestamps of the 2 fields 814 should be offset by 1501 / 1502. In addition to the jitter of the presentation timestamps of the fields from the video transcoder 700, which may match those of the frames in a one-to-one manner, it is also possible that the jitter of the presentation timestamps in the remaining fields of the conversion process does not match those of the frames in a one-to-one manner.

[0061] To accommodate the possibility that jitter in a field does not match jitter in a frame, such as a result of a 3:2 pulldown technique, table 700 may also be extended to generate additional presentation timestamps for frame 800. For example, for frame 802, corresponding presentation timestamps may be provided in table 700 for field 2 and field 1, e.g., by incrementing the presentation timestamp for frame 802 by 1501 and by 1501+1502, respectively. For example, for frame 806, corresponding presentation timestamps may be provided in table 700 for field 1 and field 2, e.g., by incrementing the presentation timestamp for frame 806 by 1502 and by 1502+1501, respectively. For example, for frame 804, corresponding presentation timestamps may be provided in table 700 for field 1, e.g., by incrementing the presentation timestamp for frame 804 by 1501. For example, for frame 808, corresponding presentation timestamps may be provided in table 700 for field 2, e.g., by incrementing the presentation timestamp for frame 808 by 1502. In a similar manner, to accommodate the possibility that the jitter in a field does not match the jitter of a frame, table 700 may also be extended to generate additional decoding timestamps for frame 800 .

[0062] The presentation timestamp jitter determination process 750 can retrieve a matching presentation timestamp from the extended table 730. The timestamp update process 760 can also modify the presentation timestamps in the transcoded output video stream 742 to the matching presentation timestamps in the extended table 730 identified by the presentation timestamp jitter determination process 750. The presentation timestamp jitter determination process 750 can also retrieve a matching decoding timestamp from the extended table 730 based on the matching presentation or decoding timestamps. The timestamp update process 760 can also modify the decoding timestamps in the transcoded output video stream 742 to the matching decoding timestamps in the extended table 730 identified by the presentation timestamp jitter determination process 750. In this way, the presentation timestamps and decoding timestamps can be updated accordingly to reduce jitter even if corresponding frames do not exist in the source video content.

[0063] Typically, a video stream includes multiple video clips that are streamed together in a serial manner. Due to the multiple video clips being streamed together, the presentation timestamps between the corresponding video clips often include discontinuities. Such discontinuities in presentation timestamps are also observed when a video clip wraps around to its end during the serial presentation.

[0064] Unfortunately, video transcoder 700 often processes the input video stream in a manner in which any discontinuities in presentation timestamps typically associated with different video segments cause the discontinuities in presentation timestamps in the transcoded video stream to not align with the discontinuities in presentation timestamps of the input video stream. Consequently, the presentation timestamp of a first video segment in the transcoded video stream may sequentially extend into a portion of a second video segment that temporally follows the first video segment. Consequently, the presentation timestamp of a second video segment in the transcoded video stream may sequentially extend into a portion of the first video segment that temporally precedes the second video segment.

[0065] Unfortunately, when attempting to modify the resulting video stream to account for jitter and modifying the resulting video stream to account for offsets in presentation timestamps, it can be difficult to accurately determine the correct location of the discontinuity based on the presentation timestamps of the input video stream. Furthermore, if the presentation timestamp from the video transcoder appears to be erroneous, the frame associated with the presentation timestamp is often discarded due to the error. Furthermore, when attempting to insert advertisements into the transcoded video stream, inserting advertisements into discontinuities between segments is problematic because discontinuities in presentation timestamps do not necessarily match discontinuities in video frames.

[0066] To accommodate the possibility that presentation timestamps may not match properly in discontinuous regions, table 700 may be further extended to generate additional presentation timestamps in a series of presentation timestamps for frames proximate to those discontinuous regions 800. A discontinuity in presentation timestamps may be determined based on an expected sequence of increments of presentation timestamps being significantly different from expected, e.g., a difference of greater than 5%.

[0067] refer to Figure 9, table 700 is extended to include a series of additional presentation timestamps. A first video segment 900 may include an associated set of video frames and a presentation timestamp 910. A second video segment 920 may include an associated set of video frames and a presentation timestamp 930. A discontinuity 940 exists between the first video segment 900 and the second video segment 920, which is also represented as a discontinuity in the presentation timestamps of the first video segment 900 and the second video segment 920. When a discontinuity is identified in the presentation timestamps, for example, by a sequence of presentation timestamps including a sufficiently large discontinuity, table 700 is extended with additional presentation timestamps. It is determined that the presentation timestamps 950 of the first extended series actually extend (forward) the presentation timestamp of the first video segment 900, while the presentation timestamps 950 of the first extended series do not actually correspond to any video frame. It is determined that the presentation timestamps 960 of the second extended series actually extend (backward) the presentation timestamp of the second video segment 920, while the presentation timestamps 960 of the second extended series do not actually correspond to any video frame. The result is a set of presentation timestamps 970 for the first video segment 900 and a set of presentation timestamps 980 for the second video segment 920.

[0064] By way of example, the duration of the first extended series of presentation timestamps 950 and the second extended series of presentation timestamps 960 may be 1 second.

[0068] By extending table 700 to include additional presentation timestamps, these presentation timestamps may be used with the jitter reduction process and / or the incremental presentation timestamp determination process for accurate adjustments.

[0069] The offset process, the dither process and / or the discontinuity can be combined with each other as needed.In addition, the table can be in any format or manner, including any data structure or stored in a memory or storage device in other ways.

[0070] In addition, each functional block or various features in each of the foregoing embodiments can be implemented or performed by a circuit, which is typically an integrated circuit or multiple integrated circuits. The circuits designed to perform the functions described in this specification may include a general-purpose processor, a digital signal processor (DSP), a dedicated or general-purpose integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic, or discrete hardware components or a combination thereof. The general-purpose processor can be a microprocessor, or the processor can be a conventional processor, a controller, a microcontroller or a state machine. The general-purpose processor or each circuit can be configured by a digital circuit, or can be configured by an analog circuit. In addition, when technologies for manufacturing integrated circuits that replace current integrated circuits emerge due to advances in semiconductor technology, integrated circuits manufactured using these technologies can also be used.

[0071] It should be understood that the present invention is not limited to the particular embodiments described and that changes may be made therein without departing from the scope of the invention as defined in the appended claims, as interpreted according to principles of prevailing law, including the doctrine of equivalents or any other doctrine that expands the enforceable scope of a claim beyond its literal scope. Unless the context indicates otherwise, a reference in a claim to the number of instances of an element (whether to one instance or more than one instance) requires at least the specified number of instances of the element, but is not intended to exclude from the scope of the claim a structure or method having more than the number of instances of the element. When used in a claim, the word "comprise" or its derivatives is used in a non-exclusive sense that is not intended to exclude the presence of other elements or steps in the claimed structure or method.

Claims

1. A method for transcoding a digital video stream, comprising: (a) receiving a digital video stream including an input video stream and an input audio stream as an input audio and video stream; (b) extracting a first set of presentation timestamps from the input video stream; (c) embedding the first set of presentation timestamps at a first position in a first audio stream, wherein the first audio stream is different from the input audio / video stream; (d) providing the input video stream and the first audio stream to a video transcoder and providing the input audio stream to an audio transcoder in synchronization with each other; (e) transcoding, by the video transcoder, the input video stream including the first set of presentation timestamps from an initial set of characteristics to modified set of characteristics including a second set of presentation timestamps, the second set of presentation timestamps being different from the first set of presentation timestamps, and providing the transcoded input video stream and the first audio stream from the video transcoder in synchronization with each other; (f) determining offsets for the second set of presentation timestamps of the transcoded input video stream based on the first set of presentation timestamps embedded in the first audio stream from the video transcoder; (g) combining the transcoded input video stream and the transcoded input audio stream into an output audio-video stream without the first audio stream based on the offset.

2. The method according to claim 1, wherein The input video stream comprises video frames and the input audio stream comprises audio frames, wherein the input video stream and the input audio stream are received as input packetized elementary streams.

3. The method according to claim 1, wherein The first location includes at least one of the following: (1) a transport stream packet table; (2) a transport stream adaptation field table; (3) a packetized elementary stream packet table; (4) a packetized elementary stream packet header; (5) a packetized elementary stream packet data byte field; (6) Descriptors within the program stream; (7) Descriptors within the transport stream; and (8) Private Segment Table.

4. The method according to claim 1, wherein The first audio stream is not transcoded by the video transcoder.

5. The method according to claim 1, wherein The first audio stream is transcoded by the video transcoder.

6. The method according to claim 1, wherein The transcoded input video stream includes video frames, and the first audio stream includes audio frames.

7. The method according to claim 1, wherein The combining of the transcoded input video stream and the transcoded input audio stream based on the offset is a packetized elementary stream.

8. The method according to claim 1, wherein The transcoded input video stream comprises video frames and the transcoded input audio stream comprises audio frames, wherein the transcoded input video stream and the transcoded input audio stream are provided as output packetized elementary streams.

Images