Audio and video transcoding transport stream multiplexing method

Abstract

The application discloses an audio and video transcoding transmission stream multiplexing method. In the audio and video transcoding process, audio and video decoding is re-encoded to form an audio and video code stream, and then the audio and video code stream enters a multiplexer. The multiplexer outputs TS packets one by one to form a TS transmission stream through the following steps. S11: if the difference between the current system time and the time of the latest audio packet output is greater than the audio packet time interval, the current audio packet is output, and the step S13 is executed. Otherwise, the audio packet output is stopped, and the step S12 is executed. S12: the time difference between the DTS time of the current video frame and the reference PCR time is calculated. If the time difference is less than a video abnormal threshold value, the current video packet is output. Otherwise, video frame abnormality detection is executed. According to the abnormality detection result, the current video frame is marked as abnormal, and the current video packet is output. S13: the DTS and PTS times of the audio frame, the actual PCR time and the reference PCR time are adjusted and updated.

Description

Technical Field

[0001] This invention belongs to the technical field of audio and video transcoding and transmission, and specifically relates to a method for multiplexing audio and video transcoding and transmission streams. Background Technology

[0002] The MPEG-2 system is a specification that combines video, audio, and other elementary streams (ES) into one or more data streams for storage or transmission. The ES is then packetized to output a Packetized Elementary Stream (PES). PES packets are of variable length; video packets are typically one frame (one image) per packet; audio packets are typically one audio frame long, not exceeding 64KB. After being multiplexed, PES can be further packaged into two different streams: Program Stream (PS) and Transport Stream (TS).

[0003] PS is generally used for program storage, while TS is used for transmission across various channels. Whether in cable TV, satellite TV, or IPTV, video and audio streams are multiplexed together in the transport stream for transmission. Current applications such as terrestrial digital TV, cable TV, and satellite TV require a strictly constant bitrate for the transport stream.

[0004] In digital compression systems, the amount of data in each frame varies, depending on the image encoding method and the complexity of the image content. Since the transmission channel latency is typically fixed, the transmission latency for each frame of a moving image is variable. This means there is no natural synchronization between transmission and display. Clearly, unlike analog television transmission systems where image information is transmitted synchronously and the receiver can directly obtain the clock signal from the image synchronization signal to control the display, digital transmission systems are fundamentally different.

[0005] The timing model of the MPEG-2 system was established to solve the problem of irregular time delays mentioned above. It is a model with a constant time delay between the encoding input and decoding output. This model achieves constant delay by making the delay of each encoder and decoder buffer variable. To achieve variable buffer delay, the MPEG-2 system uses relevant clock information in the three bitstream layers of ES, PES, and TS, namely VBV-delay, Presentation Time Stamp (PTS), Decoding Time Stamp (DTS), and Program Clock Reference (PCR). Through their joint control, synchronization of encoding and decoding, as well as synchronization of audio and video output display, are achieved.

[0006] Since PTS and DTS are essential information for ensuring accurate audio and video synchronization, the first priority in transport stream multiplexing is to ensure that the audio and video PTS and DTS remain synchronized. However, due to bit errors and data loss in many cases, the audio and video PTS and DTS are only synchronized for a certain period of time, and under abnormal circumstances, the values ​​of PTS and DTS may fluctuate significantly. At the same time, during the transcoding process, bit errors and data loss in the original bitstream, as well as different transcoding delays of audio and video, cause the PTS and DTS of the transcoded audio and video frames to be misaligned. As a result, when the audio and video enter the multiplexer, it cannot be guaranteed that each frame of audio and video is completely synchronized. In addition, it is necessary to consider that the multiplexing buffer cannot be too large, otherwise it will cause excessive latency. Summary of the Invention

[0007] This invention provides a method for multiplexing audio and video transcoding transmission streams, which solves the technical problems in existing multiplexing methods, such as large fluctuations in video bitrate, abnormal audio and video PTS and DTS, packet loss in audio and video streams, and asynchrony caused by data loss during decoding.

[0008] This invention can be achieved through the following technical solutions:

[0009] An audio / video transcoding transport stream multiplexing method involves decoding and re-encoding audio and video data to form an audio stream and a video stream, respectively. The audio and video streams then enter a multiplexer, which outputs TS packets sequentially through the following steps to form a TS transport stream.

[0010] Step 1: Output the current audio / video packet

[0011] S11. If audio information exists in the audio buffer, and the difference between the current system time and the time of the most recent audio packet output is greater than the audio packet time interval T. a If the output is positive, output the current audio packet and proceed to step S13; otherwise, stop outputting the audio packet and proceed to outputting the current video packet in step S12.

[0012] S12. If video information exists in the video buffer, calculate the time difference between the DTS time of the current video frame and the reference PCR time. If the time difference is less than the video anomaly threshold T2, output the current video packet.

[0013] Otherwise, perform video frame anomaly detection, mark the current video frame as abnormal based on the anomaly detection results, and output the current video packet;

[0014] S13. Adjust the DTS and PTS times of audio and video frames and update the actual PCR time and reference PCR time.

[0015] Step 2: Output the next audio / video packet

[0016] Repeat steps S11-S13 to complete the output of the next audio / video packet. However, when executing step S12, if the current video frame has an anomaly marker, the current video packet is output directly based on the anomaly marker; otherwise, all steps of step S12 are executed.

[0017] Furthermore, when performing video frame anomaly detection, if the deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer increases as the DTS time of each audio frame in the audio buffer increases, then the current video packet is output, and the current video frame is marked as anomaly state I.

[0018] Otherwise, determine whether the minimum deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer is greater than the audio / video anomaly threshold T3.

[0019] If so, continue the evaluation:

[0020] i. If the minimum difference between the DTS time of the current video frame in the video buffer and the DTS time of each audio frame in the audio buffer is less than or equal to the audio / video buffer abnormal threshold T4, then output the current video packet and mark the current video frame as abnormal state I.

[0021] ii. If the minimum difference is greater than the audio / video buffer anomaly threshold T4, then determine: if the minimum deviation between the DTS time of the latest decoded video frame in the transcoding system and the DTS time of each audio frame in the audio buffer is less than or equal to the audio / video anomaly threshold T3, then output the current video packet and mark the current video frame as anomaly state I; otherwise, output the current video packet and mark the current video frame as anomaly state II.

[0022] If not, continue the judgment:

[0023] If an audio frame in the audio buffer has a DTS time greater than 1.25T3 than the DTS time of the current video frame, the corresponding audio frame is deleted and the current video packet is output; otherwise, the output of the current video packet is stopped.

[0024] Furthermore, during step S11, the DTS time difference T between two consecutive audio frames in the audio buffer is first calculated. d If the DTS time difference T d If the audio time exceeds the audio anomaly threshold T1, it is determined that the audio DTS time is abnormal. The DTS time of the current audio frame should be used as the reference PCR time before proceeding to the other steps of step S11; otherwise, proceed directly to the other steps of S11.

[0025] Furthermore, in step S13, the PCR deviation P is obtained by subtracting the actual PCR time from the reference PCR time. d Write the actual PCR time into the current TS packet stream; then subtract the PCR bias P from the DTS and PTS time values ​​of the audio and video. d The adjusted DTS and PTS time values ​​are obtained and written into their PES packet header bitstreams to achieve the adjustment of DTS and PTS times for audio and video frames.

[0026] After each TS packet is output, the reference PCR time and the actual PCR time should be increased by the TS packet time interval value Tt to update the actual PCR time and the reference PCR time.

[0027] The beneficial technical effects of this invention are as follows:

[0028] 1. The method of the present invention uses the DTS time of the audio as the reference PCR time during multiplexing, and arranges the DTS time of the video according to the reference PCR time, thereby maintaining the synchronization of audio and video.

[0029] 2. The method of the present invention takes into account the characteristic that audio encoding has a shorter latency than video encoding in audio-video transcoding, and the multiplexing requires a small video buffer, thus effectively reducing video latency.

[0030] 3. The method of the present invention is designed to handle various abnormal situations, readjust and rearrange the PTS and DTS times of audio and video, maintain the correctness of the multiplexed bitstream, improve the robustness of the multiplexer, and can be applied to the multiplexing and transmission of various disordered PTS and DTS time audio and video bitstreams. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0032] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0033] During audio and video stream transcoding, MPEG-2 TS demultiplexing and decoding are performed first, followed by audio and video encoding, and finally, the audio and video are multiplexed back into a TS stream. During video transcoding, errors and packet loss during TS stream transmission cause timing errors in some frames after video decoding, creating difficulties for subsequent encoding and multiplexing. Furthermore, video decoding starts from the keyframe I of the video stream; all previous frames cannot be decoded and must be discarded, resulting in video image timing loss. However, audio decoding does not require discarding audio data, thus avoiding audio timing loss. This difference in timing between transcoded audio and video leads to different audio and video timing losses. If timing matching is not performed during audio and video multiplexing, audio and video will become out of sync. Additionally, due to the significant variations in ultra-high-definition video scene content and current limitations in implementation capabilities, the bitrate of the stream generally fluctuates considerably to meet video quality requirements, easily causing buffer overflow and underflow during multiplexing.

[0034] The above analysis shows that the audio decoding data is lossless, has low encoding latency, and a constant bitrate, thus it can be used as a reference PCR time for multiplexing. During transcoding, the reference PCR time is constantly tracked by the audio's DTS time. Fluctuations in the audio's DTS time are used to determine if any anomalies occur in the audio and video streams. If an anomaly occurs, the audio's DTS time remains unchanged, and the reference PCR time is updated with the new audio DTS time. Simultaneously, the video's DTS time is arranged according to the reference PCR time, maintaining audio and video synchronization. Finally, based on the difference between the reference PCR time and the actual PCR time, the audio and video's DTS and PTS times are adjusted and written into the stream, ensuring the correctness of the multiplexed stream and generating a transport stream with good audio-video synchronization performance. This effectively solves the problem of transport stream multiplexing in ultra-high-definition video transcoding under various harsh environments.

[0035] like Figure 1 As shown, the present invention provides an audio and video transcoding transport stream multiplexing method. In the audio and video transcoding process, the audio and video are decoded and re-encoded to form an audio stream and a video stream. The audio stream and the video stream enter a multiplexer. The multiplexer outputs TS packets one by one through the following steps to form a TS transport stream.

[0036] Before explaining the specific steps, let's define the following:

[0037] In audio / video transcoding, the video bitrate is set to R. V The audio bitrate is set to R. a TS bitrate set to T r The actual PCR time was set as RCR. f The reference PCR time is set as PCR. ref .

[0038] The audio packet time interval is set to T.a Calculated as in formula (1):

[0039]

[0040] In the formula P a It is the audio packet load.

[0041] The video packet time interval is set to T. V Calculated as in formula (2):

[0042]

[0043] In the formula T f It is the video frame interval time, P V This is the current number of video frame payload packets.

[0044] The TS packet time interval value is set to T. t Calculated as in formula (3):

[0045]

[0046] In the formula, TL is the length of the TS packet, which is usually 188.

[0047] In an audio bitstream, the DTS time of the audio is equal to the PTS time of the audio, and the DTS time difference between two consecutive audio frames is set as T. d T d The abnormal threshold value is set to the audio abnormal threshold value T1, such as 500 milliseconds;

[0048] The abnormal threshold value of the difference between the DTS time of a video frame and the reference PCR time is set as the video abnormal threshold value T2, such as 800 milliseconds;

[0049] The minimum deviation between the DTS time of the video frame and the DTS time of the audio frame in the audio buffer is set as the audio-video anomaly threshold T3, such as 1000 milliseconds.

[0050] The minimum difference between the DTS time of video frames in the video buffer and the DTS time of audio frames in the audio buffer is set as the abnormal threshold value T4 of the audio and video buffer, such as 1000 milliseconds.

[0051] Step 1: Output the current audio / video packet

[0052] S11. If the audio buffer contains audio information, first calculate the DTS time difference T between two consecutive frames of audio in the audio buffer. d If the DTS time difference T dIf the time difference exceeds the audio anomaly threshold T1, the audio's DTS time is determined to be abnormal. The current audio frame's DTS time is used as the reference PCR time, and then the difference between the current system time and the most recent audio packet output time is determined to be greater than the audio packet time interval T. a If the output is positive, output the current audio packet and proceed to step S13; otherwise, stop outputting the audio packet and proceed to outputting the current video packet in step S12.

[0053] If the DTS time difference T d If the time difference is not greater than the audio anomaly threshold T1, no anomaly has occurred, and there is no need to replace the reference PCR time, then it is directly determined that the difference between the current system time and the time of the most recent audio packet output is greater than the audio packet time interval T. a If the output is successful, output the current audio packet and proceed to step S13; otherwise, stop outputting the audio packet and proceed to outputting the current video packet in step S12.

[0054] S12. If video information exists in the video buffer, calculate the time difference between the DTS time of the current video frame and the reference PCR time. If the time difference is less than the video anomaly threshold T2, output the current video packet.

[0055] Otherwise, perform video frame anomaly detection, mark the current video frame as abnormal based on the anomaly detection results, and output the current video packet;

[0056] The process of performing video frame anomaly detection is as follows:

[0057] If the deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer increases as the DTS time of each audio frame in the audio buffer increases, then the current video packet is output and the current video frame is marked as abnormal state I.

[0058] Otherwise, determine whether the minimum deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer is greater than the audio / video anomaly threshold T3.

[0059] If so, continue the evaluation:

[0060] i. If the minimum difference between the DTS time of the current video frame in the video buffer and the DTS time of each audio frame in the audio buffer is less than or equal to the audio / video buffer abnormal threshold T4, then output the current video packet and mark the current video frame as abnormal state I.

[0061] ii. If the minimum difference is greater than the audio / video buffer anomaly threshold T4, then determine: if the minimum deviation between the DTS time of the latest decoded video frame in the transcoding system and the DTS time of each audio frame in the audio buffer is less than or equal to the audio / video anomaly threshold T3, then output the current video packet and mark the current video frame as anomaly state I; otherwise, output the current video packet and mark the current video frame as anomaly state II.

[0062] If not, continue the judgment:

[0063] If an audio frame in the audio buffer has a DTS time greater than 1.25T3 than the DTS time of the current video frame, the corresponding audio frame is deleted and the current video packet is output; otherwise, the output of the current video packet is stopped.

[0064] S13. Adjust the DTS and PTS times of audio and video frames, and update the actual PCR time and reference PCR time, as follows:

[0065] The PCR deviation P is obtained by subtracting the actual PCR time from the reference PCR time. d Write the actual PCR time into the current TS packet stream; then subtract the PCR bias P from the DTS and PTS time values ​​of the audio and video. d The adjusted DTS and PTS time values ​​are obtained and written into their PES packet header bitstreams to achieve the adjustment of DTS and PTS times for audio and video frames.

[0066] After each TS packet is output, the reference PCR time and the actual PCR time should be increased by the TS packet time interval value Tt to update the actual PCR time and the reference PCR time.

[0067] Step 2: Output the next audio / video packet

[0068] Repeat steps S11-S13 to complete the output of the next audio / video packet. However, when executing step S12, if the current video frame has an anomaly marker, the current video packet is output directly based on the anomaly marker; otherwise, all steps of step S12 are executed.

[0069] The following example illustrates part of the processing steps of the above method using actual bitstream multiplexing data:

[0070] Where af represents the audio frame number, vf represents the video frame number, av represents the number of audio and video frames in the buffer, and dts represents the DTS time of the audio or video.

[0071] Bitstream multiplexing segment 1

[0072] vf:7927 av:12|0 dts:20056767988

[0073] vf:7928 av:12|0 dts:20056807988

[0074] af:2208 av:13|1 dts:20055956755

[0075] af:2209 audio dts:19738004755 video dts:19739687988 audio-video time difference:1683233: In the audio buffer, if the DTS time of an audio frame is greater than 1.25T3 (=1250000) than the DTS time of the current video frame, then these audio frames are deleted and the video packet is output.

[0076] af:2210 audio dts:19738148755 video dts:19739687988 audio-video time difference:1539233 In the audio buffer, if the DTS time of an audio frame is greater than 1.25T3 (=1250000) than the DTS time of the current video frame, then these audio frames are deleted and the video packet is output.

[0077] af:2211 audio dts:19738292755 video dts:19739687988 audio-video time difference:1395233 In the audio buffer, if the DTS time of an audio frame is greater than 1.25T3 (=1250000) than the DTS time of the current video frame, then these audio frames are deleted and the video packet is output.

[0078] af:2212 audio dts:19738436755 video dts:19739687988 audio-video time difference:1251233 In the audio buffer, if the DTS time of an audio frame is greater than 1.25T3 (=1250000) than the DTS time of the current video frame, then these audio frames are deleted and the video packet is output.

[0079] The previous audio DTS time was 20055956755, and the current audio DTS time is 19738580755, a difference of 317376000, exceeding the threshold T1 (500ms), thus indicating an audio DTS time anomaly. If an anomaly occurs, the DTS time of the current audio frame is used as the reference PCR time.

[0080] af:2213 av:8|1 dts:19738580755

[0081] af:2214 av:7|1 dts:19738724755

[0082] vf:7929 av:7|0 dts:19739687988

[0083] Stream multiplexing segment 2

[0084] af:9889 av:55|1 dts:2399898377

[0085] af:9890 av:54|1 dts:2399924500

[0086] The previous audio DTS time was 2399924500, and the current audio DTS time is 2141573477, a difference of 258351023, which exceeds the threshold T1 (500ms). Therefore, the audio DTS time is considered abnormal. If an abnormality occurs, the DTS time of the current audio frame is used as the reference PCR value.

[0087] vf:6447 av:54|0 dts:2141769400

[0088] af:9891 av:53|0 dts:2141573477

[0089] af:9892 av:56|3 dts:2141599600

[0090] af:9893 av:55|3 dts:2141625722

[0091] If the minimum deviation between the DTS time of the current video frame (2400209400) and the DTS time of the audio frame in the audio buffer (257146945) is greater than the audio-video anomaly threshold T3 (1000000), and the minimum difference between the DTS time of the video frame in the video buffer and the DTS time of the audio frame in the audio buffer (2456) is less than or equal to the audio-video buffer anomaly threshold T4 (1000000), then output a video packet and determine that the current state is anomaly 1.

[0092] vf:6448 av:55|2 dts:2400209400

[0093] af:9894 av:54|2 dts:2141651844

[0094] af:9895 av:53|2 dts:2141677966

[0095] Bitstream multiplexing segment 3

[0096] af:10763 av:64|1 dts:2192690322

[0097] af:10764 av:63|2 dts:2192714322

[0098] The previous audio DTS time was 2192714322, and the current audio DTS time is 1934378322, a difference of 258336000, which exceeds the threshold T1 (500ms). Therefore, the audio DTS time is considered abnormal. If an abnormality occurs, the DTS time of the current audio frame is used as the reference PCR value.

[0099] vf:6441 av:63|1 dts:1934690322

[0100] af:10765 av:62|1 dts:1934378322

[0101] af:10766 av:61|1 dts:1934402322

[0102] af:10767 av:60|1 dts:1934426322

[0103] af:10768 av:59|1 dts:1934450322

[0104] vf:6442 av:59|0 dts:1934730322

[0105] vf:6443 av:60|0 dts:1934770322

[0106] af:10769 av:59|0 dts:1934474322

[0107] If the minimum deviation between the DTS time of the current video frame (2193010322) and the DTS time of the audio frame in the audio buffer (257120000) is greater than the audio / video anomaly threshold T3 (1000000), and the minimum difference between the DTS time of the video frame in the video buffer and the DTS time of the audio frame in the audio buffer (257120000) is greater than the audio / video buffer anomaly threshold T4 (1000000), and the minimum deviation between the DTS time of the latest decoded video frame in the transcoding system and the DTS time of the audio frame in the audio buffer (240000) is less than the threshold T3 (1000000), then output a video packet, and determine that the current state is anomaly 1.

[0108] vf:6444 av:59|0 dts:2193010322

[0109] af:10770 av:58|0 dts:1934498322

[0110] af:10771 av:57|0 dts:1934522322

[0111] af:10772 av:60|2 dts:1934546322

[0112] If the minimum deviation between the DTS time of the current video frame (2193050322) and the DTS time of the audio frame in the audio buffer (257064000) is greater than the audio-video anomaly threshold T3 (1000000), and the minimum difference between the DTS time of the video frame in the video buffer and the DTS time of the audio frame in the audio buffer (0) is less than or equal to the audio-video buffer anomaly threshold T4 (1000000), then output a video packet and determine that the current state is anomaly 1.

[0113] vf:6445 av:60|1 dts:2193050322

[0114] af:10773 av:59|1 dts:1934570322

[0115] vf:6446 av:59|0 dts:1934810322

[0116] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for multiplexing audio and video transcoding transport streams, characterized in that: During audio and video transcoding, audio and video are decoded and re-encoded to form audio and video streams. These streams then enter a multiplexer, which outputs TS packets one by one through the following steps to form a TS transport stream. Step 1: Output the current audio / video packet S11. If the audio buffer contains audio information, first calculate the DTS time difference T between two consecutive frames of audio in the audio buffer. d If the DTS time difference T d The audio anomaly threshold T1 is greater than the DTS time difference T. d If the abnormal threshold value is reached, it is determined that the DTS time of the audio is abnormal. The DTS time of the current audio frame should be used as the reference PCR time. If the difference between the current system time and the time of the most recent audio packet output is greater than the audio packet time interval T, then the abnormality is determined. a If the output is positive, output the current audio packet and proceed to step S13; otherwise, stop outputting the audio packet and proceed to outputting the current video packet in step S12. If the DTS time difference T d If the time difference is not greater than the audio anomaly threshold T1, no anomaly has occurred, and there is no need to replace the reference PCR time, then it is directly determined that the difference between the current system time and the time of the most recent audio packet output is greater than the audio packet time interval T. a If the output is positive, output the current audio packet and proceed to step S13; otherwise, stop outputting the audio packet and proceed to outputting the current video packet in step S12. S12. If video information exists in the video buffer, calculate the time difference between the DTS time of the current video frame and the reference PCR time. If the time difference is less than the video anomaly threshold T2, output the current video packet. The video anomaly threshold T2 is set as the anomaly threshold value between the DTS time of the video frame and the reference PCR time. Otherwise, perform video frame anomaly detection, mark the current video frame as abnormal based on the anomaly detection results, and output the current video packet; S13. Adjust the DTS and PTS times of audio and video frames and update the actual PCR time and reference PCR time. When performing step S13, the PCR deviation P is obtained by subtracting the actual PCR time from the reference PCR time. d Write the actual PCR time into the current TS packet stream; then subtract the PCR bias P from the DTS and PTS time values ​​of the audio and video. d The adjusted DTS and PTS time values ​​are obtained and written into their PES packet header bitstreams to achieve the adjustment of DTS and PTS times for audio and video frames. After each TS packet is output, the reference PCR time and the actual PCR time should be increased by the TS packet time interval value Tt to update the actual PCR time and the reference PCR time. Step 2: Output the next audio / video packet Repeat steps S11-S13 to complete the output of the next audio / video packet. However, when executing step S12, if the current video frame has an anomaly marker, the current video packet is output directly based on the anomaly marker; otherwise, all steps of step S12 are executed.

2. The audio / video transcoding transport stream multiplexing method according to claim 1, characterized in that: When performing video frame anomaly detection, if the deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer increases as the DTS time of each audio frame in the audio buffer increases, then the current video packet is output and the current video frame is marked as anomaly state I. Otherwise, it is determined whether the minimum deviation between the DTS time of the current video frame and the DTS time of each audio frame in the audio buffer is greater than the audio-video anomaly threshold T3, where the audio-video anomaly threshold T3 is set as the anomaly threshold value of the minimum deviation. If so, continue the evaluation: i. If the minimum difference between the DTS time of the current video frame in the video buffer and the DTS time of each audio frame in the audio buffer is less than or equal to the audio-video buffer exception threshold T4, then output the current video packet and mark the current video frame as exception state I. The audio-video buffer exception threshold T4 is set as the exception threshold of the minimum difference. ii. If the minimum difference is greater than the audio / video buffer anomaly threshold T4, then determine: if the minimum deviation between the DTS time of the latest decoded video frame in the transcoding system and the DTS time of each audio frame in the audio buffer is less than or equal to the audio / video anomaly threshold T3, then output the current video packet and mark the current video frame as anomaly state I; otherwise, output the current video packet and mark the current video frame as anomaly state II. If not, continue the judgment: If an audio frame in the audio buffer has a DTS time greater than 1.25T3 than the DTS time of the current video frame, the corresponding audio frame is deleted and the current video packet is output; otherwise, the output of the current video packet is stopped.

Images