A method and system for audio-video synchronization based on TS stream and a storage medium

Abstract

The present application relates to the technical field of audio and video synchronization processing, more particularly to an audio and video synchronization method and system based on TS stream and a storage medium. The method comprises reading TS stream data frames and unpacking the TS stream data frames; a synchronization device acquires and calculates the difference between the PTS of the current frame and the previous frame, determines whether the difference is a TS abnormal data type; when the difference is a TS abnormal data type, the synchronization device further determines the type of the current frame; if the type of the current frame is a video frame, the synchronization device discards the current frame; if the type of the current frame is an audio frame, the synchronization device adjusts the offset value and determines whether the current frame is subjected to decoding processing; if the current frame is subjected to decoding processing, a transcoder performs decoding processing. The present application realizes normal transcoding and audio and video synchronization by offset detection and correction error avoidance stream.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio and video synchronization processing, and more particularly to an audio and video synchronization method and system based on TS stream and a storage medium. BACKGROUND

[0002] The transcoder usually has the following processing: the received data is sent into the transcoder for decoding. During the decoding process, when it is found that the PTS difference between the previous frame and the next frame is large enough to reach a certain threshold, the PTS offset value is recalculated, which can achieve the continuity of the PTS value after processing the PTS discontinuous stream. This is used to ensure the continuity of the entire video after transcoding, which also improves the compatibility of the transcoder for PTS discontinuous stream.

[0003] However, in the actual production environment, there is a kind of TS abnormal data stream, and the PTS between the audio frame and the video frame has a large offset, that is, the offset absolute value is greater than 1000 milliseconds, which is different from the PTS discontinuous data stream (the deviation is instantaneous), and this offset exists for a long time. Taking this abnormal TS stream as the input source of the transcoding system and performing real-time transcoding. The decoding and recalculation of the offset value will map the audio and video frames that do not correspond (there is an offset) to the same time node, which leads to the fact that the decoded picture and sound are not synchronized in this case, thereby affecting the subsequent transcoding output result and causing the phenomenon of audio and video out of synchronization. SUMMARY

[0004] The present application provides an audio and video synchronization method and system based on TS stream and a storage medium to overcome the problem of audio and video out of synchronization after decoding due to the large offset absolute value between the audio frame and the video frame.

[0005] To solve the above technical problems, the technical solutions of the present application are as follows:

[0006] In a first aspect, the present application provides an audio and video synchronization method based on TS stream, comprising:

[0007] reading the TS stream data frame and unpacking the TS stream data frame;

[0008] The synchronization device acquires and calculates the difference between the PTS of the current frame and the previous frame, and determines whether the current frame is of the TS abnormal data type according to the difference;

[0009] When the difference is of the TS abnormal data type, the synchronization device further determines the type of the current frame;

[0010] If the type of the current frame is a video frame, the synchronization device discards the current frame;

[0011] If the type of the current frame is an audio frame, the synchronizing device adjusts the offset value again and determines whether the current frame is decoded.

[0012] If the current frame is decoded, the transcoder decodes the current frame.

[0013] Preferably, the algorithm for calculating the difference between the PTS of the current frame and the PTS of the previous frame comprises:

[0014] cur_pts = curpts - ts_offset;

[0015] last_pts = lastpts - ts_offset;

[0016] delta = cur_pts - last_pts;

[0017] wherein cur_pts represents the PTS of the current frame after the offset; curpts represents the PTS of the current frame; ts_offset represents the PTS offset value of the decoder; last_pts represents the PTS of the previous frame after the offset; lastpts represents the PTS of the previous frame; and delta represents the difference between the PTS of the current frame and the PTS of the previous frame.

[0018] Preferably, the determination of whether the current frame is TS abnormal data type according to the difference comprises:

[0019] determining whether the difference is within a preset threshold range;

[0020] if the difference is within the threshold range, determining that the current frame is not TS abnormal data type;

[0021] if the difference is not within the threshold range, determining that the current frame is TS abnormal data type.

[0022] Preferably, when the difference is TS abnormal data type, the synchronizing device further determines the type of the current frame, comprising:

[0023] when the difference is TS abnormal data type, the synchronizing device determines the type of the current frame according to the duration of the current frame.

[0024] Preferably, if the type of the current frame is an audio frame, the synchronizing device adjusts the offset value again and determines whether the current frame is decoded, comprising:

[0025] if the type of the current frame is an audio frame, the synchronizing device adjusts the offset value again, and then determines whether the adjusted offset value is within the threshold range;

[0026] If not, it is determined that the current frame offset value is large, and the current frame is discarded;

[0027] If yes, it is determined that the current frame offset value is small, and the current frame is of a discontinuous type, and the current frame is decoded.

[0028] Preferably, the algorithm for readjusting the offset value comprises:

[0029] ts_offset' = ts_offset + delta;

[0030] Wherein, ts_offset' represents the offset value of the decoder pts after readjustment; ts_offset represents the offset value of the decoder pts; and delta represents the difference between the pts of the current frame and the previous frame.

[0031] Preferably, the threshold range is greater than -1000 milliseconds and less than 1000 milliseconds.

[0032] In the second aspect, the present application provides a TS stream-based audio-video synchronization system, which comprises:

[0033] A processor for packaging and unpackaging audio-video data;

[0034] A synchronization device for processing unpackaged audio-video data transmitted by the processor;

[0035] A transcoder for processing synchronization data transmitted by the synchronization device.

[0036] The output end of the processor is in data connection with the input end of the synchronization device, and the output end of the synchronization device is in data connection with the transcoder.

[0037] Preferably, the transcoder comprises a decoding module for decoding the audio-video synchronization data processed and transmitted by the synchronization device, and an encoding module for re-encoding the decoded data transmitted by the decoding module, the output end of the synchronization device is in data connection with the decoding module, and the output end of the decoding module is in data connection with the input end of the encoding module.

[0038] In the third aspect, the present application provides a storage medium, which stores a computer program, and the computer program is executed by a processor to realize the TS stream-based audio-video synchronization method according to the first aspect.

[0039] The present application has the following beneficial effects:

[0040] The present application takes audio frames as a reference, readjusts the offset value, waits for the next video frame to arrive and then starts decoding, so as to realize the effect of audio-video synchronization. When the absolute value of the offset between the audio frame and the video frame is greater than 1000 milliseconds, if the video frame arrives first, the video frame is discarded until the audio frame arrives first, and then the offset value is readjusted based on the audio frame, and then the next video frame is decoded. The present application avoids the stream by offset detection and correction errors, so as to realize normal transcoding and audio-video synchronization. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The implementation flowchart of the audio-video synchronization method based on TS stream provided by the present application is shown.

[0042] Figure 2 The determination flowchart of the audio-video synchronization method based on TS stream provided by the present application is shown.

[0043] Figure 3 The structure schematic diagram of the audio-video synchronization system based on TS stream provided by the present application is shown.

[0044] Figure 4 The implementation flowchart of the audio-video synchronization system based on TS stream provided by the present application is shown.

[0045] Among them: processor 10, synchronization device 20, transcoder 30. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0047] In the drawings of the embodiments of the present application, the same or similar reference numerals correspond to the same or similar components; in the description of the present application, it should be understood that if there are terms such as "upper", "lower", "left", "right", "top", "bottom", "inner", "outer" and the like indicating the orientation or positional relationship, it is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element indicated must have a particular orientation, structure and operation, therefore the terms describing the positional relationship in the drawings are only used for exemplary description, and cannot be understood as a limitation of the present patent.

[0048] In addition, if there are terms such as "first", "second" and the like, they are only used for description purposes, mainly for distinguishing different devices, elements or components (the specific types and structures may be the same or different), and are not used to indicate or imply the relative importance and quantity of the indicated devices, elements or components, and cannot be understood as indicating or implying the relative importance.

[0049] Embodiment One

[0050] As shown in Figure 1 , the embodiment provides a method for audio and video synchronization based on TS stream, which comprises:

[0051] S101, reading TS stream data frame and unpacking the TS stream data frame.

[0052] S102, the synchronization device acquires and calculates the difference between the current frame and the previous frame PTS (Presentation Time Stamp), and determines whether the current frame is TS abnormal data type according to the difference.

[0053] S103, when the difference is TS abnormal data type, the synchronization device further determines the type of the current frame.

[0054] S104, if the type of the current frame is video frame, the synchronization device discards the current frame.

[0055] S105, if the type of the current frame is audio frame, the synchronization device adjusts the offset value and determines whether the current frame is decoded.

[0056] S106, if the current frame is decoded, the transcoder decodes the current frame.

[0057] Please refer to Figure 2 , which shows the determination flow chart of the method for audio and video synchronization based on TS stream provided by the embodiment. Figure 2

[0058] In step S101, the processor reads the TS stream data frame and transmits the unpacked TS stream data frame to the synchronization device.

[0059] In step S102, the synchronization device reads a data frame, compares the data frame with the previous frame, and calculates the difference between the current frame and the previous frame PTS. In the embodiment, the algorithm for calculating the difference between the current frame and the previous frame PTS is as follows:

[0060] cur_pts = curpts - ts_offset;

[0061] last_pts = lastpts - ts_offset;

[0062] delta = cur_pts - last_pts;

[0063] ​Wherein, cur_pts represents the PTS offset by the current frame; curpts represents the PTS of the current frame; ts_offset represents the decoder PTS offset value; last_pts represents the PTS offset by the previous frame; lastpts represents the PTS of the previous frame; delta represents the difference between the PTS of the current frame and the PTS of the previous frame.

[0064] Further, the synchronization device determines whether the current frame is within a preset threshold range according to the difference, the threshold range being greater than -1000 milliseconds and less than 1000 milliseconds, i.e. -1000ms~1000ms. If the difference is within -1000ms~1000ms, the synchronization device determines that the current frame is not of the TS abnormal data type and directly starts decoding and outputting. If the difference is not within -1000ms~1000ms, the synchronization device determines that the current frame is of the TS abnormal data type, i.e. the absolute value of the offset between the current frame and the previous frame is too large, resulting in the situation that the audio and video are out of synchronization after decoding.

[0065] In step S103, when the difference is of the TS abnormal data type, the synchronization device further determines the type of the current frame according to the duration of the playing of the current frame.

[0066] In step S104, if the type of the current frame is a video frame, the synchronization device discards the current frame until the current frame is an audio frame, and step S105 is performed.

[0067] In step S105, if the type of the current frame is an audio frame, the synchronization device first adjusts the offset value, and then determines whether the adjusted offset value is within the threshold range, the threshold range being the same as above, i.e. -1000ms~1000ms. If the adjusted offset value is not within the threshold range, it is determined that the offset value of the current frame is large, and the current frame is discarded. If the adjusted offset value is within the threshold range, it is determined that the offset value of the current frame is small, and the current frame is of the discontinuous type, and the decoding of the current frame can be started.

[0068] In the embodiment, the algorithm for adjusting the offset value is as follows:

[0069] ts_offset’ = ts_offset+delta;

[0070] Wherein, ts_offset’ represents the adjusted decoder pts offset value; ts_offset represents the decoder pts offset value; and delta represents the difference between the pts of the current frame and the pts of the previous frame.

[0071] In step S106, the frame data transmitted by the synchronization device is decoded and encoded by the transcoder.

[0072] Finally, the processor re-encapsulates the encoded data into a synchronous TS stream, i.e., audio-video synchronization.

[0073] The above method takes an audio frame as a reference, and in an application scenario where the PTS of the input TS stream audio-video data packet has an offset, the method avoids errors of the stream by offset detection and correction, and further realizes the effect of normal transcoding.

[0074] Embodiment Two

[0075] As shown in Figure 3 , the embodiment provides a TS stream-based audio-video synchronization system, which includes a processor 10, a synchronization device 20, and a transcoder 30. The output end of the processor 10 is in data connection with the input end of the synchronization device 20, and the output end of the synchronization device 20 is in data connection with the transcoder 30. Among them, the processor 10 is mainly used for encapsulating and decapsulating audio-video data. The synchronization device 20 is mainly used for processing the decapsulated audio-video data transmitted by the processor 10. The transcoder 30 is mainly used for processing the synchronization data transmitted by the synchronization device 20.

[0076] In this embodiment, the transcoder 30 includes a decoding module and an encoding module, and the output end of the synchronization device 20 is in data connection with the decoding module. The output end of the decoding module is in data connection with the input end of the encoding module. Among them, the decoding module is mainly used for decoding the audio-video synchronization data processed and transmitted by the synchronization device 20. The encoding module is mainly used for re-encoding the decoded data transmitted by the decoding module.

[0077] Please refer to Figure 4 , Figure 4 The figure shows the implementation flowchart of the TS stream-based audio-video synchronization system of the embodiment.

[0078] Through the above structural connection, the TS stream-based audio-video synchronization method of the embodiment is as follows:

[0079] S101, read the TS stream data frame and decapsulate the TS stream data frame.

[0080] S102, the synchronization device acquires and calculates the difference between the PTS (Presentation Time Stamp, display time stamp) of the current frame and the previous frame, and determines whether the current frame is a TS abnormal data type according to the difference.

[0081] S103, when the difference is a TS abnormal data type, the synchronization device further determines the type of the current frame.

[0082] S104, if the type of the current frame is a video frame, the synchronization device discards the current frame.

[0083] S105, if the type of the current frame is an audio frame, the synchronization device adjusts the offset value again and determines whether the current frame is decoded.

[0084] S106, if the current frame is decoded, the transcoder decodes the current frame.

[0085] Firstly, the processor 10 acquires the frame data and unpacks the frame data. Then, the synchronization device 20 reads the unpacked frame data of the processor 10, compares the current frame with the previous frame, and calculates the difference between the PTS of the current frame and the PTS of the previous frame.

[0086] Further, the synchronization device 20 determines whether the current frame is within a preset threshold range according to the difference, the threshold range being greater than -1000 milliseconds and less than 1000 milliseconds, i.e. -1000ms~1000ms. If the difference is within -1000ms~1000ms, the synchronization device 20 determines that the current frame is not a TS abnormal data type and directly starts decoding and outputting. If the difference is not within -1000ms~1000ms, the synchronization device 20 determines that the current frame is a TS abnormal data type, i.e. the absolute value of the offset between the current frame and the previous frame is too large, resulting in the situation that the audio and video are out of synchronization after decoding.

[0087] When the difference is a TS abnormal data type, the synchronization device 20 further determines the type of the current frame according to the playing duration of the current frame.

[0088] If the type of the current frame is an audio frame, the synchronization device 20 adjusts the offset value again, and then determines whether the adjusted offset value is within the threshold range, the threshold range being the same as above, i.e. -1000ms~1000ms. If the adjusted offset value is not within the threshold range, it is determined that the offset value of the current frame is large, and the current frame is discarded. If the adjusted offset value is within the threshold range, it is determined that the offset value of the current frame is small, and the current frame is a discontinuous type, and the decoding process of the current frame can be started.

[0089] The transcoder 30 decodes and encodes the frame data transmitted by the synchronization device 20, and then, through the processing of the processor 10, the audio and video can be re-packaged into a synchronized TS stream, i.e. audio and video synchronization.

[0090] The above system implementing the audio and video synchronization method based on TS stream can solve the problem of audio and video out of synchronization caused by TS stream abnormality, and improve the compatibility of the transcoder 30 to the stream.

[0091] Embodiment three

[0092] The embodiment provides a storage medium, and the storage medium stores a computer program. The computer program is executed by the processor 10 to implement the audio and video synchronization method based on TS stream as described in the embodiment one.

[0093] Please refer to Figures 1-4 When the computer program is executed by the processor 10, the flow of the audio and video synchronization method based on TS stream is as follows:

[0094] The processor 10 reads the TS stream data frame and unpacks the TS stream data frame.

[0095] The synchronization device 20 acquires and calculates the difference value of the PTS (Presentation Time Stamp) of the current frame and the previous frame, and determines whether the current frame is of the TS abnormal data type according to the difference value.

[0096] When the difference value is of the TS abnormal data type, the synchronization device 20 further determines the type of the current frame.

[0097] If the type of the current frame is a video frame, the synchronization device 20 discards the current frame.

[0098] If the type of the current frame is an audio frame, the synchronization device 20 re-adjusts the offset value and determines whether the current frame is subjected to decoding processing.

[0099] If the decoding processing is performed, the transcoder 30 performs the decoding processing.

[0100] Although the description of the present application is combined with the above specific embodiments, it is obvious for those skilled in the art to make many substitutions, modifications and changes according to the above content. Therefore, all such substitutions, improvements and changes are included in the spirit and scope of the appended claims. The above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Other different forms of changes or variations can be made on the basis of the above description for those skilled in the art. Here, it is unnecessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A method for synchronizing audio and video based on TS stream, characterized in that, The system comprises: reading TS stream data frames and unpacking the TS stream data frames; a synchronization device acquires and calculates the difference between the current frame and the previous frame PTS, and determines whether the current frame is a TS abnormal data type according to the difference; when the difference is a TS abnormal data type, the synchronization device further determines the type of the current frame; if the type of the current frame is a video frame, the synchronization device discards the current frame; if the type of the current frame is an audio frame, the synchronization device re-adjusts the offset value and determines whether the current frame is subjected to decoding processing; if the decoding processing is performed, a transcoder performs decoding processing; if the type of the current frame is an audio frame, the synchronization device re-adjusts the offset value and determines whether the current frame is subjected to decoding processing, comprising: if the type of the current frame is an audio frame, the synchronization device first re-adjusts the offset value, and then determines whether the re-adjusted offset value is within a threshold range; if not, it is determined that the current frame offset value is large, and the current frame is discarded; if yes, it is determined that the current frame offset value is small, and the current frame is of a discontinuous type, and the current frame is subjected to decoding processing; the algorithm for re-adjusting the offset value comprises: ts_offset' = ts_offset + delta; wherein ts_offset' represents the offset value of the re-adjusted decoder pts; ts_offset represents the decoder pts offset value; and delta represents the difference between the current frame and the previous frame pts.

2. The method of claim 1, wherein, the algorithm for calculating the difference between the current frame and the previous frame PTS comprises: cur_pts = curpts - ts_offset; last_pts = lastpts - ts_offset; delta = cur_pts - last_pts; wherein cur_pts represents the offset PTS of the current frame; curpts represents the PTS of the current frame; ts_offset represents the decoder PTS offset value; last_pts represents the offset PTS of the previous frame; lastpts represents the PTS of the previous frame; and delta represents the difference between the current frame and the previous frame PTS.

3. The method of claim 1, wherein, the determination of whether the current frame is a TS abnormal data type according to the difference comprises: determining whether the difference is within a preset threshold range; if the difference is within the threshold range, it is determined that the current frame is not a TS abnormal data type; if the difference is not within the threshold range, it is determined that the current frame is a TS abnormal data type.

4. The method of claim 1, wherein, when the difference is a TS abnormal data type, the synchronization device further determines the type of the current frame, comprising: when the difference is a TS abnormal data type, the synchronization device determines the type of the current frame according to the playing duration of the current frame.

5. The method of claim 1, wherein, the threshold range is greater than -1000 milliseconds and less than 1000 milliseconds.

6. A system for synchronizing audio and video based on TS stream, characterized in that, The system comprises: a processor for packaging and unpackaging audio and video data; a synchronization device for processing unpackaged audio and video data transmitted by the processor; a transcoder for processing synchronization data transmitted by the synchronization device; An output of the processor is connected with an input of the synchronization device, an output of the synchronization device is connected with the transcoder, and the processor realizes the TS stream-based audio and video synchronization method of any one of claims 1-5 through the synchronization device and the transcoder.

7. The TS stream based audio-video synchronization system according to claim 6, wherein, The transcoder comprises a decoding module for decoding the audio and video synchronization data transmitted by the synchronization device and an encoding module for re-encoding the decoded data transmitted by the decoding module, and an output of the synchronization device is connected with the decoding module; an output of the decoding module is connected with an input of the encoding module.

8. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to realize the TS stream-based audio and video synchronization method of any one of claims 1-5.

Images