Method and device for detecting whether audio stream and video stream are synchronized

Abstract

The present invention discloses a method for detecting whether an audio stream and a video stream are synchronized. The method comprises: obtaining time information Pa of the last audio frame of the audio stream and time information Pv of the last video frame of the video stream; calculating the audio stream playback duration Ta and the video stream playback duration Tv based on a preset algorithm and Pa and Pv obtained in step S1; and comparing the absolute value of the difference between Ta and Tv calculated in step S2 with a preset threshold T. If the absolute value of the difference between the two is greater than the preset threshold, that is, |Tv-Ta|>T, then determining that the audio stream and the video stream are out of synchronization. The method runs quickly, has high detection accuracy, and significantly reduces the manpower required to detect audio and video stream asynchrony in video files.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video decoding, and in particular to a method and device for detecting whether an audio stream and a video stream are synchronized. Background Art

[0002] With advancements in digital video software and hardware technology and significant cost reductions, digital video applications are being adopted by a growing number of industries and fields. Financial institutions such as banks, securities firms, and insurance companies are required by regulatory authorities to conduct audio and video recording (referred to as dual recording) when selling wealth management products, precious metals, and insurance products. Furthermore, dual recording must comply with a series of regulatory standards and requirements. As a digital medium for display, dual recording must first ensure that both the audio and video streams are properly displayed and played. Proper playback typically requires clear audio and video content, smooth playback without interruptions, and synchronization. Audio and video synchronization is a common indicator of substandard recording quality, where the audio in a video may appear ahead or behind the video. Manually checking audio and video synchronization is a severe test of human energy and endurance when dealing with large numbers of videos. Furthermore, manual review inevitably suffers from drawbacks such as low efficiency, poor accuracy, inconsistent judgment criteria, and high costs. To address this issue, automated, intelligent auditing of dual recordings through computers has become imperative. However, the method of understanding the semantics of audio and video and then analyzing their synchronization is extremely technically difficult. This is because computers often have difficulty accurately understanding the semantics of audio and video, which affects their judgment of whether audio and video are synchronized. Summary of the Invention

[0003] In order to overcome the problems existing in the prior art, the present invention provides a method for detecting whether an audio stream and a video stream are synchronized, the method comprising:

[0004] S1: Obtain the time information Pa of a specified audio frame of an audio stream and the time information Pv of a specified video frame of a video stream;

[0005] S2: Calculate the audio stream playback time Ta and the video stream playback time Tv according to a preset algorithm and Pa and Pv obtained in step S1;

[0006] S3: Based on Ta and Tv calculated in step S2, the absolute value of the difference between the two is compared with the preset threshold T. If the absolute value of the difference between the two is greater than the preset threshold, that is, |Tv-Ta|>T, it is determined that the audio stream and the video stream are out of synchronization.

[0007] Furthermore, the method further comprises:

[0008] S0: Load the video file to be detected and go to step S1; wherein the video file includes an audio stream and a video stream.

[0009] Furthermore, step S3 also includes: comparing the absolute value of the difference between Ta and Tv calculated in step S2 with the preset threshold T; if the absolute value of the difference between the two is not greater than the preset threshold, that is, |Tv-Ta|≤T, it is determined that the audio stream and the video stream are synchronized.

[0010] Furthermore, the time information Pa and Pv in step S1 is a display timestamp PTS; the designated audio frame is the last audio frame; and the designated video frame is the last video frame.

[0011] Furthermore, the preset algorithm in step S2 is: Tv=Pv×Bv; Ta=Pa×Ba; and the preset threshold T in step S3 is 3 seconds.

[0012] The present invention also provides a device for detecting whether an audio stream and a video stream are synchronized, the device comprising: an information acquisition module for acquiring the time information Pa of the last audio frame of the audio stream and the time information Pv of the last video frame of the video stream; a duration calculation module for calculating the audio stream playback duration Ta and the video stream playback duration Tv based on a preset algorithm and Pa and Pv acquired by the information acquisition module; a threshold comparison module for comparing the absolute value of the difference between Ta and Tv calculated in the duration calculation module with a preset threshold T. If the absolute value of the difference between the two is greater than the preset threshold, that is, |Tv-Ta|>T, it is determined that the audio stream and the video stream are not synchronized.

[0013] Furthermore, the device further comprises: a file loading module for loading the video file to be detected and transferring the loaded video file to the information acquisition module for processing; wherein the video file comprises an audio stream and a video stream.

[0014] Furthermore, the threshold comparison module is also used to compare the absolute value of the difference between Ta and Tv calculated in the duration calculation module with the preset threshold T. If the absolute value of the difference between the two is not greater than the preset threshold, that is, |Tv-Ta|≤T, it is determined that the audio stream and the video stream are synchronized.

[0015] Furthermore, the time information Pa and Pv in the information acquisition module are the display timestamp PTS; the designated audio frame is the last audio frame; and the designated video frame is the last video frame.

[0016] Furthermore, the preset algorithm in the duration calculation module is: Tv=Pv×Bv; Ta=Pa×Ba; the preset threshold T in the threshold comparison module is 3 seconds.

[0017] The present invention can calculate the total duration of the audio and video streams from the time information of the audio and video streams in the tested video file, and then compare the absolute value of the difference between the two with a preset threshold T to determine whether the audio and video of the video file are synchronized. The present invention runs quickly, has high detection accuracy, and significantly saves labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the logical structure and typical hardware environment of a device for detecting whether an audio stream and a video stream are synchronized provided by the present invention;

[0019] Figure 2 A flow chart of a method for detecting whether an audio stream and a video stream are synchronized provided by the present invention;

[0020] Figure 3 A schematic diagram of the logical structure and typical hardware environment of another device for detecting whether an audio stream and a video stream are synchronized provided by the present invention;

[0021] Figure 4 A flow chart of another method for detecting whether an audio stream and a video stream are synchronized provided by the present invention. DETAILED DESCRIPTION

[0022] The present invention provides a method and apparatus for detecting whether an audio stream and a video stream are synchronized, for efficiently and quickly detecting whether the audio stream and the video stream are synchronized in a video file. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] Next, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views illustrating device structures may be partially enlarged and not to scale when describing the embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, three-dimensional dimensions, including length, width, and depth, should be included.

[0024] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0025] Example 1

[0026] The present invention provides a method and device for detecting whether an audio stream and a video stream are synchronized, which is used to efficiently and quickly detect whether an audio stream and a video stream are synchronized in a video file. Figure 1 In a preferred embodiment, the present invention provides a device for detecting whether an audio stream and a video stream are synchronized. The basic operating hardware environment of the device includes a CPU, memory, non-volatile memory and various hardware. From a logical level, the device includes: an information acquisition module, a duration calculation module and a threshold comparison module. Please refer to Figure 2 , the method provided by the present invention comprises the following steps:

[0027] Step 101: The information acquisition module acquires the time information Pa of a specified audio frame of an audio stream and the time information Pv of a specified video frame of a video stream;

[0028] Step 102: The duration calculation module calculates the audio stream playback duration Ta and the video stream playback duration Tv according to a preset algorithm and Pa and Pv obtained by the information acquisition module;

[0029] In step 103, the threshold comparison module compares the absolute value of the difference between Ta and Tv calculated in the duration calculation module with a preset threshold T. If the absolute value of the difference between the two is greater than the preset threshold, that is, |Tv-Ta|>T, it is determined that the audio stream and the video stream are out of synchronization.

[0030] The present invention can calculate the total duration of the audio and video streams from the time information of the audio and video streams in the tested video file, and then compare the absolute value of the difference between the two with a preset threshold T to determine whether the audio and video of the video file are synchronized. The present invention runs quickly, has high detection accuracy, and significantly saves labor costs.

[0031] Example 2

[0032] The present invention provides a method and device for detecting whether an audio stream and a video stream are synchronized, which is used to efficiently and quickly detect whether an audio stream and a video stream are synchronized in a video file. Figure 3 In a preferred embodiment, the present invention provides a device for detecting whether an audio stream and a video stream are synchronized. The basic operating hardware environment of the device includes a CPU, memory, non-volatile memory and various hardware. From a logical level, the device includes: a file loading module, an information acquisition module, a duration calculation module and a threshold comparison module. Please refer to Figure 4 , the method provided by the present invention comprises the following steps:

[0033] Step 201: The file loading module loads the video file to be detected and transfers the loaded video file to the information acquisition module for processing; wherein the video file includes an audio stream and a video stream.

[0034] Usually a video file contains audio data and video data, which form audio data stream and video data stream when they are played. If we want to detect multiple video files, we first need to load these video files one by one for subsequent processing.

[0035] Step 202: Obtain time information Pa of a specified audio frame of an audio stream and time information Pv of a specified video frame of a video stream.

[0036] The time information mentioned here is usually the Presentation Time Stamp, or PTS, which is used to tell the player when to display the data of this frame. Let's take a brief look at some professional terms related to video files. As mentioned earlier, a video file contains an audio stream and a video stream. Usually, when playing a file, the audio stream and video stream in the video file need to be decoded separately by a decoder. Since how the decoder decodes is an existing technology, it will not be described here. When the decoder decodes to the last frame of the audio stream and video stream, it will retain the PTS of the frame. A frame is the smallest unit of data transmission. The smallest data transmission unit of an audio stream is an audio frame, and the smallest data transmission unit of a video stream is a video frame. In step 202, the specified audio frame refers to the last audio frame; the specified video frame refers to the last video frame. Therefore, Pa is the PTS of the last audio frame of the audio stream, and Pv is the PTS of the last video frame of the video stream.

[0037] In addition, the time information mentioned here may also be a decoding timestamp (Decode Time Stamp), that is, DTS. The significance of this timestamp is to tell the player when to decode the data of this frame. Generally, DTS and PTS are consistent, but affected by the video recording equipment and encoding standards, DTS often needs to be recalculated when decoding video files. Therefore, in the method provided by the present invention, PTS is preferably used as the time information. In fact, as long as the parameter information can be used to calculate the playback duration of the audio and video stream, it can be used as the time information in step 202, and is not limited to PTS. Similarly, the specified video frame is not limited to the last video frame; the specified audio frame is not limited to the last audio frame. They are just preferred options.

[0038] Step 203: The duration calculation module calculates the audio stream playback duration Ta and the video stream playback duration Tv according to a preset algorithm and Pa and Pv obtained by the information acquisition module;

[0039] As can be seen from step 202 above, Pa is the PTS of the last audio frame in the audio stream, and Pv is the PTS of the last video frame in the video stream. The default algorithm in step 203 is Ta = Pa × Ba, Tv = Pv × Bv. Ta is the playback duration of the audio stream, Tv is the playback duration of the video stream, Ba is the time base of the audio stream, and Bv is the time base of the video stream. The time base is the display duration of each frame. For example, if a video stream displays 50 frames per second, that is, the next video is displayed every 1 / 50 second, then the time base of this video stream is 1 / 50.

[0040] Next, let's take another example to explain the algorithm in step 203 in detail. Assume that for a video file, after the video stream is decoded, the PTS of the last video frame, Pv, is 4037632, and the time base Bv is 1 / 12288; after the audio stream is decoded, the PTS of the last audio frame, Pa, is 7097674, and the time base Ba is 1 / 22050.

[0041] By using the algorithm Tv=Pv×Bv, we can get:

[0042] Tv = 4037632 × (1 / 12288) = 328.58 (seconds)

[0043] By using the algorithm Ta=Pa×Ba, we can get:

[0044] Ta = 7097674 × (1 / 22050) = 321.89 (seconds)

[0045] This indicates that the video stream of this video file is 328.58 seconds long and the audio stream is 321.89 seconds long.

[0046] Of course, the algorithm is pre-set, and the formulas Tv = Pv × Bv and Ta = Pa × Ba listed here are pre-set algorithms based on the time information in the above steps being PTS. That is, if the time information in step 201 is not PTS, the pre-set algorithm will change. The pre-set algorithm here is merely the best choice and is not limited to this.

[0047] In step 204, the threshold comparison module compares the absolute value of the difference between Ta and Tv calculated in the duration calculation module with a preset threshold T. If the absolute value of the difference between the two is greater than the preset threshold, that is, |Tv-Ta|>T, it is determined that the audio stream and the video stream are out of sync; otherwise, it is determined that the audio stream and the video stream are synchronized.

[0048] The threshold T mentioned here can be set according to the specific application scenario. Generally speaking, the lower the threshold, the higher the sensitivity, and vice versa. The threshold T is usually set to 3 seconds. That is, if the playback duration of the audio and video streams differs by more than 3 seconds, they are considered out of sync; if the playback duration of the audio and video streams differs by 3 seconds or less, they are considered in sync.

[0049] For example, assuming that the playback time of the audio stream Ta = 321.89 (seconds), the playback time of the video stream Tv = 328.58 (seconds), and the threshold value T = 3 (seconds). Because |Tv-Ta| = 6.69 seconds, this value indicates that the playback time of the video stream and the audio stream differ by 6.69 seconds, which is greater than the threshold value T of 3 seconds, and it is determined that the audio stream and the video stream in the video file are not synchronized. For another example, assuming that the playback time of the audio stream Ta = 321.89 (seconds), the playback time of the video stream Tv = 323.58 (seconds), and the threshold value T = 3 (seconds). Because |Tv-Ta| = 1.69 seconds, this value indicates that the playback time of the video stream and the audio stream differ by 1.69 seconds, which is less than the threshold value T of 3 seconds, and it is determined that the audio stream and the video stream in the video file are synchronized. For another example, suppose the audio stream's playback time Ta = 321.89 seconds, the video stream's playback time Tv = 324.89 seconds, and the threshold value T = 3 seconds. Since |Tv - Ta| = 3 seconds, this value indicates that the playback time difference between the video and audio streams is 3 seconds, which is equal to the threshold value T of 3 seconds. Similarly, the audio and video streams in this video file are considered synchronized.

[0050] The specific implementation described in this article is only a specific explanation of the present invention. The present invention mainly calculates the actual duration of the encoded stream through the PTS of the last frame. Technicians in the relevant technical field can make various fine-tuning modifications or supplements to the described specific embodiments or adopt similar methods to replace them. For example, the threshold parameter T can be adjusted or the PTS of the second-to-last frame can be used to calculate its duration. The duration compared with the threshold T is still approximately the duration of the audio stream and the video stream, but it does not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0051] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting whether an audio stream and a video stream are synchronized, wherein: The audio stream includes a plurality of audio frames, and the video stream includes a plurality of video frames, wherein the method comprises: S1: Obtain the time information Pa of a specified audio frame of an audio stream and the time information Pv of a specified video frame of a video stream; S2: Calculate the audio stream playback time Ta and the video stream playback time Tv according to a preset algorithm and Pa and Pv obtained in step S1; S3: Based on Ta and Tv calculated in step S2, the absolute value of the difference between the two is compared with a preset threshold value T. If the absolute value of the difference between the two is greater than the preset threshold value, that is, |Tv-Ta|>T, it is determined that the audio stream and the video stream are out of synchronization; The preset algorithm in step S2 is: Tv=Pv×Bv; Ta=Pa×Ba, where Bv is the time base of the video stream and Ba is the time base of the audio stream; The time information Pa and Pv in step S1 are the display timestamp PTS; The designated audio frame is the last audio frame; the designated video frame is the last video frame.

2. The method according to claim 1, characterized in that The method further comprises: S0: Load the video file to be detected and go to step S1; wherein the video file includes an audio stream and a video stream.

3. The method according to claim 1, characterized in that The step S3 further comprises: According to Ta and Tv calculated in step S2, the absolute value of the difference between the two is compared with the preset threshold T. If the absolute value of the difference between the two is not greater than the preset threshold, that is, |Tv-Ta|≤T, it is determined that the audio stream and the video stream are synchronized.

4. The method according to any one of claims 1 to 3, characterized in that: The preset threshold T in step S3 is 3 seconds.

5. A device for detecting whether an audio stream and a video stream are synchronized, wherein: The audio stream includes a plurality of audio frames, and the video stream includes a plurality of video frames, wherein the device comprises: an information acquisition module for acquiring time information Pa of a specified audio frame of the audio stream, and time information Pv of a specified video frame of the video stream; A duration calculation module, configured to calculate the audio stream playback duration Ta and the video stream playback duration Tv based on a preset algorithm and Pa and Pv obtained by the information acquisition module; a threshold comparison module, configured to compare the absolute value of the difference between Ta and Tv calculated in the duration calculation module and a preset threshold value T, and determine that the audio stream and the video stream are out of sync if the absolute value of the difference between the two is greater than the preset threshold, i.e., |Tv-Ta|>T; The preset algorithm in the duration calculation module is: Tv=Pv×Bv; Ta=Pa×Ba, where Bv is the time base of the video stream and Ba is the time base of the audio stream; The time information Pa and Pv in the information acquisition module are the display timestamp PTS; The designated audio frame is the last audio frame; the designated video frame is the last video frame.

6. The device according to claim 5, characterized in that The device further comprises: The file loading module is used to load the video file to be detected and transfer the loaded video file to the information acquisition module for processing; wherein the video file includes an audio stream and a video stream.

7. The device according to claim 5, characterized in that The threshold comparison module is further configured to compare the absolute value of the difference between Ta and Tv calculated in the duration calculation module and a preset threshold value T. If the absolute value of the difference between the two is not greater than the preset threshold value, i.e., |Tv-Ta|≤T, the audio stream and the video stream are determined to be synchronized.

8. The device according to any one of claims 5 to 7, characterized in that: The preset threshold T in the threshold comparison module is 3 seconds.

Images