A method and system for detecting audio-visual asynchrony in primary and backup programs.

Abstract

This invention discloses a method and system for detecting audio-visual asynchrony in primary and backup channel programs. The method involves acquiring primary and backup video signals and extracting their feature values, aligning the primary and backup video frames, and calculating the PTS difference between the aligned primary and backup video frames. It also involves acquiring primary and backup audio signals and extracting their feature values, aligning the primary and backup audio frames, and calculating the PTS difference between the aligned primary and backup audio frames. The absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames is calculated, and it is determined whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup channels is audio-visually asynchronous; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization. This method can automatically detect the audio-visual synchronization status of primary and backup channel programs, and can promptly issue an alarm once audio-visual asynchrony occurs, greatly saving manpower, avoiding missed detections, and improving program broadcast quality.

Description

Technical Field

[0001] This invention relates to the field of audio-visual synchronization detection technology, specifically to a method and system for detecting audio-visual asynchrony between primary and backup programs. Background Art

[0002] In both the broadcast television and internet video industries, program broadcast quality has always been a crucial issue for operators. To ensure the quality of the final broadcast content, numerous technical means and methods are employed. These include improving the quality and stability of encoding / decoding equipment and broadcasting equipment; and adding backup broadcast systems to ensure flawless program delivery. Increasingly, broadcast systems are acquiring and analyzing the actual broadcast signal at the final output stage. If any technical flaws are detected, the system can automatically switch between primary and backup systems.

[0003] Existing program signal analysis and detection technologies mainly include: 1) Video anomaly detection: video loss / black field / color field / color bar / still frame / mosaic detection, etc.; 2) Audio anomaly detection: mute / low volume / fixed volume / fixed frequency / white noise / stereo inversion / stereo independent detection, etc. However, there is a lack of effective automatic detection methods for audio and video synchronization issues, which can only be judged manually. This is not only inefficient, but also prone to omissions due to human fatigue and negligence. Summary of the Invention

[0004] To address this issue, the present invention provides a method and system for detecting audio-visual asynchrony in primary and backup programs, thereby solving the problem that existing technologies lack effective automatic detection methods for audio-visual synchronization, which can only be judged manually, resulting in low efficiency and easy omissions due to human fatigue and negligence.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] According to a first aspect of the present invention, a method for detecting audio-visual asynchrony in primary and backup programs is provided, the method comprising:

[0007] Acquire the main channel video signal and extract the main channel video feature value; acquire the backup channel video signal and extract the backup channel video feature value; align the main and backup channel video frames according to the main channel and backup channel video feature values; and calculate the PTS difference between the aligned main and backup channel video frames.

[0008] Acquire the main audio signal and extract the main audio feature value; acquire the backup audio signal and extract the backup audio feature value; align the main and backup audio frames according to the main and backup audio feature values; and calculate the PTS difference between the aligned main and backup audio frames.

[0009] Calculate the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and determine whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup programs has audio-visual asynchrony; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization.

[0010] Furthermore, methods for extracting video features include:

[0011] A video frame with width W and height H is divided into small squares with width w and height h. The number of small squares is calculated as: (W / w)×(H / h).

[0012] Calculate the average brightness / chromaticity of each small square in turn;

[0013] The calculated average values ​​of luminance and chrominance are used to form a small video frame with a width of W / w and a height of H / h, and this small video frame is used as the feature value of the original video frame.

[0014] Furthermore, the main and backup video frames are aligned based on the video feature values ​​of the main and backup routes, including:

[0015] The primary and backup video frames are aligned using a video comparison and synchronization search algorithm, specifically as follows:

[0016] Cache video feature values ​​of a certain length;

[0017] Take the video feature value of the most recent frame of the main program, calculate the frame difference with all video feature values ​​of the backup program cache in turn, and record the minimum frame difference as v_min_diff_m, and the corresponding frame position.

[0018] Take the video feature value of the most recent frame of the backup program, calculate the frame difference with all the video feature values ​​of the main program buffer, and record the minimum frame difference as v_min_diff_b, and the corresponding frame position.

[0019] The smaller of v_min_diff_m and v_min_diff_b is taken as the minimum frame difference for synchronization search, and the corresponding frame position is the video frame alignment result.

[0020] Furthermore, the method for calculating the frame difference includes:

[0021] Calculate the difference in brightness / chromaticity of the nth pixel within the feature value range, take the absolute value of the result, and denote it as pixel_diff_n;

[0022] Calculate the sum of chroma / luminance of the nth pixel within the feature value range, denoted as pixel_sum_n;

[0023] Calculate the offset of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n;

[0024] Calculate the average of all pixel offsets within the feature value range: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference.

[0025] Further, the PTS difference between the aligned primary and backup video frames is calculated, specifically including:

[0026] The PTS difference between the aligned primary and backup video frames is denoted as v_pts_diff:

[0027] v_pts_diff=PTS_main_v–PTS_backup_v;

[0028] Where PTS_main_v is the PTS value corresponding to the main video frame; PTS_back_v is the PTS value corresponding to the backup video frame.

[0029] Further, the PTS difference between the aligned primary and backup audio frames is calculated, specifically including:

[0030] The PTS difference between the aligned primary and backup audio frames is denoted as a_pts_diff:

[0031] a_pts_diff=PTS_main_a–PTS_backup_a;

[0032] Where PTS_main_a is the PTS value corresponding to the main audio frame; PTS_backup_a is the PTS value corresponding to the backup audio frame.

[0033] According to a second aspect of the present invention, a detection system for audio-visual asynchrony between primary and backup channels is provided, the system comprising:

[0034] The video acquisition module is used to acquire the main channel video signal and extract the main channel video feature value, and to acquire the backup channel video signal and extract the backup channel video feature value;

[0035] The video comparison module is used to align the main and backup video frames based on the video feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup video frames.

[0036] The audio acquisition module is used to acquire the main audio signal and extract the main audio feature value, and to acquire the backup audio signal and extract the backup audio feature value.

[0037] The audio comparison module is used to align the main and backup audio frames based on the audio feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup audio frames.

[0038] The synchronization detection module is used to calculate the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and to determine whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup programs has audio-visual asynchrony; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization.

[0039] Furthermore, the video comparison module is specifically used for:

[0040] The primary and backup video frames are aligned using a video comparison and synchronization search algorithm, specifically as follows:

[0041] Cache video feature values ​​of a certain length;

[0042] Take the video feature value of the most recent frame of the main program, calculate the frame difference with all video feature values ​​of the backup program cache in turn, and record the minimum frame difference as v_min_diff_m, and the corresponding frame position.

[0043] Take the video feature value of the most recent frame of the backup program, calculate the frame difference with all the video feature values ​​of the main program buffer, and record the minimum frame difference as v_min_diff_b, and the corresponding frame position.

[0044] The smaller of v_min_diff_m and v_min_diff_b is taken as the minimum frame difference for synchronization search, and the corresponding frame position is the video frame alignment result.

[0045] Furthermore, the video comparison module is specifically used for:

[0046] Calculate the difference in brightness / chromaticity of the nth pixel within the feature value range, take the absolute value of the result, and denote it as pixel_diff_n;

[0047] Calculate the sum of chroma / luminance of the nth pixel within the feature value range, denoted as pixel_sum_n;

[0048] Calculate the offset of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n;

[0049] Calculate the average of all pixel offsets within the feature value range: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference.

[0050] The present invention has the following advantages:

[0051] This invention proposes a method and system for detecting audio-visual asynchrony in primary and backup channel programs. The method involves acquiring the primary video signal and extracting its feature values, acquiring the backup video signal and extracting its feature values, aligning the primary and backup video frames based on their feature values, and calculating the PTS difference between the aligned primary and backup video frames. It also involves acquiring the primary audio signal and extracting its feature values, acquiring the backup audio signal and extracting its feature values, aligning the primary and backup audio frames based on their feature values, and calculating the PTS difference between the aligned primary and backup audio frames. Finally, the method calculates the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and determines whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup channels is audio-visually asynchronous; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization. It can automatically detect the audio-visual synchronization status of the primary and backup channels. Once audio-visual desynchronization occurs, it can issue an alarm in time, which greatly saves manpower, avoids missed reports, and improves the quality of program broadcasting. Attached Figure Description

[0052] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a method for detecting audio-visual asynchrony between primary and backup channels provided in Embodiment 1 of the present invention.

[0054] Figure 2 This is a flowchart illustrating the specific implementation of a method for detecting audio-visual asynchrony between primary and backup channels, as provided in Embodiment 1 of the present invention. Detailed Implementation

[0055] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 and Figure 2 As shown in the figure, this embodiment proposes a method for detecting audio-visual asynchrony between primary and backup programs. The method includes:

[0058] S100. Acquire the main channel video signal and extract the main channel video feature value. Acquire the backup channel video signal and extract the backup channel video feature value. Align the main and backup channel video frames according to the main and backup channel video feature values, and calculate the PTS difference between the aligned main and backup channel video frames.

[0059] Specifically, the acquisition module acquires the main channel video signal and extracts the video feature value vid_cv_m, then sends the feature value to the video comparison module; the acquisition module acquires the backup channel video signal and extracts the video feature value vid_cv_b, then sends the feature value to the video comparison module.

[0060] The video signal acquisition interface includes: SDI / ASI / HDMI / IP / ST2110 / NDI, etc., without limitation.

[0061] One method for extracting video features is as follows: 1) Divide a video frame with width W and height H into small squares with width w and height h. The number of small squares can be calculated as (W / w) x (H / h); 2) Calculate the average luminance / chrominance of each small square; 3) Use the calculated average values ​​to form a small video frame with width (W / w) and height (H / h), and use this small video frame as the feature value of the original video frame. Other methods can also be used for extracting video features; there are no limitations.

[0062] The video comparison module uses video feature values ​​to align the primary and backup video frames through synchronous search.

[0063] An algorithm for synchronous search is as follows: 1) Cache video feature values ​​of a certain length, such as 10 seconds; 2) Take the most recent frame video feature value of the main channel program, and calculate the frame difference with all video feature values ​​cached by the backup channel program in turn, recording the minimum frame difference as v_min_diff_m and the corresponding frame position; 3) Take the most recent frame video feature value of the backup channel program, and calculate the frame difference with all video feature values ​​buffered by the main channel program in turn, recording the minimum frame difference as v_min_diff_b and the corresponding frame position; 4) Take the smaller of v_min_diff_m and v_min_diff_b as the minimum frame difference for synchronous search, and its corresponding frame position is the video frame alignment result.

[0064] Corresponding to the aforementioned video feature extraction method, the frame difference is calculated as follows: 1) Calculate the difference in luminance / chrominance of the nth pixel within the feature value, and take the absolute value of the result, denoted as pixel_diff_n; 2) Calculate the sum of chrominance / luminance of the nth pixel within the feature value, denoted as pixel_sum_n; 3) Calculate the deviation of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n; 4) Calculate the average value of the deviations of all pixels within the feature value: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference. Other methods can be sampled for video comparison and synchronous search, without limitation.

[0065] The video comparison module calculates the PTS difference between the aligned primary and backup video frames, denoted as v_pts_diff:

[0066] v_pts_diff=PTS_main_v–PTS_backup_v;

[0067] PTS_main_v is the PTS value corresponding to the main video frame (PTS, Presentation TimeStamp: the timestamp used to tell the player when to display the data of this frame); PTS_back_v is the PTS value corresponding to the backup video frame.

[0068] S200: Acquire the main audio signal and extract the main audio feature value; acquire the backup audio signal and extract the backup audio feature value; align the main and backup audio frames according to the main and backup audio feature values; and calculate the PTS difference between the aligned main and backup audio frames.

[0069] The main audio signal is acquired by the acquisition module, and the audio feature value aud_cv_m is extracted. The feature value is then sent to the audio comparison module. The backup audio signal is acquired by the acquisition module, and the audio feature value aud_cv_b is extracted. The feature value is then sent to the audio comparison module.

[0070] The interfaces for audio signal acquisition include: SDI / ASI / HDMI / IP / ST2110 / NDI, etc., without limitation.

[0071] One method for extracting audio features is to downsample the audio data to 16kHz and use the resulting samples as the feature values. Other methods can also be used for audio feature extraction; there are no limitations on the methods used.

[0072] The audio comparison module uses audio feature values ​​to align the primary and backup audio frames through synchronous search.

[0073] There are already several different implementations of audio comparison and synchronization search algorithms. As long as the audio frames of the primary and backup programs can be accurately synchronized, any one of them can be used without limitation.

[0074] The audio comparison module calculates the PTS difference between the aligned primary and backup audio frames, denoted as a_pts_diff:

[0075] a_pts_diff=PTS_main_a–PTS_backup_a;

[0076] Where PTS_main_a is the PTS value corresponding to the main audio frame; PTS_backup_a is the PTS value corresponding to the backup audio frame.

[0077] S300. Calculate the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and determine whether the absolute value is greater than a preset threshold. If it is greater, determine that at least one program in the primary and backup programs has audio-visual asynchrony; otherwise, consider that both primary and backup programs are in audio-visual synchronization.

[0078] Specifically, calculate the difference between v_pts_diff and a_pts_diff, and take its absolute value, denoted as av_diff:

[0079] av_diff=abs(v_pts_diff-a_pts_diff);

[0080] If av_diff is greater than the user-defined threshold, it is determined that at least one of the main and backup programs is out of sync with the audio; otherwise, it is assumed that both the main and backup programs are in a synchronized audio-visual state.

[0081] That is, if av_diff > threshold_diff, then the audio and video are out of sync; threshold_diff is a threshold set by the user, and a typical value can be 100 milliseconds.

[0082] Example 2

[0083] Corresponding to Embodiment 1 above, this embodiment proposes a detection system for audio-visual asynchrony between primary and backup programs, the system comprising:

[0084] The video acquisition module is used to acquire the main channel video signal and extract the main channel video feature value, and to acquire the backup channel video signal and extract the backup channel video feature value;

[0085] The video comparison module is used to align the main and backup video frames based on the video feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup video frames.

[0086] The audio acquisition module is used to acquire the main audio signal and extract the main audio feature value, and to acquire the backup audio signal and extract the backup audio feature value.

[0087] The audio comparison module is used to align the main and backup audio frames based on the audio feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup audio frames.

[0088] The synchronization detection module is used to calculate the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and to determine whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup programs has audio-visual asynchrony; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization.

[0089] Furthermore, the video comparison module is specifically used for:

[0090] The primary and backup video frames are aligned using a video comparison and synchronization search algorithm, specifically as follows:

[0091] Cache video feature values ​​of a certain length;

[0092] Take the video feature value of the most recent frame of the main program, calculate the frame difference with all video feature values ​​of the backup program cache in turn, and record the minimum frame difference as v_min_diff_m, and the corresponding frame position.

[0093] Take the video feature value of the most recent frame of the backup program, calculate the frame difference with all the video feature values ​​of the main program buffer, and record the minimum frame difference as v_min_diff_b, and the corresponding frame position.

[0094] The smaller of v_min_diff_m and v_min_diff_b is taken as the minimum frame difference for synchronization search, and the corresponding frame position is the video frame alignment result.

[0095] Furthermore, the video comparison module is specifically used for:

[0096] Calculate the difference in brightness / chromaticity of the nth pixel within the feature value range, take the absolute value of the result, and denote it as pixel_diff_n;

[0097] Calculate the sum of chroma / luminance of the nth pixel within the feature value range, denoted as pixel_sum_n;

[0098] Calculate the offset of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n;

[0099] Calculate the average of all pixel offsets within the feature value range: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference.

[0100] The functions performed by each component in the detection system for audio and video asynchrony of primary and backup channels provided in this embodiment of the invention have been described in detail in the above embodiment 1, so they will not be repeated here.

[0101] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for detecting audio-visual asynchrony in primary and backup programs, characterized in that, The method includes: The process involves acquiring the main channel video signal and extracting its feature values, acquiring the backup channel video signal and extracting its feature values, aligning the main and backup channel video frames based on their feature values, and calculating the PTS difference between the aligned main and backup channel video frames. Specifically, aligning the main and backup channel video frames based on their feature values ​​includes using a video comparison synchronization search algorithm. Cache video feature values ​​of a certain length; Take the video feature value of the most recent frame of the main program, calculate the frame difference with all video feature values ​​of the backup program cache in turn, and record the minimum frame difference as v_min_diff_m and the corresponding frame position. Take the video feature value of the most recent frame of the backup program, calculate the frame difference with all the video feature values ​​of the main program buffer in turn, and record the minimum frame difference as v_min_diff_b, and the corresponding frame position. The smaller of v_min_diff_m and v_min_diff_b is taken as the minimum frame difference for synchronization search, and the corresponding frame position is the video frame alignment result. Acquire the main audio signal and extract the main audio feature value; acquire the backup audio signal and extract the backup audio feature value; align the main and backup audio frames according to the main and backup audio feature values; and calculate the PTS difference between the aligned main and backup audio frames. Calculate the absolute value of the difference between the PTS difference of the primary and backup video frames and the PTS difference of the primary and backup audio frames, and determine whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the primary and backup programs has audio-visual asynchrony; otherwise, it is considered that both primary and backup programs are in audio-visual synchronization.

2. The method for detecting audio-visual asynchrony in primary and backup programs according to claim 1, characterized in that, Methods for extracting video features include: A video frame with width W and height H is divided into small squares with width w and height h. The number of small squares is calculated as: (W / w)×(H / h). Calculate the average brightness / chromaticity of each small square in turn; The calculated average values ​​of luminance and chrominance are used to form a small video frame with a width of W / w and a height of H / h, and this small video frame is used as the feature value of the original video frame.

3. The method for detecting audio-visual asynchrony in primary and backup programs according to claim 1, characterized in that, The method for calculating the frame difference includes: Calculate the difference in brightness / chromaticity of the nth pixel within the feature value range, take the absolute value of the result, and denote it as pixel_diff_n; Calculate the sum of chroma / luminance of the nth pixel within the feature value range, denoted as pixel_sum_n; Calculate the offset of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n; Calculate the average value of all pixel offsets within the feature value range: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference.

4. The method for detecting audio-visual asynchrony in primary and backup programs according to claim 1, characterized in that, Calculate the PTS difference between the aligned primary and backup video frames, specifically including: The PTS difference between the aligned primary and backup video frames is denoted as v_pts_diff: v_pts_diff=PTS_main_v–PTS_backup_v; Where PTS_main_v is the PTS value corresponding to the main video frame; PTS_back_v is the PTS value corresponding to the backup video frame.

5. The method for detecting audio-visual asynchrony in primary and backup programs according to claim 1, characterized in that, Calculate the PTS difference between the aligned primary and backup audio frames, specifically including: The PTS difference between the aligned primary and backup audio frames is denoted as a_pts_diff: a_pts_diff=PTS_main_a–PTS_backup_a; Where PTS_main_a is the PTS value corresponding to the main audio frame; PTS_backup_a is the PTS value corresponding to the backup audio frame.

6. A system for detecting audio-visual asynchrony between primary and backup programs, characterized in that, The system includes: The video acquisition module is used to acquire the main channel video signal and extract the main channel video feature value, and to acquire the backup channel video signal and extract the backup channel video feature value; The video comparison module is used to align the main and backup video frames based on the video feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup video frames. The audio acquisition module is used to acquire the main audio signal and extract the main audio feature value, and to acquire the backup audio signal and extract the backup audio feature value. The audio comparison module is used to align the main and backup audio frames based on the audio feature values ​​of the main and backup channels, and to calculate the PTS difference between the aligned main and backup audio frames. The synchronization detection module is used to calculate the absolute value of the difference between the PTS difference of the main and backup video frames and the PTS difference of the main and backup audio frames, and to determine whether the absolute value is greater than a preset threshold. If it is greater, it is determined that at least one program in the main and backup programs has audio-visual asynchrony; otherwise, it is considered that both the main and backup programs are in audio-visual synchronization. The video comparison module is specifically used for: The primary and backup video frames are aligned using a video comparison and synchronization search algorithm, specifically as follows: Cache video feature values ​​of a certain length; Take the video feature value of the most recent frame of the main program, calculate the frame difference with all video feature values ​​of the backup program cache in turn, and record the minimum frame difference as v_min_diff_m and the corresponding frame position. Take the video feature value of the most recent frame of the backup program, calculate the frame difference with all the video feature values ​​of the main program buffer in turn, and record the minimum frame difference as v_min_diff_b, and the corresponding frame position. The smaller of v_min_diff_m and v_min_diff_b is taken as the minimum frame difference for synchronization search, and the corresponding frame position is the video frame alignment result.

7. The detection system for audio-visual asynchrony in primary and backup programs according to claim 6, characterized in that, The video comparison module is also specifically used for: Calculate the difference in brightness / chromaticity of the nth pixel within the feature value range, take the absolute value of the result, and denote it as pixel_diff_n; Calculate the sum of chroma / luminance of the nth pixel within the feature value range, denoted as pixel_sum_n; Calculate the offset of the nth pixel within the feature value: pixel_offset_n = pixel_diff_n / pixel_sum_n; Calculate the average value of all pixel offsets within the feature value range: pixel_offset = sum(pixel_offset_n) / n. Use pixel_offset as the final frame difference.

Images