Audio-picture synchronization detection method and apparatus, and device and storage medium

Abstract

Provided in the present disclosure are an audio-picture synchronization detection method and apparatus, and a device and a storage medium. The method comprises: acquiring a first video to be subjection to detection, wherein the first video is a video obtained by means of transcoding a second video with audios and pictures synchronized (S110); extracting a first key frame in the first video and second key frames in the second video, and aligning the first key frame with the second key frames, so as to determine a target second key frame which is aligned with the first key frame (S120); extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame (S130); and matching the first audio data with the second audio data, so as to determine an audio-picture synchronization detection result for the first video (S140). The present disclosure makes it possible to perform automatic audio-picture synchronization detection on a transcoded video, thereby improving the efficiency for audio-picture synchronization detection and ensuring the accuracy of audio-picture synchronization detection.

Description

Audio and video synchronization detection method, device, equipment and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202310813430.4 filed on July 4, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a method, apparatus, device and storage medium for detecting audio and video synchronization. Background Art

[0003] With the rapid development of computer technology, video transcoding is often necessary. However, during this process, audio and video desynchronization often occurs—that is, the audio and video images do not accurately match. For example, a video may show someone speaking but without the corresponding sound, significantly reducing the user's viewing experience. Currently, manual video monitoring is often used to determine whether transcoding has caused audio and video desynchronization. This manual detection method is time-consuming and labor-intensive, reducing the efficiency of audio and video synchronization detection.

[0004] Summary of the Invention

[0005] The present disclosure provides a method, apparatus, device and storage medium for detecting audio and video synchronization, so as to automatically detect audio and video synchronization of transcoded videos, thereby improving the detection efficiency of audio and video synchronization and ensuring the accuracy of audio and video synchronization detection.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for detecting audio and video synchronization, including:

[0007] Obtaining a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

[0008] Extracting a first key frame from the first video and a second key frame from the second video, aligning the first key frame with the second key frame, and determining a target second key frame aligned with the first key frame;

[0009] Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

[0010] The first audio data is matched with the second audio data to determine an audio-video synchronization detection result of the first video.

[0011] In a second aspect, the embodiments of the present disclosure further provide an audio-video synchronization detection device, comprising:

[0012] A first video acquisition module is configured to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

[0013] a key frame alignment module, configured to extract a first key frame from the first video and a second key frame from the second video, align the first key frame with the second key frame, and determine a target second key frame aligned with the first key frame;

[0014] an audio data extraction module, configured to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

[0015] The audio-video synchronization detection module is configured to match the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video.

[0016] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio-visual synchronization detection method as described in any one of the embodiments of the present disclosure.

[0020] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the audio-visual synchronization detection method as described in any one of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0022] FIG1 is a flow chart of a method for detecting audio and video synchronization provided by an embodiment of the present disclosure;

[0023] FIG2 is a waveform matching diagram of first audio data and second audio data involved in an embodiment of the present disclosure;

[0024] FIG3 is a flow chart of another method for detecting audio and video synchronization provided by an embodiment of the present disclosure;

[0025] FIG4 is an example of audio data extraction according to an embodiment of the present disclosure;

[0026] FIG5 is a schematic structural diagram of an audio-video synchronization detection device provided by an embodiment of the present disclosure;

[0027] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0031] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] Figure 1 is a flow chart of a method for detecting audio and video synchronization provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where audio and video synchronization detection is performed on a transcoded video. The method can be performed by an audio and video synchronization detection device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0035] As shown in FIG1 , the audio-video synchronization detection method specifically includes the following steps:

[0036] S110: Obtain a first video to be detected, where the first video is obtained by transcoding a second video with synchronized audio and video.

[0037] The first video is the transcoded video to be tested. The second video is the pre-transcoded video. The audio and video images in the second video are synchronized, so the second video can be used as a reference video to test the audio and video synchronization of the first video.

[0038] Specifically, users can create and publish videos in the client. After receiving the video uploaded by the client, the server needs to transcode the uploaded video, such as adjusting the video resolution and bit rate, so as to improve the picture quality, and send the transcoded video to other clients for playback. In the above application scenario, the video uploaded to the server can be used as the second video with audio and video synchronization, and the transcoded video can be used as the first video to be detected. By using the second video before transcoding as the reference video for audio and video synchronization, it is possible to accurately detect whether the first video has audio and video asynchrony caused by video transcoding.

[0039] S120: Extract a first key frame from the first video and a second key frame from the second video, align the first key frame and the second key frame, and determine a target second key frame aligned with the first key frame.

[0040] Among them, a key frame may refer to a video frame that plays a decisive role in the video content, such as a video frame with scene switching, a video frame with motion changes, a video frame where a key action is located, etc. A first key frame may refer to a key frame in the first video. The number of first key frames can be one or more. A second key frame may refer to a key frame in the second video. The number of second key frames can also be one or more. The number of first key frames and the number of second key frames may be the same or different. Alignment refers to the operation of aligning key frames with the same picture. The target second key frame refers to the second key frame with the same picture as the first key frame.

[0041] Specifically, key frames of the first video and the second video are detected, and all first key frames in the first video and all second key frames in the second video are extracted. The first key frames and the second key frames can be aligned by detecting the image similarity between the first key frame and the second key frame. For example, for each first key frame, the image similarity between the first key frame and each second key frame can be determined, and the second key frame with the highest image similarity that is greater than or equal to a preset similarity threshold can be used as the target second key frame to be aligned with the first key frame, thereby obtaining a target second key frame having the same picture as the first key frame.

[0042] Image similarity can be represented using any metric that can measure the similarity between two images. For example, image similarity can be represented using the Structural Similarity (SSIM) metric. The SSIM metric ranges from 0 to 1, with a larger SSIM metric indicating higher image similarity.

[0043] It should be noted that if there is no target second key frame that is the same as a first key frame among all the second key frames, it means that the first key frame cannot be aligned. At this time, the first key frame can be ignored, and only the first key frame that can be aligned can be used for subsequent audio and video synchronization detection.

[0044] S130: Extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame.

[0045] Specifically, for each first key frame, the first audio data corresponding to the preset audio duration can be extracted at the first key frame position in the first video. The second audio data corresponding to the preset audio duration is also extracted at the target second key frame position in the second video, so that the extracted first audio data and the second audio data have the same audio duration. For example, the extracted first audio data is audio data with a preset audio duration based on the playback timestamp of the first key frame as the preset reference moment. The extracted second audio data is audio data with a preset audio duration based on the playback timestamp of the target second key frame as the preset reference moment. The preset reference moment may refer to the start moment, center moment or end moment of the audio data.

[0046] It should be noted that the picture content of the aligned first key frame and the target second video frame is the same, but their corresponding playback timestamps may be different. When the extracted first audio data and second audio data have the same audio duration, the first audio data and the second audio data need to be extracted at the same position of the key frame to ensure the accuracy of subsequent audio and video synchronization detection. For example, the first audio data with a preset audio duration starting at the playback timestamp of the first key frame is extracted from the first video. The second audio data with a preset audio duration starting at the playback timestamp of the target second key frame is extracted from the second video.

[0047] S140: Match the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video.

[0048] Specifically, Figure 2 shows a waveform matching diagram for first and second audio data. As shown in Figure 2, the first and second audio data have the same audio duration. Therefore, the matching result between the first and second audio data can be determined based on the similarity between the two audio waveforms in Figure 2. For example, the covariance between the first and second audio data can be used as the audio similarity between the first and second audio data. When the audio similarity is greater than or equal to a preset similarity threshold, the first and second audio data are determined to be successfully matched. When the audio similarity is less than the preset similarity threshold, the first and second audio data are determined to be unmatched. If the match is successful, the audio and video images at the first keyframe are synchronized. If the match is unsuccessful, the audio and video images at the first keyframe are out of sync, indicating that there is audio and video asynchrony caused by transcoding. Based on the matching results corresponding to each first keyframe in the first video, the audio and video synchronization detection result of the first video can be determined. For example, if the matching results for all first keyframes in the first video are successful, the audio and video synchronization detection result of the first video is determined to be synchronized. If the matching result corresponding to at least one first key frame in the first video is a matching failure, it is determined that the audio and video synchronization detection result of the first video is audio and video asynchrony, and the audio and video asynchrony in the first video is caused by transcoding.

[0049] The technical solution of the embodiment of the present disclosure detects the transcoded first video by using the second video with synchronized audio and video before transcoding as a reference, and matches the first audio data corresponding to the first key frame after key frame alignment with the second audio data corresponding to the target second key frame, so as to accurately determine whether the first video has audio and video asynchrony caused by transcoding, realize automatic detection of audio and video synchronization, and ensure the accuracy of audio and video synchronization detection.

[0050] As an optional embodiment, "aligning the first key frame and the second key frame to determine a target second key frame aligned with the first key frame" in S120 may include: obtaining an optional second key frame sequence for the first key frame; obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame; if the image similarity is greater than or equal to a preset similarity threshold, determining the current second video frame as the target second key frame aligned with the first key frame.

[0051] Image similarity can be represented using any metric that can measure the similarity between two images. For example, image similarity can be represented using the Structural Similarity (SSIM) metric. The SSIM metric ranges from 0 to 1, with a larger SSIM metric indicating higher image similarity.

[0052] Specifically, all extracted first key frames can be sorted in the order in which the video was played, obtaining a first key frame sequence, such as {A1, A2, ..., An}. Similarly, all extracted second key frames can be sorted in the order in which the video was played, obtaining a second key frame sequence, such as {B1, B2, ..., Bm}. The target second key frame to be aligned with each first key frame can be determined from the second key frame sequence in the order of the first key frame sequence.

[0053] For example, the optional second key frame sequence of the first first key frame A1 is {B1, B2, ..., Bm}. At this time, according to the order of the second key frame sequence, the second key frame B1 can be first used as the current second key frame, and the image similarity between the first key frame A1 and the second key frame B1 can be detected to be greater than or equal to the preset similarity threshold; if so, it indicates that the first key frame A1 and the second key frame B1 are similar in picture. At this time, the second key frame B1 can be directly used as the target second key frame to be aligned with the first key frame A1; if not, the next second key frame B2 is used as the current second key frame to re-detect whether the image similarity between the first key frame A1 and the second key frame B2 is greater than or equal to the preset similarity threshold, until the second key frame matching the first key frame A1 is determined, thereby completing the alignment of the first key frame A1. Assuming that the target second key frame aligned with the first key frame A1 is B2, when aligning the second first key frame A2, since the target second key frame aligned with the first key frame A2 appears in the key frame after the second key frame B2, the second key frame after the second key frame B2 can be used as the optional second key frame sequence of the first key frame A2, that is, {B3, B4, ..., Bm}, and based on the same detection process as above, the target second key frame aligned with the first key frame A2 is determined from the second key frame sequence {B3, B4, ..., Bm}. Similarly, the target second key frame aligned with each first key frame can be obtained more quickly, thereby improving the key frame alignment speed and further improving the detection efficiency of audio and video synchronization.

[0054] Figure 3 is a flow chart of another audio-video synchronization detection method provided by an embodiment of the present disclosure. Based on the aforementioned disclosed embodiments, this disclosed embodiment optimizes the step of "matching the first audio data with the second audio data to determine the audio-video synchronization detection result for the first video." Explanations of terms that are identical or corresponding to those in the aforementioned disclosed embodiments are omitted here.

[0055] As shown in FIG3 , the audio-video synchronization detection method specifically includes the following steps:

[0056] S310: Obtain a first video to be detected, where the first video is obtained by transcoding a second video with synchronized audio and video.

[0057] S320: Extract a first key frame from the first video and a second key frame from the second video, align the first key frame and the second key frame, and determine a target second key frame aligned with the first key frame.

[0058] S330: Extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame.

[0059] The audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration. The first preset duration is greater than the second preset duration. For example, the first preset duration is 500ms and the second preset duration is 20ms.

[0060] Specifically, for each first key frame, first audio data corresponding to a first preset duration can be extracted at the first key frame position in the first video. Second audio data corresponding to a second preset duration can be extracted at the target second key frame position in the second video. This ensures that the audio duration of the extracted first audio data is greater than the audio duration of the second audio data, allowing the second audio data to be subsequently matched with the first audio data.

[0061] Exemplarily, the extraction of the first audio data needs to be performed within the left and right range of the first key frame position to facilitate subsequent accurate audio matching. For example, the first audio data may be audio data having a first preset duration centered around the playback timestamp of the first key frame. FIG4 shows an example of extracting the first audio data. Referring to FIG4 , first audio data having a first preset duration of 500ms, centered around the playback timestamp of each first key frame, is extracted from the audio of the first video.

[0062] Exemplarily, there are multiple ways to extract the second audio data used as a reference, and it can be extracted to the left, left, or left and right of the target second key frame position. For example, the second audio data can be audio data with a second preset duration and a preset reference moment based on the playback timestamp of the target second key frame. The preset reference moment can be one of the following: a start moment, a center moment, and an end moment. FIG4 also shows an example of second audio data extraction. Referring to FIG4 , second audio data with a second preset duration of 20ms and a start moment based on the playback timestamp of each target second key frame is extracted from the audio of the second video.

[0063] S340: Determine, from the first audio data, target audio data having a second preset duration and matching the second audio data.

[0064] The target audio data may refer to the audio data in the first audio data that successfully matches the second audio data. In other words, the target audio data and the second audio data may be approximately the same data.

[0065] Specifically, because the duration of the first audio data is greater than that of the second audio data, the first audio data can be traversed to extract multiple candidate audio data with a second preset duration, for example, extracting one candidate audio data with the second preset duration every 1 ms. The audio similarity between each candidate audio data and the second audio data can be determined, and the candidate audio data with the greatest audio similarity is selected as the target audio data to match the second audio data.

[0066] Exemplarily, S340 may include: sliding the first audio data based on a preset sliding duration and a second preset duration, obtaining the current audio data with the second preset duration corresponding to the sliding; determining the audio similarity between the current audio data and the second audio data; and determining the target audio data that matches the second audio data based on the audio similarity corresponding to the current audio data.

[0067] Specifically, based on the preset sliding duration, the first audio data can be slid successively to obtain current audio data with a second preset duration. For example, when the preset sliding duration is 1ms, referring to Figure 4, the current audio data obtained by the first sliding is the audio data from 0ms to 20ms in the first audio data, the current audio data obtained by the second sliding is the audio data from 1ms to 21ms in the first audio data, and so on. After each sliding, the covariance between the current audio data obtained by the sliding and the second audio data can be determined, and the covariance can be used as the audio similarity corresponding to the current audio data. If the audio similarity is greater than or equal to the preset similarity threshold, it can be determined that the current audio data matches the second audio data, and the current audio data is used as the target audio data. If the audio similarity is less than the preset similarity threshold, it can be determined that the current audio data does not match the second audio data. At this time, it is necessary to update the current audio data by sliding, and detect whether the current audio data obtained by the next sliding matches the second audio data. When the matching target audio data is determined, the sliding is stopped. Through the above-mentioned sliding matching method, there is no need to traverse all audio data with the second preset time length in the first audio data. The target audio data matching the second audio data can be obtained more quickly while ensuring the accuracy of synchronization detection, thereby further improving the efficiency of audio and video synchronization detection.

[0068] S350: Determine an audio-visual offset between the target audio data and the first key frame using the second audio data and the target second key frame as an audio-visual synchronization reference.

[0069] The audio-video offset may refer to the time difference between the sound of the target audio data and the picture of the first key frame.

[0070] Specifically, since the second video is synchronized with the audio and video, and thus the second audio data and the target second key frame are also synchronized with the audio and video, the first time difference between the second audio data and the target second key frame can be used as a reference time difference for audio and video synchronization. The target time difference between the target audio data and the first key frame is detected. If the target time difference is equal to the reference time difference, it indicates that the target audio data and the first key frame are synchronized with the audio and video, and the audio and video offset is 0. If the target time difference is not equal to the reference time difference, the specific audio and video offset can be determined based on the difference between the target time difference and the reference time difference.

[0071] Exemplarily, S350 may include: determining the reference time difference for audio and video synchronization based on the start timestamp of the second audio data and the playback timestamp of the target second key frame; determining the target time difference corresponding to the first key frame based on the start timestamp of the target audio data and the playback timestamp of the first key frame; determining the audio and video offset corresponding to the first key frame based on the target time difference and the reference time difference.

[0072] Specifically, the start timestamp of the second audio data can be subtracted from the playback timestamp of the target second key frame, and the time difference obtained can be determined as the reference time difference for audio and video synchronization. For the extraction method of the second audio data in Figure 4, the start timestamp of the second audio data is the playback timestamp of the target second key frame, so that the reference time difference for audio and video synchronization is 0. The start timestamp of the target audio data is subtracted from the playback timestamp of the first key frame, and the time difference obtained is determined as the target time difference corresponding to the first key frame. The target time difference is subtracted from the reference time difference, and the time difference obtained is determined as the audio and video offset corresponding to the first key frame. Referring to Figure 4, the reference parallax is 0, and the target time difference is the audio and video offset. If the audio and video offset is greater than 0, it indicates that the audio is ahead of the video picture. If the audio and video offset is less than 0, it indicates that the audio lags behind the video picture.

[0073] S360: Determine an audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame.

[0074] Specifically, based on the audio and video offset corresponding to each first key frame in the first video, the audio and video synchronization detection result of the first video as a whole can be determined.

[0075] Exemplarily, S360 may include: if the audio and video offset corresponding to each first video frame in the first video is within a preset allowable range, then determining that the audio and video synchronization detection result of the first video is audio and video synchronization; if there is at least one first video frame in the first video whose audio and video offset corresponding to the first video is not within the preset allowable range, then determining that the audio and video synchronization detection result of the first video is audio and video synchronization.

[0076] The preset allowable range may be a time difference range that is pre-set based on a detection standard and allows for audio and video offset. For example, the preset allowable range is [-185, 90], which indicates that, relative to the video image, an audio lag of 185ms or an audio advance of 90ms is acceptable and considered to be synchronized with the video.

[0077] Specifically, it is detected whether the audio and video offset corresponding to each first video frame in the first video is within the preset allowable range. If the audio and video offsets corresponding to all the first video frames are within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video synchronization, that is, the audio and video of the transcoded first video is also synchronized, and the video transcoding does not cause audio and video asynchrony. If there is at least one first video frame whose audio and video offset is not within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video asynchrony, that is, audio and video asynchrony caused by transcoding occurs at the position of the first video frame that is not within the preset allowable range. At this time, the audio and video offset corresponding to the first video frame that is not within the preset allowable range can be output to remind the user of the specific position of the audio and video asynchrony and the degree of the audio and video asynchrony, so that the user can quickly process it.

[0078] The technical solution of the embodiment of the present disclosure extracts first audio data with a longer audio duration, extracts target audio data matching the second audio data again from the first audio data, and uses the second audio data and the target second key frame as the audio-visual synchronization benchmark to determine the audio-visual offset between the target audio data and the first key frame. Based on the audio-visual offset corresponding to the first key frame, the audio-visual synchronization detection result of the first video can be determined more accurately, thereby further improving the accuracy of the audio-visual synchronization detection.

[0079] Figure 5 is a structural diagram of an audio-video synchronization detection device provided by an embodiment of the present disclosure. As shown in Figure 5, the device specifically includes: a first video acquisition module 510, a key frame alignment module 520, an audio data extraction module 530 and an audio-video synchronization detection module 540.

[0080] Among them, the first video acquisition module 510 is used to obtain the first video to be detected, where the first video is a video obtained by transcoding the second video with synchronized audio and video; the key frame alignment module 520 is used to extract the first key frame in the first video and the second key frame in the second video, and align the first key frame and the second key frame to determine the target second key frame aligned with the first key frame; the audio data extraction module 530 is used to extract the first audio data corresponding to the first key frame and the second audio data corresponding to the target second key frame; the audio and video synchronization detection module 540 is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

[0081] The technical solution provided by the embodiment of the present disclosure detects the transcoded first video by using the second video with synchronized audio and video before transcoding as a reference, and matches the first audio data corresponding to the first key frame after key frame alignment with the second audio data corresponding to the target second key frame, so as to accurately determine whether the first video has audio and video asynchrony caused by transcoding, thereby realizing automatic detection of audio and video synchronization and ensuring the accuracy of audio and video synchronization detection.

[0082] Based on the above technical solution, the key frame alignment module 520 is specifically configured to:

[0083] Obtain an optional second key frame sequence for the first key frame; obtain the current second key frame in sequence according to the order of the second key frame sequence, and determine the image similarity between the first key frame and the current second key frame; if the image similarity is greater than or equal to a preset similarity threshold, determine the current second video frame as the target second key frame aligned with the first key frame.

[0084] Based on the above technical solutions, the audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first preset duration is greater than the second preset duration.

[0085] Based on the above technical solutions, the first audio data is audio data with a first preset duration and a playback timestamp of the first key frame as the center moment;

[0086] The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following:

[0087] The beginning moment, the center moment, and the end moment.

[0088] Based on the above technical solutions, the audio and video synchronization detection module 540 includes:

[0089] a target audio data determining unit, configured to determine, from the first audio data, target audio data having the second preset duration and matching the second audio data;

[0090] an audio-video offset determining unit, configured to determine an audio-video offset between the target audio data and the first key frame using the second audio data and the target second key frame as an audio-video synchronization reference;

[0091] The audio-video synchronization detection unit is configured to determine an audio-video synchronization detection result of the first video based on the audio-video offset corresponding to the first key frame.

[0092] Based on the above technical solutions, the target audio data determination unit is specifically configured to:

[0093] Based on the preset sliding duration and the second preset duration, the first audio data is slid to obtain the current audio data with the second preset duration corresponding to the sliding; the audio similarity between the current audio data and the second audio data is determined; based on the audio similarity corresponding to the current audio data, the target audio data that matches the second audio data is determined.

[0094] Based on the above technical solutions, the audio-video offset determination unit is specifically configured to:

[0095] Based on the start timestamp of the second audio data and the playback timestamp of the target second key frame, determine the reference time difference for audio and video synchronization; based on the start timestamp of the target audio data and the playback timestamp of the first key frame, determine the target time difference corresponding to the first key frame; based on the target time difference and the reference time difference, determine the audio and video offset corresponding to the first key frame.

[0096] Based on the above technical solutions, the audio and video synchronization detection unit is specifically used to:

[0097] If the audio and video offset corresponding to each first video frame in the first video is within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video synchronization; if there is at least one first video frame in the first video whose audio and video offset corresponding to the first video is not within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video synchronizationlessness.

[0098] The audio-video synchronization detection device provided in the embodiments of the present disclosure can execute the audio-video synchronization detection method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the audio-video synchronization detection method.

[0099] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0100] FIG6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG6 , a schematic diagram of the structure of an electronic device (such as a terminal device or server in FIG6 ) 500 suitable for implementing an embodiment of the present disclosure is shown below. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG6 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0101] As shown in FIG6 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0102] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 illustrates the electronic device 500 with various devices, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0103] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0104] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0105] The electronic device provided by the embodiment of the present disclosure and the audio-visual synchronization detection method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0106] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the audio and video synchronization detection method provided in the above embodiment is implemented.

[0107] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0108] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0109] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0110] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; extracts a first key frame in the first video and a second key frame in the second video, and aligns the first key frame and the second key frame to determine a target second key frame aligned with the first key frame; extracts first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; matches the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

[0111] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0114] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] According to one or more embodiments of the present disclosure, Example 1 provides a method for detecting audio and video synchronization, including:

[0117] Obtaining a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

[0118] Extracting a first key frame from the first video and a second key frame from the second video, aligning the first key frame with the second key frame, and determining a target second key frame aligned with the first key frame;

[0119] Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

[0120] The first audio data is matched with the second audio data to determine an audio-video synchronization detection result of the first video.

[0121] According to one or more embodiments of the present disclosure, Example 2 provides a method for detecting audio-video synchronization, further comprising:

[0122] Optionally, aligning the first key frame and the second key frame to determine a target second key frame aligned with the first key frame includes:

[0123] Obtain a second key frame sequence that is optional for the first key frame;

[0124] Obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame;

[0125] If the image similarity is greater than or equal to a preset similarity threshold, the current second video frame is determined as a target second key frame aligned with the first key frame.

[0126] According to one or more embodiments of the present disclosure, Example 3 provides a method for detecting audio-video synchronization, further comprising:

[0127] Optionally, the audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first preset duration is greater than the second preset duration.

[0128] According to one or more embodiments of the present disclosure, Example 4 provides a method for detecting audio-video synchronization, further comprising:

[0129] Optionally, the first audio data is audio data having a first preset duration with a playback timestamp of the first key frame as the center moment;

[0130] The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following:

[0131] The beginning moment, the center moment, and the end moment.

[0132] According to one or more embodiments of the present disclosure, Example 5 provides a method for detecting audio-video synchronization, further comprising:

[0133] Optionally, matching the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video includes:

[0134] Determining, from the first audio data, target audio data having the second preset duration and matching the second audio data;

[0135] Determining an audio / video offset between the target audio data and the first key frame using the second audio data and the target second key frame as an audio / video synchronization reference;

[0136] An audio and video synchronization detection result of the first video is determined based on the audio and video offset corresponding to the first key frame.

[0137] According to one or more embodiments of the present disclosure, Example 6 provides a method for detecting audio-video synchronization, further comprising:

[0138] Optionally, determining, from the first audio data, target audio data having the second preset duration and matching the second audio data includes:

[0139] Slide the first audio data based on the preset sliding duration and the second preset duration, and obtain current audio data corresponding to the sliding with the second preset duration;

[0140] determining an audio similarity between the current audio data and the second audio data;

[0141] Based on the audio similarity corresponding to the current audio data, target audio data matching the second audio data is determined.

[0142] According to one or more embodiments of the present disclosure, Example 7 provides a method for detecting audio-video synchronization, further comprising:

[0143] Optionally, determining the audio-visual offset between the target audio data and the first key frame using the second audio data and the target second key frame as an audio-visual synchronization reference includes:

[0144] Determining a reference time difference for audio and video synchronization based on a start timestamp of the second audio data and a playback timestamp of the target second key frame;

[0145] Determining a target time difference corresponding to the first key frame based on a start timestamp of the target audio data and a play timestamp of the first key frame;

[0146] Based on the target time difference and the reference time difference, an audio and video offset corresponding to the first key frame is determined.

[0147] According to one or more embodiments of the present disclosure, Example 8 provides a method for detecting audio-video synchronization, further comprising:

[0148] Optionally, determining the audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame includes:

[0149] If the audio and video offset corresponding to each first video frame in the first video is within a preset allowable range, determining that the audio and video synchronization detection result of the first video is audio and video synchronization;

[0150] If the audio and video offset corresponding to at least one first video frame in the first video is not within a preset allowable range, it is determined that the audio and video synchronization detection result of the first video is that the audio and video are out of synchronization.

[0151] According to one or more embodiments of the present disclosure, Example 9 provides an audio-video synchronization detection device, including:

[0152] A first video acquisition module is configured to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

[0153] a key frame alignment module, configured to extract a first key frame from the first video and a second key frame from the second video, align the first key frame with the second key frame, and determine a target second key frame aligned with the first key frame;

[0154] an audio data extraction module, configured to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

[0155] The audio-video synchronization detection module is configured to match the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video.

[0156] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0157] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0158] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for detecting synchronization between audio and video, comprising: Acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; Extracting a first key frame in the first video and a second key frame in the second video, aligning the first key frame and the second key frame, and determining a target second key frame aligned with the first key frame; Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; The first audio data is matched with the second audio data to determine an audio-visual synchronization detection result of the first video.

2. The method for detecting audio-visual synchronization according to claim 1, wherein: Aligning the first key frame and the second key frame to determine a target second key frame aligned with the first key frame includes: Obtain a second key frame sequence that is optional for the first key frame; Obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame; If the image similarity is greater than or equal to a preset similarity threshold, the current second video frame is determined as a target second key frame aligned with the first key frame.

3. The method for detecting audio-visual synchronization according to claim 1, wherein: The audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first preset duration is greater than the second preset duration.

4. The method for detecting audio-visual synchronization according to claim 3, wherein: The first audio data is audio data with a play timestamp of the first key frame as the center moment and having a first preset duration; The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following: The beginning moment, the center moment, and the end moment.

5. The method for detecting audio-visual synchronization according to claim 3, wherein: The matching the first audio data with the second audio data to determine the audio-video synchronization detection result of the first video includes: Determine, from the first audio data, target audio data having the second preset duration and matching the second audio data; Taking the second audio data and the target second key frame as an audio-visual synchronization reference, determining an audio-visual offset between the target audio data and the first key frame; Based on the audio and video offset corresponding to the first key frame, an audio and video synchronization detection result of the first video is determined.

6. The method for detecting audio-visual synchronization according to claim 5, wherein: The determining, from the first audio data, target audio data having the second preset duration and matching the second audio data comprises: Based on the preset sliding duration and the second preset duration, sliding the first audio data, and acquiring the current audio data with the second preset duration corresponding to the sliding; Determining audio similarity between the current audio data and the second audio data; Based on the audio similarity corresponding to the current audio data, target audio data matching the second audio data is determined.

7. The method for detecting audio-visual synchronization according to claim 5 or 6, wherein: The step of determining the audio-visual offset between the target audio data and the first key frame by taking the second audio data and the target second key frame as an audio-visual synchronization reference includes: Determine a reference time difference for audio and video synchronization based on a start timestamp of the second audio data and a play timestamp of the target second key frame; Determining a target time difference corresponding to the first key frame based on a start timestamp of the target audio data and a play timestamp of the first key frame; Based on the target time difference and the reference time difference, an audio and video offset corresponding to the first key frame is determined.

8. The method for detecting audio-visual synchronization according to any one of claims 5 to 7, wherein: The determining, based on the audio and video offset corresponding to the first key frame, the audio and video synchronization detection result of the first video includes: If the audio and video offset corresponding to each first video frame in the first video is within a preset allowable range, determining that the audio and video synchronization detection result of the first video is audio and video synchronization; If the audio and video offset corresponding to at least one first video frame in the first video is not within a preset allowable range, it is determined that the audio and video synchronization detection result of the first video is that the audio and video are out of synchronization.

9. A device for detecting synchronization between audio and video, comprising: A first video acquisition module is used to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; A key frame alignment module, used to extract a first key frame in the first video and a second key frame in the second video, and align the first key frame with the second key frame to determine a target second key frame aligned with the first key frame; An audio data extraction module, used to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; The audio and video synchronization detection module is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

10. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the audio-visual synchronization detection method as described in any one of claims 1-8.

11. A storage medium comprising computer executable instructions, wherein: The computer executable instructions are used to execute the audio-video synchronization detection method as described in any one of claims 1-8 when executed by a computer processor.

Images