Sound and picture synchronization detection method and device, electronic equipment and terminal

By extracting and matching the subtitle information of image frames and audio frames from videos, the accuracy and reliability of audio and video synchronization detection are achieved, and the problem of low accuracy of audio and video synchronization detection in the prior art is solved, which improves the user experience.

CN119031124APending Publication Date: 2024-11-26BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411262515.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The accuracy of the intermediate audio and video synchronization detection in the prior art is low, affecting the user's viewing experience.

Method used

By synchronously extracting image frames and audio frames from video, identifying image frames with the same subtitles, and determining synchronized audio frames based on the time stamps of image frames and audio frames, using subtitles information to perform audio and video synchronization detection, improving the accuracy of detection results Sex and reliability.

Benefits of technology

It improves the accuracy and reliability of audio and video synchronization detection, avoids missed detection problems, reduces detection costs, and improves the user's experience of watching videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031124A_ABST
    Figure CN119031124A_ABST
Patent Text Reader

Abstract

The invention provides a sound and picture synchronization detection method and device, electronic equipment and a terminal, and relates to the field of image processing, in particular to the technical fields of computer vision, artificial intelligence and the like, and the method comprises the steps: synchronously extracting a first image frame and a first audio frame from a video; determining at least one second image frame with the same subtitle from the first image frames; determining a second audio frame synchronous with the second image frame from the first audio frame; and according to the second image frame and the second audio frame, carrying out audio and picture synchronization detection on the video. According to the method and the device, the at least one second image frame with the same subtitle is determined from the first image frames, and the audio and picture synchronization detection is performed on the video according to the second image frame and the second audio frame synchronized with the second image frame, so that the accuracy and the reliability of obtaining the audio and picture synchronization detection result of the video are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the technical fields of computer vision and artificial intelligence, and in particular to a method, device, electronic device and terminal for detecting synchronization between audio and video. Background Art

[0002] In the process of video editing, encoding, and playback, the subtitles and sounds in the video may not match, affecting the user's viewing experience. In related technologies, reference methods are often used to detect audio and video synchronization, such as adding marker information or comparing with reference videos, or using non-reference methods to detect audio and video synchronization. However, the above methods often obtain audio and video detection results with low accuracy. Therefore, how to improve the accuracy and reliability of audio and video synchronization detection has become an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, terminal and computer program product for detecting audio and video synchronization.

[0004] According to a first aspect of the present disclosure, a method for detecting audio and video synchronization is proposed, comprising: synchronously extracting a first image frame and a first audio frame from a video; determining at least one second image frame having the same subtitles from the first image frame; determining a second audio frame synchronized with the second image frame from the first audio frame; and performing audio and video synchronization detection on the video based on the second image frame and the second audio frame.

[0005] According to a second aspect of the present disclosure, a device for detecting audio and video synchronization is proposed, comprising: an extraction module, used to synchronously extract a first image frame and a first audio frame from a video; a first determination module, used to determine at least one second image frame having the same subtitles from the first image frame; a second determination module, used to determine a second audio frame synchronized with the second image frame from the first audio frame; and a detection module, used to perform audio and video synchronization detection on the video based on the second image frame and the second audio frame.

[0006] According to the third aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio and video synchronization detection method proposed in the first aspect above.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a terminal is provided, comprising the electronic device proposed in the embodiment of the third aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is proposed, including a computer program, wherein when the computer program is executed by a processor, the audio and video synchronization detection method proposed in the first aspect is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;

[0012] Figure 2 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;

[0013] Figure 3 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;

[0014] Figure 4 It is a structural schematic diagram of an audio and video synchronization detection device according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure;

[0016] Figure 6 A schematic block diagram of a terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.

[0019] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.

[0020] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0021] In the embodiment of the present disclosure, a video reading tool can be used to read the video to be subjected to audio and video synchronization detection, and the video can be decoded by the video reading tool to obtain the video, and audio and video detection can be performed on the video to obtain the audio and video synchronization detection result of the video.

[0022] The video decoding formats include but are not limited to: (1) H.264 / AVC (Advanced Video Coding), the video decoder is libx264; (2) H.265 / HEVC (High Efficiency Video Coding), the video decoder is libx265; (3) VP8 (Video Coding Format by Google), the video decoder is libvpx; (4) VP9 (Successor to VP8), the video decoder is libvpx-vp9; (5) AV1 (AOMedia Video1), the video decoder is libaom-av1; (6) MPEG-2 (Moving Picture Experts Group Phase2), the video decoder is mpeg2video; (7) MPEG-4 (Moving Picture Experts Group Phase4), the video decoder is mpeg4; (8) Theora (Free and Open Video Codec), the video decoder is libtheora; ProRes (AppleProRes), the video decoder is prores; (10) DNxHD (Digital Nonlinear Extensible High Definition), the video decoder is dnxhd; (11) WMV (Windows Media Video), the video decoder is wmv2; (12) FLV1 (Flash Video), the video decoder is flv; (13) MJPEG (Motion JPEG), the video decoder is mjpeg

[0023] The audio decoding formats include but are not limited to: (1) AAC (Advanced Audio Coding), the audio decoder is aac; (2) MP3 (MPEG-1 Audio Layer 3), the audio decoder is libmp3lame; (3) Vorbis (Ogg Vorbis), the audio decoder is libvorbis; (4) Opus (Audio Codec), the audio decoder is libopus; (5) FLAC (Free Lossless Audio Codec), the audio decoder is flac; (6) ALAC (Apple Lossless Audio Code), the audio decoder is alac; (7) AC3 (Audio Codec 3), the audio decoder is ac3; (8) WMA (Windows Media Audio), the audio decoder is wmav2; (9) WAV (Waveform Audio File Format), the audio decoder is the built-in WAV decoder; (10) AMR (Adaptive Audio File Format), the audio decoder is the built-in WAV decoder; Multi-Rate) encoder: the audio decoder is libopencore_amrnb (NB) and libopencore_amrwb (WB); (11) PCM (Pulse Code Modulation), the audio decoder is pcm_s16le, pcm_s24le, etc.

[0024] In the video encoding and decoding process, the basic process is as follows: in the encoder, a frame of image is divided into blocks, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, the original block of the current block minus the prediction block to obtain the residual block, the residual block is transformed and quantized to obtain the quantization coefficient matrix, and the quantization coefficient matrix is ​​entropy encoded and output to the bitstream. In the decoder, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, on the other hand, the bitstream is decoded to obtain the quantization coefficient matrix, the quantization coefficient matrix is ​​inversely quantized and inversely transformed to obtain the residual block, and the prediction block and the residual block are added to obtain the reconstructed block. The reconstructed blocks constitute the reconstructed image, and the reconstructed image is loop-filtered based on the image or block to obtain the decoded image. The encoder also needs similar operations as the decoder to obtain the decoded image. The decoded image can be used as a reference frame for inter-frame prediction for subsequent frames. The block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information determined by the encoder need to be output to the bitstream if necessary; then the decoder determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information as the encoder by parsing and analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder. The decoded image obtained by the encoder is also usually called a reconstructed image. The current block can be divided into prediction units during prediction, and the current block can be divided into transformation units during transformation. The division of prediction units and transformation units can be different. The above is the basic process of the video encoder and decoder under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The embodiment of the present application is applicable to the basic process under the block-based hybrid coding framework, but is not limited to the framework and process.

[0025] At present, the common video coding and decoding standards all adopt the block-based hybrid coding framework. Each frame in the video image is divided into square LCUs of the same size (such as 128×128, 64×64, etc.), and each LCU can also be divided into rectangular CUs according to rules; and CUs may also be divided into smaller PUs, TUs, etc. Specifically, the coding framework may include steps such as prediction, transform, quantization, entropy coding, and in-loop filtering. Among them, prediction can be divided into intra-frame prediction and inter-frame prediction, and inter-frame prediction can include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the use of intra-frame prediction in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels; however, since there is also a strong similarity between adjacent frames in a video image, the use of inter-frame prediction in video coding and decoding technology can eliminate the temporal redundancy between adjacent frames, thereby improving the coding and decoding efficiency.

[0026] Videos are composed of multiple images. In order to make the video look smooth, each second of video contains dozens or even hundreds of frames, such as 24 frames per second, 30 frames per second, 50 frames per second, 60 frames per second or 120 frames per second. Therefore, there is a lot of temporal redundancy in the video, or in other words, there is a lot of temporal correlation in the video. Inter-frame prediction uses this temporal correlation to improve compression efficiency. Inter-frame prediction often uses "motion" to exploit temporal correlation. A very simple "motion" model is that an object is at a certain position on the image corresponding to a certain moment, and after a certain period of time, it moves to another position on the image corresponding to the current moment. This is the most basic and commonly used translation motion in video coding and decoding. Inter-frame prediction uses motion information to represent "motion". Basic motion information includes the information of the reference frame and the information of the MV; the reference frame can also be understood as the reference picture. The codec determines the reference frame / reference picture based on the information of the reference frame / reference picture, and determines the coordinates of the reference block based on the information of the MV and the coordinates of the current block. The reference block is determined by the coordinates of the reference block in the reference picture. Using a determined reference block as a prediction block is the most basic prediction method for inter-frame prediction.

[0027] Figure 1FIG. 1 is a flow chart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0028] S101, synchronously extracting a first image frame and a first audio frame from a video.

[0029] It should be noted that the execution subject of the audio-video synchronization detection method of the embodiment of the present disclosure may be a hardware device with data audio-video synchronization detection capability and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc.

[0030] The video may be any video that requires audio and video synchronization detection.

[0031] For example, a video of any length read from the storage space can be used as the video that needs to be detected for audio and video synchronization; for another example, the obtained video can be intercepted according to a preset length, and the intercepted part of the video can be used as the video that needs to be detected for audio and video synchronization.

[0032] A video generally includes image frames and audio frames, and the image frames and audio frames are aligned on the time axis.

[0033] It should be noted that the present disclosure does not limit the specific method for synchronously extracting the first image frame and the first audio frame from the video, and the specific method can be selected according to actual conditions.

[0034] Optionally, a preset time interval may be obtained, and the first image frames {L1, L2, ..., L n}, and extract the first audio frame {A1, A2, ...A n}.

[0035] For example, a video frame extraction tool may be used to extract image frames from a video at time intervals, and an audio extraction tool may be used to extract audio frames from a video at time intervals.

[0036] In the embodiment of the present disclosure, after the first image frame is acquired, the first image frame {L1, L2, ...L n}, stored in the image list {l}, after obtaining the first audio frame, the first audio frame {A1, A2, ...A n} is stored in the audio list {A}.

[0037] S102: Determine at least one second image frame having the same subtitles from the first image frame.

[0038] Optionally, the subtitle content of the first image frame may be extracted, and a second image frame having the same subtitle may be determined.

[0039] For example, for the first image frame L1, by extracting the subtitle content of the first image frame L1, it is determined that the subtitle content of the first image frame L1 is "How is the weather today?" The remaining first image frames L2, ...L n , extract the remaining first image frames L2, ...L n The subtitle content of the first image frame L2 and the first image frame L3 is determined from the subtitle content to be "How is the weather today?" Then the second image frames with the same subtitles are determined from the first image frame L1 to be L2 and L3.

[0040] S103: Determine a second audio frame synchronized with the second image frame from the first audio frame.

[0041] In the embodiment of the present disclosure, the timestamp of the second image frame and the timestamp of the first audio frame can be determined, and the second audio frame synchronized with the second image frame can be determined from the first audio frame based on the timestamp of the second image frame and the timestamp of the first audio frame.

[0042] It should be noted that the present disclosure does not limit the specific method of determining the second audio frame synchronized with the second image frame from the first audio frame according to the timestamp of the second image frame and the timestamp of the first audio frame, and it can be selected according to actual conditions.

[0043] Optionally, the timestamp of the second image frame and the timestamp of the first audio frame may be determined, and a second audio frame synchronized with the second image frame may be determined from the first audio frame according to the timestamp of the second image frame and the timestamp of the first audio frame.

[0044] Optionally, the start and end timestamps of the same subtitle may be determined according to the timestamp of the second image frame, and a second audio frame synchronized with the second image frame may be determined from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame.

[0045] S104: Perform audio-visual synchronization detection on the video according to the second image frame and the second audio frame.

[0046] In an embodiment of the present disclosure, the second audio frames can be combined in chronological order to obtain an audio frame sequence, and the audio frame sequence can be input into a language recognition model to obtain a speech recognition result of the second audio frame. Based on the same subtitles and speech recognition results, the video can be subjected to audio and video synchronization detection.

[0047] Optionally, the similarity between the same subtitles and speech recognition results may be obtained, and whether the audio and video of the video are synchronized may be determined based on the similarity and a set similarity (similarity threshold).

[0048] For example, in response to the similarity being greater than a set similarity, it is determined that the video, audio, and picture are synchronized, and in response to the similarity being less than or equal to the set similarity, it is determined that the video, audio, and picture are not synchronized.

[0049] The audio-visual synchronization detection method proposed in the present disclosure synchronously extracts a first image frame and a first audio frame from a video; determines at least one second image frame having the same subtitles from the first image frame; determines a second audio frame synchronized with the second image frame from the first audio frame; and performs audio-visual synchronization detection on the video according to the second image frame and the second audio frame. The present disclosure improves the accuracy and reliability of the audio-visual synchronization detection result of the video by determining at least one second image frame having the same subtitles from the first image frame, and performing audio-visual synchronization detection on the video according to the second image frame and the second audio frame synchronized with the second image frame.

[0050] Figure 2 It is a flowchart of the audio-video synchronization detection method according to the second embodiment of the present disclosure.

[0051] like Figure 2 As shown, in Figure 2 Based on the illustrated embodiment, the audio and video synchronization detection method of the embodiment of the present disclosure may specifically include the following steps:

[0052] The step S101 of “synchronously extracting image frames and audio frames from the video” in the above embodiment may specifically include the following steps S201 and S202 .

[0053] S201, extracting a first image frame from a video at a time interval and storing it in an image list.

[0054] It should be noted that the present disclosure does not limit the setting of the time interval, and it can be set according to actual conditions. For example, the time interval can be set to 10 milliseconds; for another example, the time interval can be set to 30 milliseconds.

[0055] It should be noted that the present disclosure does not limit the specific method of extracting image frames from a video according to time intervals, and the method can be selected according to actual conditions.

[0056] Optionally, a video frame extraction tool may be used to extract image frames from the video at time intervals.

[0057] For example, for video A, the time interval is 30 milliseconds, and a frame is extracted from video A every 30 milliseconds by a video frame extraction tool to obtain image frames {L1, L2, ...L n}.

[0058] In the embodiment of the present disclosure, after the first image frame is acquired, the first image frame may be stored in the image list {L}.

[0059] S202, extracting the first audio frame from the video according to the time interval, and storing it in the audio list.

[0060] It should be noted that, in order to ensure the synchronization of the image frame and the audio frame on the time axis, the time interval for extracting the audio frame is the same as the time interval for extracting the audio frame.

[0061] It should be noted that the present disclosure does not limit the specific method of extracting audio frames from a video according to time intervals, and the method can be selected according to actual conditions.

[0062] Optionally, an audio extraction tool may be used to extract audio frames from the video at time intervals.

[0063] For example, for video A, the time interval is 30 milliseconds. The audio extraction tool extracts one frame from video A every 30 milliseconds to obtain audio frames {A1, A2, ...A n}.

[0064] In the embodiment of the present disclosure, after the first audio frame is acquired, the first audio frame may be stored in the audio list {A}.

[0065] In the above embodiment, step S102 of “determining at least one second image frame having the same subtitle from the first image frame” may specifically include the following steps S203 and S204.

[0066] S203, extracting the subtitle content of the first image frame in the image list, and determining a second image frame having the same subtitle.

[0067] It should be noted that the present disclosure does not limit the specific method for extracting the subtitle content of the first image frame in the image list, and it can be selected according to actual conditions.

[0068] Optionally, a subtitle area in the first image frame may be determined, and subtitle content in the subtitle area in the first image frame may be extracted by using an optical character recognition (OCR) technology.

[0069] Optionally, a pre-trained subtitle text recognition model may be obtained, and the first image frame may be input into the subtitle text recognition model to extract subtitle content in the subtitle area in the first image frame.

[0070] In the embodiment of the present disclosure, after the subtitle content of the first image frame is acquired, the content in the first image frame may be compared to acquire the similarity between the subtitle contents, so as to determine the second image frame having the same subtitle.

[0071] S204, sorting the second image frames according to the timestamps of the second image frames to obtain an image frame sequence.

[0072] In the above embodiment, step S103 of “determining a second audio frame synchronized with the second image frame from the first audio frame” may specifically include the following steps S205 and S206 .

[0073] S205: Determine a timestamp of the second image frame and a timestamp of the first audio frame.

[0074] It should be noted that after the second image frame is acquired, the timestamp of the first image frame may be queried to determine the timestamp corresponding to the second image frame.

[0075] For example, for video A, one frame is extracted every 30 milliseconds, and the first image frames are image frames L1, L2, L3, L4, L5, and L6. If the timestamp of image frame L1 is 00:15:04, the timestamp of image frame L2 is 00:15:34, the timestamp of image frame L3 is 00:16:04, the timestamp of image frame L4 is 00:16:34, the timestamp of image frame L5 is 00:17:04, and the timestamp of image frame L6 is 00:17:34. If the second image frames with the same subtitles are L2 and A3, the timestamps of the second image frames L2 and L3 are 00:15:34 and 00:16:04 respectively.

[0076] It should be noted that after the first audio frame is acquired, the timestamp of the first audio frame may be determined.

[0077] For example, for video A, one frame is extracted every 30 milliseconds, and the first audio frames are audio frames A1, A2, A3, A4, A5, and A6. If the timestamp of audio frame A1 is 00:15:04, the timestamp of audio frame A2 is 00:15:34, the timestamp of audio frame A3 is 00:16:04, the timestamp of audio frame A4 is 00:16:34, the timestamp of audio frame A5 is 00:17:04, and the timestamp of audio frame A6 is 00:17:34.

[0078] S206: Determine, from the first audio frame, a second audio frame synchronized with the second image frame according to the timestamp of the second image frame and the timestamp of the first audio frame.

[0079] Optionally, the timestamp of the second image frame and the timestamp of the first audio frame may be matched to determine a second audio frame synchronized with the second image frame from the first audio frame.

[0080] For example, if the timestamps of the second image frames L2 and L3 are 00:15:34 and 00:16:04, by matching the timestamps of the second image frames with the timestamps of the first audio frames, the second audio frames synchronized with the second image frames are determined to be A2 and A3 from the first audio frames.

[0081] In the above embodiment, step S103 of “determining a second audio frame synchronized with the second image frame from the first audio frame” may specifically include the following steps S207 and S208 .

[0082] S207: Determine the start and end timestamps of the same subtitle according to the timestamp of the second image frame.

[0083] In an embodiment of the present disclosure, the first second image frame and the last second image frame in an image frame sequence can be determined, the timestamp of the first second image frame can be determined as the start timestamp of the same subtitles, and the timestamp of the last second image frame can be determined as the end timestamp of the same subtitles.

[0084] S208: Determine, from the first audio frame, a second audio frame synchronized with the second image frame according to the start and end timestamps and the timestamp of the first audio frame.

[0085] In the disclosed embodiment, the start timestamp and end timestamp of the same subtitle can be determined according to the start and end timestamps, and an audio frame with the same timestamp as the start timestamp can be determined from the audio list corresponding to the first audio frame as the first second audio frame, and an audio frame with the same timestamp as the end timestamp can be determined from the audio list corresponding to the first audio frame as the last second audio frame, and the audio frames from the first second audio frame to the last second audio frame in the audio list can be determined as the second audio frames.

[0086] In the above embodiment, step S104 of “performing audio and video synchronization detection on the video according to the second image frame and the second audio frame” may specifically include the following steps S209 and S2010.

[0087] S209: Obtain a language recognition result of the second audio frame.

[0088] In the embodiment of the present disclosure, the second audio frames are combined in time order to obtain an audio frame sequence, and the audio frame sequence is input into a language recognition model to obtain a speech recognition result.

[0089] S2010: Perform audio and video synchronization detection on the video based on the same subtitle and speech recognition results.

[0090] In the disclosed embodiment, the similarity between the same subtitles and speech recognition results can be obtained, and the audio-visual synchronization detection of the video can be performed based on the similarity and the set similarity (similarity threshold).

[0091] Optionally, the same subtitle and speech recognition results can be input into a pre-trained subtitle and speech similarity detection model, which performs cross-modal similarity calculation on the same subtitle and speech recognition results to obtain the similarity of the same subtitle and speech recognition results.

[0092] It should be noted that the present disclosure does not limit the setting of the preset similarity. For example, the similarity can be set to 100%; for another example, the similarity can be set to 98%.

[0093] Optionally, in response to the similarity between the same subtitle and speech recognition results being greater than a set similarity, the video audio and video synchronization is determined.

[0094] For example, for video A, if the similarity is set to 98%, and the similarity of the same subtitle and speech recognition results is 100%, it is determined that the audio and video of video A are synchronized.

[0095] Optionally, in response to the similarity between the same subtitle and speech recognition results being less than or equal to a set similarity, it is determined that the audio and picture of the video are not synchronized.

[0096] For example, for video A, if the similarity is set to 98%, and the similarity of the same subtitle and speech recognition results is 90%, it is determined that the audio and video of video A are not synchronized.

[0097] In summary, the audio-visual synchronization detection method proposed in the present disclosure extracts a first image frame from a video at a time interval and stores it in an image list, extracts a first image frame from a video at a time interval and stores it in an image list, extracts the subtitle content of the first image frame in the image list, and determines a second image frame with the same subtitle, sorts the second image frame according to the timestamp of the second image frame to obtain an image frame sequence, determines a second audio frame synchronized with the second image frame from the first audio frame, obtains a language recognition result of the second audio frame, and performs audio-visual synchronization detection on the video according to the same subtitle and speech recognition result. Therefore, the present disclosure determines at least one second image frame with the same subtitle from the first image frame, and performs audio-visual synchronization detection on the video according to the second image frame and the second audio frame synchronized with the second image frame, and determines whether the video has audio-visual synchronization according to the similarity and set similarity of the same subtitle and speech recognition results, thereby avoiding the problem of missed detection in the process of audio-visual synchronization detection, improving the accuracy and reliability of obtaining the audio-visual synchronization detection result of the video, improving the user's experience of watching the video, and having a wider range of applications.

[0098] The specific process of the audio and video synchronization detection method proposed in the embodiment of the present disclosure is explained below.

[0099] For example, Figure 3 As shown, the specific process of the audio-video synchronization detection method may include the following steps:

[0100] S301, obtaining a video segment with a video length of T, extracting image frames from the video at time intervals, storing them in an image list, and extracting audio frames from the video at time intervals, storing them in an audio list.

[0101] S302: Determine, from the first audio frame, a second audio frame synchronized with the second image frame according to the timestamp of the second image frame and the timestamp of the first audio frame.

[0102] S303, combining the second audio frames in time order to obtain an audio frame sequence, and inputting the audio frame sequence into a language recognition model to obtain a speech recognition result.

[0103] S304: Input the same subtitle and speech recognition results into a subtitle and speech similarity detection model to obtain the similarity between the same subtitle and speech recognition results.

[0104] S305, performing audio and video synchronization detection on the video according to the similarity between the same subtitles and the speech recognition results, that is, determining whether the audio and video of the video are synchronized.

[0105] S306: In response to the similarity being greater than the set similarity, determining that the video audio and video are synchronized.

[0106] S307 , in response to the similarity being less than or equal to the set similarity, determining that the video audio and picture are not synchronized.

[0107] In summary, the audio-visual synchronization detection method proposed in the present invention determines at least one second image frame with the same subtitles from the first image frame, obtains the speech recognition result of the second audio frame synchronized with the second image frame, and inputs the same subtitles and speech recognition results into the subtitle speech similarity detection model to obtain the similarity of the same subtitles and speech recognition results. According to the similarity of the same subtitles and speech recognition results, it is judged whether the video has audio-visual synchronization, thereby avoiding the problem of missed detection in the process of audio-visual synchronization detection, preventing the problem of secondary detection due to missed detection in the process of audio-visual synchronization detection, reducing the cost in the process of audio-visual synchronization detection, improving the efficiency of audio-visual synchronization detection, and at the same time ensuring the accuracy and reliability of the audio-visual synchronization detection result of the video, thereby improving the user's experience of watching the video, and having a wide range of applications.

[0108] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] According to an embodiment of the present disclosure, the present disclosure further provides an audio-video synchronization detection device, which is used to implement the above-mentioned audio-video synchronization detection method.

[0110] Figure 4 It is a block diagram of an audio-video synchronization detection device according to an embodiment of the present disclosure.

[0111] like Figure 4 As shown, the audio-video synchronization detection device 400 includes: an extraction module 401 , a first determination module 402 , a second determination module 403 and a detection module 404 .

[0112] An extraction module 401 is used to synchronously extract a first image frame and a first audio frame from a video;

[0113] A first determining module 402, configured to determine at least one second image frame having the same subtitles from the first image frame;

[0114] A second determining module 403, configured to determine, from the first audio frame, a second audio frame synchronized with the second image frame;

[0115] The detection module 404 is configured to perform audio-visual synchronization detection on the video according to the second image frame and the second audio frame.

[0116] In one embodiment of the present disclosure, the detection module 404 is used to: obtain a language recognition result of the second audio frame; and perform audio-visual synchronization detection on the video according to the same subtitles and the speech recognition result.

[0117] In one embodiment of the present disclosure, the second determination module 403 is used to: determine the timestamp of the second image frame and the timestamp of the first audio frame; and determine a second audio frame synchronized with the second image frame from the first audio frame based on the timestamp of the second image frame and the timestamp of the first audio frame.

[0118] In one embodiment of the present disclosure, the second determination module 403 is used to: determine the timestamp of the second image frame and the timestamp of the first audio frame; and determine a second audio frame synchronized with the second image frame from the first audio frame based on the timestamp of the second image frame and the timestamp of the first audio frame.

[0119] In one embodiment of the present disclosure, the extraction module 401 is used to: extract the first image frame from the video at a time interval and store it in an image list; extract the first audio frame from the video at a time interval and store it in an audio list.

[0120] In one embodiment of the present disclosure, the first determination module 402 is used to: extract the subtitle content of the first image frame in the image list, and determine the second image frame with the same subtitle; sort the second image frames according to the timestamps of the second image frames to obtain an image frame sequence.

[0121] In one embodiment of the present disclosure, the first determination module 402 is used to: extract the subtitle content of the first image frame in the image list, and determine the second image frame with the same subtitle; sort the second image frames according to the timestamps of the second image frames to obtain an image frame sequence.

[0122] In one embodiment of the present disclosure, the second determination module 403 is used to: determine the start timestamp and the end timestamp of the same subtitle according to the start and end timestamps; determine, from the audio list corresponding to the first audio frame, an audio frame with the same timestamp as the start timestamp as the first second audio frame; determine, from the audio list corresponding to the first audio frame, an audio frame with the same timestamp as the end timestamp as the last second audio frame; determine the audio frames from the first second audio frame to the last second audio frame in the audio list as the second audio frame.

[0123] In one embodiment of the present disclosure, the detection module 404 is used to: combine the second audio frames in time order to obtain an audio frame sequence; input the audio frame sequence into a language recognition model to obtain the speech recognition result.

[0124] In one embodiment of the present disclosure, the detection module 404 is used to: obtain the similarity between the same subtitle and the speech recognition result; in response to the similarity being greater than a set similarity, determine that the video audio and picture are synchronized; in response to the similarity being less than or equal to the set similarity, determine that the video audio and picture are not synchronized.

[0125] The audio-visual synchronization detection device proposed in the present disclosure synchronously extracts a first image frame and a first audio frame from a video; determines at least one second image frame with the same subtitles from the first image frame; determines a second audio frame synchronized with the second image frame from the first audio frame; and performs audio-visual synchronization detection on the video according to the second image frame and the second audio frame. The present disclosure improves the accuracy and reliability of the audio-visual synchronization detection result of the video by determining at least one second image frame with the same subtitles from the first image frame, and performing audio-visual synchronization detection on the video according to the second image frame and the second audio frame synchronized with the second image frame.

[0126] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a terminal and a computer program product.

[0127] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0129] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the audio-visual synchronization detection method. For example, in some embodiments, the audio-visual synchronization detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the audio-visual synchronization detection method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the audio-visual synchronization detection method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides a terminal, such as Figure 6 As shown, the terminal 600 includes the above-mentioned electronic device 500.

[0138] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the audio-visual synchronization detection method described in the above embodiment of the present disclosure are implemented.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting synchronization between audio and video, wherein: The method comprises: Synchronously extracting a first image frame and a first audio frame from the video; Determining at least one second image frame having the same subtitles from the first image frame; Determine a second audio frame synchronized with the second image frame from the first audio frame; Perform audio and video synchronization detection on the video according to the second image frame and the second audio frame.

2. The method according to claim 1, wherein: The performing audio-visual synchronization detection on the video according to the second image frame and the second audio frame includes: Obtaining a language recognition result of the second audio frame; Perform audio and video synchronization detection on the video according to the same subtitles and the speech recognition result.

3. The method according to claim 1, wherein: The determining, from the first audio frame, a second audio frame synchronized with the second image frame comprises: determining a timestamp of the second image frame and a timestamp of the first audio frame; A second audio frame synchronized with the second image frame is determined from the first audio frame according to the timestamp of the second image frame and the timestamp of the first audio frame.

4. The method according to claim 1, wherein: The determining, from the first audio frame, a second audio frame synchronized with the second image frame comprises: Determining the start and end timestamps of the same subtitle according to the timestamp of the second image frame; The second audio frame synchronized with the second image frame is determined from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame.

5. The method according to claim 4, wherein: The step of synchronously extracting the first image frame and the first audio frame from the video includes: Extracting a first image frame from the video at a time interval and storing it in an image list; The first audio frame is extracted from the video at a time interval and stored in an audio list.

6. The method according to claim 5, wherein: The determining at least one second image frame having the same subtitle from the first image frame comprises: Extracting the subtitle content of the first image frame in the image list, and determining the second image frame having the same subtitle; The second image frames are sorted according to the timestamps of the second image frames to obtain an image frame sequence.

7. The method according to claim 6, wherein: The step of determining the start and end timestamps of the same subtitle according to the timestamp of the second image frame includes: Determine a first second image frame and a last second image frame in the image frame sequence; Determine the timestamp of the first second image frame as the start timestamp of the same subtitle; The timestamp of the last second image frame is determined as the termination timestamp of the same subtitle.

8. The method according to claim 6 or 7, wherein: The determining, from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame, the second audio frame synchronized with the second image frame comprises: Determine the start timestamp and the end timestamp of the same subtitle according to the start and end timestamps; Determine, from the audio list corresponding to the first audio frame, an audio frame whose timestamp is the same as the start timestamp as the first second audio frame; Determine, from the audio list corresponding to the first audio frame, an audio frame whose timestamp is the same as the end timestamp as the last second audio frame; The audio frames from the first second audio frame to the last second audio frame in the audio list are determined as the second audio frames.

9. The method according to any one of claims 2 to 7, wherein: The obtaining of the language recognition result of the second audio frame includes: Combining the second audio frames in time sequence to obtain an audio frame sequence; The audio frame sequence is input into a language recognition model to obtain the speech recognition result.

10. The method according to claim 9, wherein: The step of performing audio and video synchronization detection on the video according to the same subtitles and the speech recognition result includes: Obtaining the similarity between the same subtitle and the speech recognition result; In response to the similarity being greater than a set similarity, determining that the audio and video of the video are synchronized; In response to the similarity being less than or equal to the set similarity, it is determined that the audio and picture of the video are not synchronized.

11. A device for detecting synchronization between audio and video, wherein: The device comprises: An extraction module, used for synchronously extracting a first image frame and a first audio frame from a video; A first determining module, configured to determine at least one second image frame having the same subtitles from the first image frame; A second determining module, configured to determine, from the first audio frame, a second audio frame synchronized with the second image frame; A detection module is used to perform audio and video synchronization detection on the video according to the second image frame and the second audio frame.

12. The device according to claim 11, wherein The detection module is used for: Obtaining a language recognition result of the second audio frame; Perform audio and video synchronization detection on the video according to the same subtitles and the speech recognition result.

13. The device according to claim 11, wherein: The second determining module is used to: determining a timestamp of the second image frame and a timestamp of the first audio frame; A second audio frame synchronized with the second image frame is determined from the first audio frame according to the timestamp of the second image frame and the timestamp of the first audio frame.

14. The device according to claim 11, wherein: The second determining module is used to: Determining the start and end timestamps of the same subtitle according to the timestamp of the second image frame; The second audio frame synchronized with the second image frame is determined from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame.

15. The device according to claim 14, wherein: The extraction module is used to: Extracting a first image frame from the video at a time interval and storing it in an image list; The first audio frame is extracted from the video at a time interval and stored in an audio list.

16. The device according to claim 15, wherein: The first determining module is used to: Extracting the subtitle content of the first image frame in the image list, and determining the second image frame having the same subtitle; The second image frames are sorted according to the timestamps of the second image frames to obtain an image frame sequence.

17. The device according to claim 16, wherein: The first determining module is used to: Determine a first second image frame and a last second image frame in the image frame sequence; Determine the timestamp of the first second image frame as the start timestamp of the same subtitle; The timestamp of the last second image frame is determined as the termination timestamp of the same subtitle.

18. The device according to claim 16 or 17, wherein: The second determining module is used to: Determine the start timestamp and the end timestamp of the same subtitle according to the start and end timestamps; Determine, from the audio list corresponding to the first audio frame, an audio frame whose timestamp is the same as the start timestamp as the first second audio frame; Determine, from the audio list corresponding to the first audio frame, an audio frame whose timestamp is the same as the end timestamp as the last second audio frame; The audio frames from the first second audio frame to the last second audio frame in the audio list are determined as the second audio frames.

19. The device according to any one of claims 12 to 17, wherein: The detection module is used for: Combining the second audio frames in time sequence to obtain an audio frame sequence; The audio frame sequence is input into a language recognition model to obtain the speech recognition result.

20. The device according to claim 19, wherein The detection module is used for: Obtaining the similarity between the same subtitle and the speech recognition result; In response to the similarity being greater than a set similarity, determining that the audio and video of the video are synchronized; In response to the similarity being less than or equal to the set similarity, it is determined that the audio and picture of the video are not synchronized.

21. An electronic device, characterized in that: including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 10.

22. A terminal, characterized in that: Comprising the electronic device as claimed in claim 21.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.