Sound and picture synchronization detection method and device, electronic equipment and terminal
By identifying whether there are subtitles in the video frame, determining the audio and video synchronization detection algorithm, and performing audio and video synchronization detection, the problem of low accuracy of audio and video synchronization detection in the existing technology is solved, and the accuracy of the detection results and user experience are improved.
Patent Information
- Application Number
- CN202411266399.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-26
AI Technical Summary
The accuracy of the intermediate audio and video synchronization detection in the prior art is low, affecting the user's viewing experience.
Extract image frames and audio frames in the video, identify the type according to whether the frame has subtitles, determine the corresponding audio and picture synchronization detection algorithm, and use subtitle voice synchronization detection or lip voice synchronization detection methods to perform audio and picture synchronization detection.
It improves the accuracy and reliability of audio and video synchronization detection, enhances the flexibility and applicability of video and audio and video synchronization detection results, and improves the user's viewing experience.
Smart Images

Figure CN119031126A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, specifically to the technical fields of computer vision and artificial intelligence, and in particular to a method, device, electronic device and terminal for detecting synchronization between audio and video. Background Art
[0002] In the process of video editing, encoding, and playback, the subtitles and sounds in the video may not match, affecting the user's viewing experience. In related technologies, reference methods are often used to detect audio and video synchronization, such as adding marker information or comparing with reference videos, or using non-reference methods to detect audio and video synchronization. However, the above methods often obtain audio and video detection results with low accuracy. Therefore, how to improve the accuracy and reliability of audio and video synchronization detection has become an urgent problem to be solved. Summary of the invention
[0003] The present disclosure provides a method, device, electronic device, terminal and computer program product for detecting audio and video synchronization.
[0004] According to a first aspect of the present disclosure, a method for detecting audio and video synchronization is proposed, including: synchronously extracting a first image frame and a first audio frame from a video; performing type identification on the first image frame to obtain a target type of the first image frame; determining a target audio and video synchronization detection algorithm according to the target type; and performing audio and video synchronization detection on the first image frame and the first audio frame based on the target audio and video synchronization detection algorithm.
[0005] According to a second aspect of the present disclosure, a device for detecting audio and video synchronization is proposed, comprising: an extraction module, used to synchronously extract a first image frame and a first audio frame from a video; an identification module, used to perform type identification on the first image frame to obtain a target type of the first image frame; a determination module, used to determine a target audio and video synchronization detection algorithm according to the target type; and a detection module, used to perform audio and video synchronization detection on the first image frame and the first audio frame based on the target audio and video synchronization detection algorithm.
[0006] According to the third aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio and video synchronization detection method proposed in the first aspect above.
[0007] According to a fourth aspect of an embodiment of the present disclosure, a terminal is provided, comprising the electronic device proposed in the embodiment of the third aspect.
[0008] According to a fifth aspect of the present disclosure, a computer program product is proposed, including a computer program, wherein when the computer program is executed by a processor, the audio and video synchronization detection method proposed in the first aspect is implemented.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0011] Figure 1 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0012] Figure 2 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0013] Figure 3 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0014] Figure 4 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0015] Figure 5 It is a structural schematic diagram of an audio and video synchronization detection device according to an embodiment of the present disclosure;
[0016] Figure 6 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure;
[0017] Figure 7 is a schematic block diagram of a terminal according to an embodiment of the present disclosure; DETAILED DESCRIPTION
[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0019] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.
[0020] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.
[0021] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.
[0022] In the embodiment of the present disclosure, a video reading tool can be used to read the video to be subjected to audio and video synchronization detection, and the video can be decoded by the video reading tool to obtain the video, and audio and video detection can be performed on the video to obtain the audio and video synchronization detection result of the video.
[0023] The video decoding formats include but are not limited to: (1) H.264 / AVC (Advanced Video Coding), the video decoder is libx264; (2) H.265 / HEVC (High Efficiency Video Coding), the video decoder is libx265; (3) VP8 (Video Coding Format by Google), the video decoder is libvpx; (4) VP9 (Successor to VP8), the video decoder is libvpx-vp9; (5) AV1 (AOMedia Video1), the video decoder is libaom-av1; (6) MPEG-2 (Moving Picture Experts Group Phase2), the video decoder is mpeg2video; (7) MPEG-4 (Moving Picture Experts Group Phase4), the video decoder is mpeg4; (8) Theora (Free and Open Video Codec), the video decoder is libtheora; ProRes (AppleProRes), the video decoder is prores; (10) DNxHD (Digital Nonlinear Extensible High Definition), the video decoder is dnxhd; (11) WMV (Windows Media Video), the video decoder is wmv2; (12) FLV1 (Flash Video), the video decoder is flv; (13) MJPEG (Motion JPEG), the video decoder is mjpeg
[0024] The audio decoding formats include but are not limited to: (1) AAC (Advanced Audio Coding), the audio decoder is aac; (2) MP3 (MPEG-1 Audio Layer 3), the audio decoder is libmp3lame; (3) Vorbis (Ogg Vorbis), the audio decoder is libvorbis; (4) Opus (Audio Codec), the audio decoder is libopus; (5) FLAC (Free Lossless Audio Codec), the audio decoder is flac; (6) ALAC (Apple Lossless Audio Code), the audio decoder is alac; (7) AC3 (Audio Codec 3), the audio decoder is ac3; (8) WMA (Windows Media Audio), the audio decoder is wmav2; (9) WAV (Waveform Audio File Format), the audio decoder is the built-in WAV decoder; (10) AMR (Adaptive Audio File Format), the audio decoder is the built-in WAV decoder; Multi-Rate) encoder: the audio decoder is libopencore_amrnb (NB) and libopencore_amrwb (WB); (11) PCM (Pulse Code Modulation), the audio decoder is pcm_s16le, pcm_s24le, etc.
[0025] In the video encoding and decoding process, the basic process is as follows: in the encoder, a frame of image is divided into blocks, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, the original block of the current block minus the prediction block to obtain the residual block, the residual block is transformed and quantized to obtain the quantization coefficient matrix, and the quantization coefficient matrix is entropy encoded and output to the bitstream. In the decoder, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, on the other hand, the bitstream is decoded to obtain the quantization coefficient matrix, the quantization coefficient matrix is inversely quantized and inversely transformed to obtain the residual block, and the prediction block and the residual block are added to obtain the reconstructed block. The reconstructed blocks constitute the reconstructed image, and the reconstructed image is loop-filtered based on the image or block to obtain the decoded image. The encoder also needs similar operations as the decoder to obtain the decoded image. The decoded image can be used as a reference frame for inter-frame prediction for subsequent frames. The block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information determined by the encoder need to be output to the bitstream if necessary; then the decoder determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information as the encoder by parsing and analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder. The decoded image obtained by the encoder is also usually called a reconstructed image. The current block can be divided into prediction units during prediction, and the current block can be divided into transformation units during transformation. The division of prediction units and transformation units can be different. The above is the basic process of the video encoder and decoder under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The embodiment of the present application is applicable to the basic process under the block-based hybrid coding framework, but is not limited to the framework and process.
[0026] At present, the common video coding and decoding standards all adopt the block-based hybrid coding framework. Each frame in the video image is divided into square LCUs of the same size (such as 128×128, 64×64, etc.), and each LCU can also be divided into rectangular CUs according to rules; and CUs may also be divided into smaller PUs, TUs, etc. Specifically, the coding framework may include steps such as prediction, transform, quantization, entropy coding, and in-loop filtering. Among them, prediction can be divided into intra-frame prediction and inter-frame prediction, and inter-frame prediction can include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the use of intra-frame prediction in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels; however, since there is also a strong similarity between adjacent frames in a video image, the use of inter-frame prediction in video coding and decoding technology can eliminate the temporal redundancy between adjacent frames, thereby improving the coding and decoding efficiency.
[0027] Videos are composed of multiple images. In order to make the video look smooth, each second of video contains dozens or even hundreds of frames, such as 24 frames per second, 30 frames per second, 50 frames per second, 60 frames per second or 120 frames per second. Therefore, there is a lot of temporal redundancy in the video, or in other words, there is a lot of temporal correlation in the video. Inter-frame prediction uses this temporal correlation to improve compression efficiency. Inter-frame prediction often uses "motion" to exploit temporal correlation. A very simple "motion" model is that an object is at a certain position on the image corresponding to a certain moment, and after a certain period of time, it moves to another position on the image corresponding to the current moment. This is the most basic and commonly used translation motion in video coding and decoding. Inter-frame prediction uses motion information to represent "motion". Basic motion information includes the information of the reference frame and the information of the MV; the reference frame can also be understood as the reference picture. The codec determines the reference frame / reference picture based on the information of the reference frame / reference picture, and determines the coordinates of the reference block based on the information of the MV and the coordinates of the current block. The reference block is determined by the coordinates of the reference block in the reference picture. Using a determined reference block as a prediction block is the most basic prediction method for inter-frame prediction.
[0028] Figure 1FIG. 1 is a flow chart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:
[0029] S101, synchronously extracting a first image frame and a first audio frame from a video.
[0030] It should be noted that the execution subject of the audio-video synchronization detection method of the embodiment of the present disclosure may be a hardware device with data audio-video synchronization detection capability and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc.
[0031] The video may be any video that requires audio and video synchronization detection.
[0032] For example, a video of any length read from the storage space can be used as the video that needs to be tested for audio and video synchronization; for another example, the obtained video can be intercepted according to a preset length, and the intercepted part of the video can be used as the video that needs to be tested for audio and video synchronization.
[0033] A video generally includes image frames and audio frames, and the image frames and audio frames are aligned on the time axis.
[0034] In the embodiment of the present disclosure, the first image frame may be extracted from the video at a time interval and stored in an image list, and the first audio frame may be extracted from the video at a time interval and stored in an audio list.
[0035] It should be noted that the present disclosure does not limit the specific method of extracting the first image frame from the video according to the time interval, and it can be selected according to the actual situation.
[0036] Optionally, a preset time interval may be obtained, and a video frame extraction tool may be used to extract the first image frame {L1, L2, ..., L1, L2 ... n}, use the audio extraction tool to extract the first audio frame {A1, A2, ...A n}.
[0037] It should be noted that the present disclosure does not limit the setting of the time interval, and it can be set according to actual conditions. For example, the time interval can be set to 10 milliseconds; for another example, the time interval can be set to 30 milliseconds.
[0038] In the embodiment of the present disclosure, after the first image frame is acquired, the first image frame {L1, L2, ...L n}, stored in the image list {L}, after obtaining the first audio frame, the first audio frame {A1, A2, ...A n} is stored in the audio list {A}.
[0039] S102: Perform type identification on the first image frame to obtain a target type of the first image frame.
[0040] In the embodiment of the present disclosure, content recognition may be performed on the first image frame to determine whether there are subtitles in the first image frame, and the target type of the first image frame may be acquired according to the subtitle recognition result of the first image frame.
[0041] Optionally, in response to the presence of subtitles in the first image frame, the target type of the first image frame is determined to be the first type.
[0042] Optionally, in response to the first image frame having no subtitles, the target type of the first image frame is determined to be the second type.
[0043] For example, for the first image frames L1 and L2, if the first image frame L1 has subtitles, the target type of the first image frame L1 is determined to be the first type; if the first image frame L2 has no subtitles, the target type of the first image frame L2 is determined to be the second type.
[0044] S103, determining a target audio and video synchronization detection algorithm according to the target type.
[0045] In the embodiment of the present disclosure, after the target type of the first image frame is acquired, a target audio and video synchronization detection algorithm may be determined according to the target type.
[0046] In an embodiment of the present disclosure, in response to the target type being the first type, the subtitle speech synchronization detection algorithm is determined to be the target audio-visual synchronization detection algorithm, and in response to the target type being the second type, the lip synchronization detection algorithm is determined to be the target audio-visual synchronization detection algorithm.
[0047] For example, in response to the target type of the first image frame L1 being the first type, the subtitle speech synchronization detection algorithm is determined as the target audio and video synchronization detection algorithm, and in response to the target type of the first image frame L2 being the second type, the lip synchronization detection algorithm is determined as the target audio and video synchronization detection algorithm.
[0048] S104: Perform audio-visual synchronization detection on the first image frame and the first audio frame based on a target audio-visual synchronization detection algorithm.
[0049] Optionally, if the target audio-visual synchronization detection algorithm is a subtitle-speech synchronization detection algorithm, at least one first image frame with the same subtitles is determined from the first image frame, a second audio frame synchronized with the second image frame is determined from the first audio frame, and a language recognition result of the second audio frame is obtained. Based on the same subtitle and speech recognition results, audio-visual synchronization detection is performed on the first image frame and the first audio frame.
[0050] Optionally, if the target audio and video synchronization detection algorithm is a lip synchronization detection algorithm, face detection and tracking are performed on the first image frame to obtain one or more face identifiers, and the first image frame is grouped according to the face identifiers to obtain multiple image lists, and based on the image frames in the image list, the mouth area picture corresponding to the image list is determined, and audio and video synchronization detection is performed on the mouth area picture corresponding to the image list and the first audio frame.
[0051] The audio-visual synchronization detection method proposed in the present invention synchronously extracts a first image frame and a first audio frame from a video, performs type identification on the first image frame, obtains a target type of the first image frame, determines a target audio-visual synchronization detection algorithm according to the target type, and performs audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm. The present invention improves the flexibility and applicability of obtaining audio-visual synchronization detection results of a video by determining whether there are subtitles in the first image frame to obtain the target type of the image frame, and determines the target audio-visual synchronization detection algorithm according to the target type of the image frame, and performs audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm, thereby improving the accuracy and reliability of obtaining audio-visual synchronization detection results of a video.
[0052] Figure 2 It is a flowchart of the audio-video synchronization detection method according to the second embodiment of the present disclosure.
[0053] like Figure 2 As shown, in Figure 2 Based on the illustrated embodiment, the audio and video synchronization detection method of the embodiment of the present disclosure may specifically include the following steps:
[0054] S201, synchronously extracting a first image frame and a first audio frame from a video.
[0055] For the relevant contents of step S201, please refer to the above embodiment, which will not be described again here.
[0056] In the above embodiment, step S102 of “performing type identification on the first image frame to obtain the target type of the first image frame” may specifically include the following steps S202 and S204 .
[0057] S202: Perform content recognition on the first image frame to determine whether there are subtitles in the first image frame.
[0058] Optionally, a pre-trained deep learning model may be used to perform content recognition on the first image frame to determine whether there are subtitles in the first image frame.
[0059] S203: In response to the existence of subtitles in the first image frame, determine that the target type of the first image frame is a first type.
[0060] S204: In response to the first image frame not having any subtitles, determining that the target type of the first image frame is a second type.
[0061] In the above embodiment, step S103 "determining a target audio-visual synchronization detection algorithm according to the target type" may specifically include the following steps S205 and S206.
[0062] S205: In response to the target type being the first type, determining the subtitle speech synchronization detection algorithm as the target audio and video synchronization detection algorithm.
[0063] S206: In response to the target type being the second type, determining the lip synchronization detection algorithm as the target audio-video synchronization detection algorithm.
[0064] In an embodiment of the present disclosure, if the target audio and video synchronization detection algorithm is a subtitle speech synchronization detection algorithm, step S104 in the above embodiment "performing audio and video synchronization detection on the first image frame and the first audio frame based on the target audio and video synchronization detection algorithm" may specifically include the following steps S207 and S2010.
[0065] S207: Determine at least one first image frame having the same subtitles from the first image frames.
[0066] In the disclosed embodiment, the subtitle content of the first image frame may be extracted, and the second image frames having the same subtitles may be determined, and the second image frames may be sorted according to their timestamps to obtain an image frame sequence.
[0067] It should be noted that the present disclosure does not limit the specific method for extracting the subtitle content of the first image frame, and it can be selected according to actual conditions.
[0068] Optionally, a subtitle area in the first image frame may be determined, and subtitle content in the subtitle area in the first image frame may be extracted by using an optical character recognition (OCR) technology.
[0069] Optionally, a pre-trained subtitle text recognition model may be obtained, and the first image frame may be input into the subtitle text recognition model to extract subtitle content in the subtitle area in the first image frame.
[0070] In the embodiment of the present disclosure, after the subtitle content of the first image frame is acquired, the content in the first image frame may be compared to acquire the similarity between the subtitle contents, so as to determine the second image frame having the same subtitle.
[0071] For example, for the first image frame L1, by extracting the subtitle content of the first image frame L1, it is determined that the subtitle content of the first image frame L1 is "It is sunny today". The remaining first image frames L2, ...L n , extract the remaining first image frames L3, ...L n The subtitle content of the first image frame L3 is determined from the subtitle content and the subtitle content of the first image frame L3 is "The weather is fine today". Then, the second image frames with the same subtitles are determined from the first image frame L1 as L2 and L3.
[0072] S208: Determine a second audio frame synchronized with the second image frame from the first audio frame.
[0073] In the embodiment of the present disclosure, the timestamp of the second image frame and the timestamp of the first audio frame can be determined, and the start and end timestamps of the same subtitles can be determined based on the timestamp of the second image frame. Based on the start and end timestamps and the timestamp of the first audio frame, a second audio frame synchronized with the second image frame can be determined from the first audio frame.
[0074] It should be noted that the first second image frame and the last second image frame in the image frame sequence can be determined, and the timestamp of the first second image frame is determined as the start timestamp of the same subtitle, and the timestamp of the last second image frame is determined as the end timestamp of the same subtitle.
[0075] Optionally, the start timestamp and end timestamp of the same subtitle can be determined based on the start and end timestamps, and the audio frame with the same timestamp as the start timestamp can be determined from the audio list corresponding to the first audio frame as the first second audio frame. The audio frame with the same timestamp as the end timestamp can be determined from the audio list corresponding to the first audio frame as the last second audio frame. The audio frames from the first second audio frame to the last second audio frame in the audio list are determined as the second audio frames.
[0076] S209: Obtain a language recognition result of the second audio frame.
[0077] It should be noted that the present disclosure does not limit the specific method for obtaining the language recognition result of the second audio frame, which can be selected according to actual conditions.
[0078] Optionally, the second audio frames are combined in time order to obtain an audio frame sequence, and the audio frame sequence is input into a language recognition model to obtain a language recognition result of the second audio frame.
[0079] S2010: Perform audio-visual synchronization detection on the first image frame and the first audio frame according to the same subtitle and speech recognition results.
[0080] In the disclosed embodiment, the similarity between the same subtitles and speech recognition results can be obtained, and the audio-visual synchronization detection of the video can be performed based on the similarity and the set similarity (similarity threshold).
[0081] Optionally, the same subtitle and speech recognition results can be input into a pre-trained subtitle and speech similarity detection model, which performs cross-modal similarity calculation on the same subtitle and speech recognition results to obtain the similarity of the same subtitle and speech recognition results.
[0082] It should be noted that the present disclosure does not limit the setting of the preset similarity. For example, the similarity can be set to 100%; for another example, the similarity can be set to 98%.
[0083] Optionally, in response to the similarity between the same subtitle and speech recognition results being greater than a set similarity, the video audio and video synchronization is determined.
[0084] For example, for video A, if the similarity is set to 98%, and the similarity of the same subtitle and speech recognition results is 100%, it is determined that the audio and video of video A are synchronized.
[0085] Optionally, in response to the similarity between the same subtitle and speech recognition results being less than or equal to a set similarity, it is determined that the audio and picture of the video are not synchronized.
[0086] For example, for video A, if the similarity is set to 98%, and the similarity of the same subtitle and speech recognition results is 90%, it is determined that the audio and video of video A are not synchronized.
[0087] In an embodiment of the present disclosure, if the target audio-visual synchronization detection algorithm is a lip synchronization detection algorithm, step S104 in the above embodiment "performing audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm" may specifically include the following steps S2011 and S2013.
[0088] S2011, performing face detection and tracking on the first image frame to obtain one or more face identifiers, and grouping the first image frame according to the face identifiers to obtain multiple image lists.
[0089] Optionally, the first image frame may be input into a pre-trained face detection model, and the face detection model may perform face detection on the image frame to obtain a face in the image frame.
[0090] It should be noted that in order to improve the efficiency of audio and video detection, the face detection area in the image frame can be obtained, and the faces that are less than or equal to the set value (face detection area threshold) can be eliminated to obtain the faces (candidate faces) in the final image frame.
[0091] Optionally, after obtaining the human face in the final image frame, the human face may be tracked to obtain one or more face identifications (Identity Document, ID for short).
[0092] In the embodiment of the present disclosure, after the face identifier is obtained, the face identifier can be marked in the image frame, and the image frames can be grouped according to the face identifier to obtain multiple image lists {F1, F2, ...F i}.
[0093] For example, for face identifier ID1, image frames containing face identifier ID1 can be determined from image frames, and the image frames containing face identifier ID1 can be divided into a group to obtain image list F1. For face identifier ID2, image frames containing face identifier ID2 can be determined from image frames, and the image frames containing face identifier ID2 can be divided into a group to obtain image list F2.
[0094] S2012: Determine, according to the first image frame in the image list, a mouth region picture corresponding to the image list.
[0095] It should be noted that the present disclosure does not limit the specific method of determining the mouth area picture corresponding to the image list according to the image frames in the image list, and it can be selected according to actual conditions.
[0096] Optionally, the first image frame in the image list may be input into a pre-trained facial feature extraction model, and the pre-trained facial feature extraction model may perform feature extraction on the first image frame to obtain a mouth region image corresponding to the image list.
[0097] For example, for image list F1, the first image frame in image list F1 can be input into a pre-trained facial feature extraction model, and the pre-trained facial feature extraction model performs feature extraction on the first image frame to obtain the mouth area picture M1 corresponding to image list F1; for image list F2, the first image frame in image list F2 can be input into a pre-trained facial feature extraction model, and the pre-trained facial feature extraction model performs feature extraction on the first image frame to obtain the mouth area picture M2 corresponding to image list F2.
[0098] S2013, performing audio-visual synchronization detection on the mouth area image and the first audio frame corresponding to the image list.
[0099] In the disclosed embodiments, audio feature extraction can be performed on the first audio frame to obtain an audio feature sequence of the first audio frame, lip movement feature extraction can be performed on the mouth area picture corresponding to the image list to obtain a lip movement feature sequence corresponding to the image list, and the lip movement feature sequence and audio feature sequence corresponding to the image list can be used to obtain the lip sound similarity corresponding to the image list. Based on the lip sound similarity corresponding to the image list, audio and video synchronization detection can be performed on the video.
[0100] In an embodiment of the present disclosure, before performing audio feature extraction on the first audio frame to obtain an audio feature sequence of the audio frame, mouth key points may be extracted from a mouth area image in a mouth area image sequence, and the mouth key points may be tracked to obtain a motion trajectory of the key points. In response to the motion trajectory indicating that the mouth has opened and closed changes, it is determined that the mouth is in a lip movement state.
[0101] Among them, when it is determined that the mouth is in a lip movement state, it can be understood as a "speaking" state.
[0102] In the embodiment of the present disclosure, after obtaining the first audio frame {A1, <2, ...A n}, the first audio frames can be sorted according to the timestamps of the first audio frames to obtain a first audio frame sequence {A t1 , A t2 , …A tn}, extract the audio features of the first audio frame sequence to obtain the audio feature sequence of the first audio frame {A′ t1 , A′ t2 …A′ tn}.
[0103] It should be noted that the present disclosure does not limit the specific method of extracting lip movement features from the mouth area pictures corresponding to the image list to obtain the lip movement feature sequence corresponding to the image list, and it can be selected according to actual conditions.
[0104] Optionally, the mouth area pictures corresponding to the image list may be sorted according to timestamps to obtain a mouth area picture sequence, and lip movement features may be extracted from the mouth area picture sequence to obtain a lip movement feature sequence corresponding to the image list.
[0105] For example, a sequence of mouth area images may be input into a pre-trained lip movement detection model, and the lip movement detection model may extract lip movement features from the lip movement feature sequence to obtain a lip movement feature sequence corresponding to the image list.
[0106] It should be noted that the present disclosure does not limit the specific method of obtaining the lip-sound similarity corresponding to the image list based on the lip movement feature sequence and audio feature sequence corresponding to the image list, and it can be selected according to actual conditions.
[0107] Optionally, the lip movement feature sequence and audio feature sequence corresponding to the image list may be input into a pre-trained lip synchronization detection model, and the lip synchronization detection model may perform cross-modal similarity calculation on the lip movement feature sequence and the audio feature sequence to obtain the lip synchronization similarity corresponding to the image list.
[0108] In the embodiment of the present disclosure, after the lip sound similarity corresponding to the image list is obtained, the video is subjected to audio and video synchronization detection according to the lip sound similarity corresponding to the image list.
[0109] Optionally, a preset similarity threshold may be determined to identify whether the lip sound similarity corresponding to the image list is greater than the similarity threshold, a statistical number of image lists whose lip sound similarity is greater than or equal to the similarity threshold may be obtained, and audio-visual synchronization detection may be performed on the video based on the statistical number.
[0110] It should be noted that the present disclosure does not limit the setting of the similarity threshold, which can be set according to actual conditions.
[0111] In the embodiment of the present disclosure, a preset quantity threshold may be determined, and audio and video synchronization detection may be performed on the video based on the statistical quantity and the preset quantity threshold.
[0112] It should be noted that the present disclosure does not limit the setting of the preset quantity threshold N. For example, the quantity threshold N can be preset to 1.
[0113] Optionally, in response to the statistical quantity being greater than or equal to a quantity threshold, it is determined that the video audio and picture are synchronized.
[0114] For example, if the statistical quantity n is 2 and the preset quantity threshold N is 1, that is, the statistical quantity is greater than the quantity threshold, then it is determined that the video audio and picture are synchronized.
[0115] Optionally, in response to the statistical quantity being less than a quantity threshold, it is determined that the video audio and picture are not synchronized.
[0116] For example, if the statistical quantity n is 0 and the preset quantity threshold N is 1, that is, the statistical quantity is less than the quantity threshold, it is determined that the video audio and picture are not synchronized.
[0117] In summary, the audio-video synchronization detection method proposed in the embodiment of the present disclosure performs content recognition on the first image frame. When the first image frame has subtitles, the subtitle voice synchronization detection algorithm is determined as the target audio-video synchronization detection algorithm. At least one second image frame with the same subtitle is determined from the first image frame, and the subtitle information of the video is fully utilized to perform audio-video synchronization detection on the video according to the second image frame and the second audio frame synchronized with the second image frame. According to the similarity and the set similarity of the same subtitle and the speech recognition result, it is determined whether the video has audio-video synchronization, thereby avoiding the problem of missed detection in the process of audio-video synchronization detection, and improving the accuracy and reliability of the audio-video synchronization detection result of the video. When the first image frame does not have subtitles, the lip-synch detection algorithm is determined as the target audio-video synchronization detection algorithm, and the image frames are grouped according to the face identifier to obtain multiple image lists, thereby solving the problem of audio-video asynchronous detection in a multi-person scenario. According to the lip-synch similarity corresponding to the image list, the video is subjected to audio-video synchronization detection, thereby avoiding the problem of missed detection in the process of audio-video synchronization detection, and improving the accuracy and reliability of the audio-video synchronization detection result of the video, thereby improving the user's experience of watching the video.
[0118] The following is an explanation of the specific process of performing audio and video synchronization detection on the first image frame and the first audio frame based on the target audio and video synchronization detection algorithm proposed in the embodiment of the present disclosure.
[0119] In the embodiment of the present disclosure, if the target audio-visual synchronization detection algorithm is a subtitle speech synchronization detection algorithm, such as Figure 3 As shown, step S104 in the above embodiment, “performing audio and video synchronization detection on the first image frame and the first audio frame based on the target audio and video synchronization detection algorithm” may specifically include the following steps S301 and S307 .
[0120] S301, obtaining a video segment with a video duration of T, extracting a first image frame from the video at a time interval and storing it in an image list, and extracting a first audio frame from the video at a time interval and storing it in an audio list.
[0121] S302: Determine at least one second image frame having the same subtitles from the first image frame, and determine a second audio frame synchronized with the second image frame from the first audio frame according to a timestamp of the second image frame and a timestamp of the first audio frame.
[0122] S303, combining the second audio frames in time order to obtain an audio frame sequence, and inputting the audio frame sequence into a language recognition model to obtain a speech recognition result.
[0123] S304: Input the same subtitle and speech recognition results into a subtitle and speech similarity detection model to obtain the similarity between the same subtitle and speech recognition results.
[0124] S305, performing audio and video synchronization detection on the video according to the similarity between the same subtitles and the speech recognition results, that is, determining whether the audio and video of the video are synchronized.
[0125] S306: In response to the similarity being greater than the set similarity, determining that the video audio and video are synchronized.
[0126] S307: In response to the similarity being less than or equal to the set similarity, determining that the video audio and picture are not synchronized.
[0127] In summary, the audio-visual synchronization detection method proposed in the present invention, if it is determined that there are subtitles in the first image frame, determines the subtitle speech synchronization detection algorithm as the target audio-visual synchronization detection algorithm, by determining at least one second image frame with the same subtitles from the first image frame, and obtaining the speech recognition result of the second audio frame synchronized with the second image frame, and inputting the same subtitles and speech recognition results into the subtitle speech similarity detection model, obtaining the similarity of the same subtitles and speech recognition results, and judging whether the video has audio-visual synchronization according to the similarity of the same subtitles and speech recognition results, thereby avoiding the problem of missed detection in the process of audio-visual synchronization detection, preventing the problem of secondary detection due to missed detection in the process of audio-visual synchronization detection, reducing the cost in the process of audio-visual synchronization detection, improving the efficiency of audio-visual synchronization detection, and at the same time ensuring the accuracy and reliability of the audio-visual synchronization detection result of the video, improving the user's experience of watching the video, and having a wide range of applications.
[0128] In the embodiment of the present disclosure, if the target audio-visual synchronization detection algorithm is a lip synchronization detection algorithm, such as Figure 4 As shown, step S104 in the above embodiment, “performing audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm” may specifically include the following steps S401 and S408 .
[0129] S401, obtaining a video segment with a video length of T, extracting a first image frame from the video at a time interval and storing it in an image list, and extracting a first audio frame from the video at a time interval and storing it in an audio list.
[0130] S402, performing face detection on the first image frame in the image list, obtaining faces whose face detection area is larger than a set value, tracking the faces according to face features, grouping the image frames according to face identifiers, and obtaining multiple image lists.
[0131] S403, traversing each image list to determine a mouth area picture sequence corresponding to each image list.
[0132] S404, determining whether the mouth is in a lip movement state.
[0133] It should be noted that the mouth key points can be extracted from the mouth area images in the mouth area image sequence, and the mouth key points can be tracked to obtain the movement trajectory of the key points. In response to the movement trajectory indicating that the mouth has opened and closed changes, it is determined that the mouth is in a lip movement state.
[0134] It should be noted that by judging whether the mouth opens and closes, that is, whether the person is speaking, and determining whether the mouth is in lip movement state before executing subsequent steps, it is suitable for the scenario of synchronous audio and video detection of multiple characters.
[0135] If it is determined that the mouth is opening and closing, that is, the mouth is in a lip movement state, step S405 is executed.
[0136] If it is determined that there is no opening or closing of the mouth, that is, the mouth is not in a lip movement state, step S403 is executed.
[0137] S405, extract lip movement features from the mouth area pictures corresponding to the image list to obtain a lip movement feature sequence corresponding to the image list, input the lip movement feature sequence and the audio feature sequence corresponding to the image list into a pre-trained lip-sound synchronization detection model, and use the lip-sound synchronization detection model to perform cross-modal similarity calculation on the lip movement feature sequence and the audio feature sequence to obtain the lip-sound similarity corresponding to the image list.
[0138] It should be noted that the lip synchronization detection model is used to calculate the cross-modal similarity of the lip movement feature sequence and the audio feature sequence, and the audio-visual synchronization detection is realized according to the lip synchronization similarity, thereby improving the accuracy of the audio-visual synchronization detection results.
[0139] S406, performing audio and video synchronization detection on the video according to the lip sound similarity corresponding to the image list, that is, determining whether the audio and video of the video are synchronized.
[0140] S407: In response to the fact that the statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold is greater than or equal to the number threshold, it is determined that the video audio and picture are synchronized.
[0141] S408: In response to the statistical number of the image list having lip sound similarity greater than or equal to the similarity threshold being less than the number threshold, it is determined that the audio and picture of the video are not synchronized.
[0142] To sum up, the audio and video synchronization detection method proposed in the present invention, if it is determined that there are no subtitles in the first image frame, determines that the subtitle voice synchronization detection algorithm is a lip synchronization detection algorithm, and groups the image frames by face identification to obtain multiple image lists, which solves the problem of audio and video asynchronous detection in a multi-person scenario, and performs audio and video synchronization detection on the video according to the lip sound similarity corresponding to the image list, avoiding the problem of missed detection in the audio and video synchronization detection process, preventing the problem of secondary detection due to missed detection in the audio and video synchronization detection process, reducing the cost in the audio and video synchronization detection process, improving the efficiency of audio and video synchronization detection, and at the same time ensuring the accuracy and reliability of the audio and video synchronization detection results of the video, thereby improving the user's experience of watching the video.
[0143] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0144] According to an embodiment of the present disclosure, the present disclosure further provides an audio-video synchronization detection device, which is used to implement the above-mentioned audio-video synchronization detection method.
[0145] Figure 5 It is a block diagram of an audio-video synchronization detection device according to an embodiment of the present disclosure.
[0146] like Figure 5 As shown, the audio-video synchronization detection device 500 includes: an extraction module 501 , a recognition module 502 , a determination module 503 and a detection module 504 .
[0147] An extraction module 501 is used to synchronously extract a first image frame and a first audio frame from a video;
[0148] An identification module 502 is used to perform type identification on the first image frame to obtain a target type of the first image frame;
[0149] A determination module 503, configured to determine a target audio-visual synchronization detection algorithm according to the target type;
[0150] The detection module 504 is configured to perform audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm.
[0151] In one embodiment of the present disclosure, the identification module 502 is used to: perform content identification on the first image frame to determine whether there are subtitles in the first image frame; in response to the presence of subtitles in the first image frame, determine that the target type of the first image frame is a first type; in response to the absence of subtitles in the first image frame, determine that the target type of the first image frame is a second type.
[0152] In one embodiment of the present disclosure, the determination module 503 is used to: in response to the target type being the first type, determine the subtitle speech synchronization detection algorithm as the target audio and video synchronization detection algorithm; in response to the target type being the second type, determine the lip synchronization detection algorithm as the target audio and video synchronization detection algorithm.
[0153] In one embodiment of the present disclosure, the target audio-visual synchronization detection algorithm is the subtitle speech synchronization detection algorithm, and the detection module 504 is used to: determine at least one first image frame with the same subtitles from the first image frame; determine the timestamp of the second image frame and the timestamp of the first audio frame; determine the start and end timestamps of the same subtitles according to the timestamp of the second image frame; determine the second audio frame synchronized with the second image frame from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame; obtain the language recognition result of the second audio frame; and perform audio-visual synchronization detection on the first image frame and the first audio frame according to the same subtitles and the speech recognition result.
[0154] In one embodiment of the present disclosure, the detection module 504 is used to: extract the subtitle content of the first image frame and determine the second image frame with the same subtitle; sort the second image frames according to the timestamps of the second image frames to obtain an image frame sequence.
[0155] In one embodiment of the present disclosure, the target audio-visual synchronization detection algorithm is the lip synchronization detection algorithm, and the detection module 504 is used to: perform face detection and tracking on the first image frame to obtain one or more face identifiers, and group the first image frames according to the face identifiers to obtain multiple image lists; determine the mouth area picture corresponding to the image list according to the image frames in the image list; perform audio-visual synchronization detection on the mouth area picture corresponding to the image list and the first audio frame.
[0156] In one embodiment of the present disclosure, the detection module 504 is used to: sort the first audio frames according to the timestamps of the first audio frames to obtain a first audio frame sequence; extract audio features from the first audio frame sequence to obtain the audio feature sequence; sort the mouth area pictures corresponding to the image list according to the timestamps to obtain a mouth area picture sequence; extract lip movement features from the mouth area picture sequence to obtain a lip movement feature sequence corresponding to the image list; obtain the lip movement feature similarity corresponding to the image list based on the lip movement feature sequence and the audio feature sequence corresponding to the image list; and perform audio-visual synchronization detection on the video based on the lip movement feature sequence corresponding to the image list.
[0157] In one embodiment of the present disclosure, the device 500 is used to: extract mouth key points from the mouth area image in the mouth area image sequence, and track the mouth key points to obtain the motion trajectory of the key points; in response to the motion trajectory indicating that the mouth has opening and closing changes, determine that the mouth is in a lip movement state.
[0158] The audio-visual synchronization detection device proposed in the present disclosure synchronously extracts a first image frame and a first audio frame from a video, performs type identification on the first image frame, obtains a target type of the first image frame, determines a target audio-visual synchronization detection algorithm according to the target type, and performs audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm. The present disclosure improves the flexibility and applicability of obtaining audio-visual synchronization detection results of a video by determining whether there are subtitles in the first image frame to obtain the target type of the image frame, and determines the target audio-visual synchronization detection algorithm according to the target type of the image frame, and performs audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm, thereby improving the accuracy and reliability of obtaining audio-visual synchronization detection results of a video.
[0159] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a terminal and a computer program product.
[0160] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0161] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0162] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0163] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the audio-visual synchronization detection method. For example, in some embodiments, the audio-visual synchronization detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the audio-visual synchronization detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the audio-visual synchronization detection method in any other appropriate manner (e.g., by means of firmware).
[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0165] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0166] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0167] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).
[0168] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0169] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0170] According to an embodiment of the present disclosure, the present disclosure also provides a terminal, such as Figure 7 As shown, the terminal 700 includes the above-mentioned electronic device 600.
[0171] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the audio-visual synchronization detection method described in the above embodiment of the present disclosure are implemented.
[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0173] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for detecting synchronization between audio and video, wherein: The method comprises: Synchronously extracting a first image frame and a first audio frame from the video; Performing type recognition on the first image frame to obtain a target type of the first image frame; Determining a target audio and video synchronization detection algorithm according to the target type; Based on the target audio-visual synchronization detection algorithm, audio-visual synchronization detection is performed on the first image frame and the first audio frame.
2. The method according to claim 1, wherein: The performing type identification on the first image frame to obtain the target type of the image frame includes: Performing content recognition on the first image frame to determine whether there are subtitles in the first image frame; In response to the presence of subtitles in the first image frame, determining that the target type of the first image frame is a first type; In response to the first image frame not having any subtitles, determining the target type of the first image frame as a second type.
3. The method according to claim 2, wherein: Determining a target audio and video synchronization detection algorithm according to the target type includes: In response to the target type being the first type, determining the subtitle speech synchronization detection algorithm to be the target audio-visual synchronization detection algorithm; In response to the target type being the second type, the lip synchronization detection algorithm is determined to be the target audio-visual synchronization detection algorithm.
4. The method according to claim 3, wherein: The target audio-visual synchronization detection algorithm is the subtitle speech synchronization detection algorithm, and the audio-visual synchronization detection is performed on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm, including: determining at least one first image frame having the same subtitles from among the first image frames; Determine a second audio frame synchronized with the second image frame from the first audio frame; Obtaining a language recognition result of the second audio frame; According to the same subtitles and the speech recognition result, audio-visual synchronization detection is performed on the first image frame and the first audio frame.
5. The method according to claim 4, wherein: The determining at least one second image frame having the same subtitle from the first image frame comprises: Extracting subtitle content of the first image frame, and determining the second image frame having the same subtitle; The second image frames are sorted according to the timestamps of the second image frames to obtain an image frame sequence.
6. The method according to claim 4, wherein: The determining, from the first audio frame, a second audio frame synchronized with the second image frame comprises: determining a timestamp of the second image frame and a timestamp of the first audio frame; Determining the start and end timestamps of the same subtitle according to the timestamp of the second image frame; The second audio frame synchronized with the second image frame is determined from the first audio frame according to the start and end timestamps and the timestamp of the first audio frame.
7. The method according to claim 3, wherein: The target audio-visual synchronization detection algorithm is the lip-synch detection algorithm, and performing audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm includes: Performing face detection and tracking on the first image frame to obtain one or more face identifiers, and grouping the first image frame according to the face identifiers to obtain a plurality of image lists; Determine, according to the first image frame in the image list, a mouth region picture corresponding to the image list; Perform audio-visual synchronization detection on the mouth area picture corresponding to the image list and the first audio frame.
8. The method according to claim 7, wherein: The performing audio-visual synchronization detection on the mouth area picture corresponding to the image list and the first audio frame includes: Extracting audio features from the first audio frame to obtain an audio feature sequence of the first audio frame; Extracting lip movement features from the mouth area pictures corresponding to the image list to obtain a lip movement feature sequence corresponding to the image list; According to the lip movement feature sequence and the audio feature sequence corresponding to the image list, obtaining the lip sound similarity corresponding to the image list; According to the lip sound similarity corresponding to the image list, the audio and video synchronization detection is performed on the video.
9. The method according to claim 8, wherein: The step of extracting lip movement features from the mouth area pictures corresponding to the image list to obtain a lip movement feature sequence corresponding to the image list includes: Sorting the mouth area pictures corresponding to the image list according to the timestamps to obtain a mouth area picture sequence; The lip movement features are extracted from the mouth area image sequence to obtain a lip movement feature sequence corresponding to the image list.
10. The method according to claim 8, wherein: The extracting audio features from the first audio frame to obtain an audio feature sequence of the first audio frame includes: Sort the first audio frames according to timestamps of the first audio frames to obtain a first audio frame sequence; Audio features are extracted from the first audio frame sequence to obtain the audio feature sequence.
11. The method according to any one of claims 9 to 10, wherein: Before extracting audio features from the first audio frame to obtain an audio feature sequence of the first audio frame, the method further includes: Extracting mouth key points from the mouth area pictures in the mouth area picture sequence, and tracking the mouth key points to obtain movement trajectories of the key points; In response to the movement trajectory indicating that the mouth has opening and closing changes, it is determined that the mouth is in a lip movement state.
12. A device for detecting synchronization between audio and video, wherein: The device comprises: An extraction module, used for synchronously extracting a first image frame and a first audio frame from a video; an identification module, configured to perform type identification on the first image frame to obtain a target type of the first image frame; A determination module, used to determine a target audio and video synchronization detection algorithm according to the target type; A detection module is used to perform audio-visual synchronization detection on the first image frame and the first audio frame based on the target audio-visual synchronization detection algorithm.
13. The device according to claim 12, wherein: The identification module is used to: Performing content recognition on the first image frame to determine whether there are subtitles in the first image frame; In response to the presence of subtitles in the first image frame, determining that the target type of the first image frame is a first type; In response to the first image frame not having any subtitles, determining the target type of the first image frame as a second type.
14. The device according to claim 13, wherein: The determining module is used to: In response to the target type being the first type, determining the subtitle speech synchronization detection algorithm to be the target audio-visual synchronization detection algorithm; In response to the target type being the second type, the lip synchronization detection algorithm is determined to be the target audio-visual synchronization detection algorithm.
15. The device according to claim 14, wherein: The target audio-visual synchronization detection algorithm is the subtitle speech synchronization detection algorithm, and the detection module is used to: determining at least one first image frame having the same subtitles from among the first image frames; determining a timestamp of the second image frame and a timestamp of the first audio frame; Determining the start and end timestamps of the same subtitle according to the timestamp of the second image frame; Determine, from the first audio frame, the second audio frame synchronized with the second image frame according to the start and end timestamps and the timestamp of the first audio frame; Obtaining a language recognition result of the second audio frame; According to the same subtitles and the speech recognition result, audio-visual synchronization detection is performed on the first image frame and the first audio frame.
16. The device according to claim 15, wherein: The detection module is used for: Extracting subtitle content of the first image frame, and determining the second image frame having the same subtitle; The second image frames are sorted according to the timestamps of the second image frames to obtain an image frame sequence.
17. The device according to claim 14, wherein: The target audio-visual synchronization detection algorithm is the lip-sync detection algorithm, and the detection module is used to: Performing face detection and tracking on the first image frame to obtain one or more face identifiers, and grouping the first image frame according to the face identifiers to obtain a plurality of image lists; Determine, according to the image frames in the image list, the mouth area picture corresponding to the image list; Perform audio-visual synchronization detection on the mouth area picture corresponding to the image list and the first audio frame.
18. The device according to claim 17, wherein: The detection module is used for: Sort the first audio frames according to timestamps of the first audio frames to obtain a first audio frame sequence; Extracting audio features from the first audio frame sequence to obtain the audio feature sequence; Sorting the mouth area pictures corresponding to the image list according to the timestamps to obtain a mouth area picture sequence; Extracting lip movement features from the mouth area image sequence to obtain a lip movement feature sequence corresponding to the image list; According to the lip movement feature sequence and the audio feature sequence corresponding to the image list, obtaining the lip sound similarity corresponding to the image list; According to the lip sound similarity corresponding to the image list, the audio and video synchronization detection is performed on the video.
19. The device according to any one of claims 17-18, wherein: The device is used for: Extracting mouth key points from the mouth area pictures in the mouth area picture sequence, and tracking the mouth key points to obtain movement trajectories of the key points; In response to the movement trajectory indicating that the mouth has opening and closing changes, it is determined that the mouth is in a lip movement state.
20. An electronic device, characterized in that: including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 11.
21. A terminal, characterized in that: Comprising the electronic device as claimed in claim 20.
22. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
Citation Information
Cited By
Multimedia data quality detection method and device, equipment and medium
CN120067089A
