Sound and picture synchronization detection method and device, electronic equipment and terminal
By extracting the image and audio data of the video segment, face detection and mouth feature extraction, and cross-modal similarity calculation is performed with audio data, which solves the problem of mismatch between lips and sound in the video and improves the synchronous detection of audio and video. Accuracy and reliability improve user experience.
Patent Information
- Application Number
- CN202411262508.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-06
AI Technical Summary
In the process of video editing, encoding and playback, the lips and sound in the video screen do not match, which affects the user's viewing experience, and often requires manual secondary detection, which consumes a lot of manpower and time, and cannot be effective in actual scenes. There is a problem with false detection of the audio and video synchronization detection method applied.
Extract image data and audio data of video segments, perform face detection and tracking, divide personal face image list, extract mouth features, and confirm the synchronization results of video segments based on audio data, and cross-modal similarity through mouth features and audio data. degree calculation to realize synchronous audio and video detection.
It improves the accuracy and reliability of audio and video synchronization detection, avoids missed detection problems, reduces detection costs, and improves the user's viewing experience.
Smart Images

Figure CN119094739A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, specifically to the technical fields of computer vision and artificial intelligence, and in particular to a method, device, electronic device and terminal for detecting synchronization between audio and video. Background Art
[0002] During video editing, encoding, and playback, there may be a mismatch between lips and voice in the video, affecting the user's viewing experience. In related technologies, a reference method is often used for audio-visual synchronization detection, such as adding marker information or comparing with reference videos, or a non-reference method is used for audio-visual synchronization detection, such as based on the opening and closing information of the lip shape or whether there is voice information. However, the above methods cannot be applied in actual scenarios, and false detection often occurs. In the end, manual secondary detection is required, which consumes a lot of manpower and time. Therefore, how to improve the accuracy and reliability of audio-visual synchronization detection has become an urgent problem to be solved. Summary of the invention
[0003] The present disclosure provides a method, device, electronic device, terminal and computer program product for detecting audio and video synchronization.
[0004] According to the first aspect of the present disclosure, a method for detecting audio-visual synchronization is proposed, comprising: extracting image data and audio data of a video segment of a target length; performing face detection and tracking based on the extracted image data to divide the data into multiple face image lists; extracting mouth features corresponding to each face image list based on the traversal results of each face image list, wherein the mouth features are used to characterize changes in lips; and confirming the synchronization result of the video segment based on the audio data and the mouth features.
[0005] According to the second aspect of the present disclosure, a device for detecting audio-visual synchronization is proposed, comprising: a first extraction module for extracting image data and audio data of a video segment of a target length; a division module for performing face detection and tracking based on the extracted image data to divide the data into multiple face image lists; a second extraction module for extracting mouth features corresponding to each face image list based on the traversal results of each face image list, wherein the mouth features are used to characterize changes in lips; and a confirmation module for confirming the synchronization result of the video segment based on the audio data and the mouth features.
[0006] According to the third aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio and video synchronization detection method proposed in the first aspect above.
[0007] According to a fourth aspect of an embodiment of the present disclosure, a terminal is provided, comprising the electronic device proposed in the embodiment of the third aspect.
[0008] According to a fifth aspect of the present disclosure, a computer program product is proposed, including a computer program, wherein when the computer program is executed by a processor, the audio and video synchronization detection method proposed in the first aspect is implemented.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0011] Figure 1 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0012] Figure 2 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0013] Figure 3 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0014] Figure 4 A flowchart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure is shown;
[0015] Figure 5 It is a structural schematic diagram of an audio and video synchronization detection device according to an embodiment of the present disclosure;
[0016] Figure 6 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure;
[0017] Figure 7 is a schematic block diagram of a terminal according to an embodiment of the present disclosure; DETAILED DESCRIPTION
[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0019] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.
[0020] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.
[0021] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.
[0022] In the embodiment of the present disclosure, a video reading tool can be used to read the video to be subjected to audio and video synchronization detection, and the video can be decoded by the video reading tool to obtain a video segment of a target length, and audio and video detection can be performed on the video segment to obtain the audio and video synchronization detection result of the video segment.
[0023] The video decoding formats include but are not limited to: (1) H.264 / AVC (Advanced Video Coding), the video decoder is libx264; (2) H.265 / HEVC (High Efficiency Video Coding), the video decoder is libx265; (3) VP8 (Video Coding Format by Google), the video decoder is libvpx; (4) VP9 (Successor to VP8), the video decoder is libvpx-vp9; (5) AV1 (AOMedia Video 1), the video decoder is libaom-av1; (6) MPEG-2 (Moving Picture Experts Group Phase 2), the video decoder is mpeg2video; (7) MPEG-4 (Moving Picture Experts Group Phase 4), the video decoder is mpeg4; (8) Theora (Free and Open Video Codec), the video decoder is libtheora; ProRes (AppleProRes), the video decoder is prores; (10) DNxHD (Digital Nonlinear Extensible High Definition), the video decoder is dnxhd; (11) WMV (Windows Media Video), the video decoder is wmv2; (12) FLV1 (Flash Video), the video decoder is flv; (13) MJPEG (Motion JPEG), the video decoder is mjpeg
[0024] The audio decoding formats include but are not limited to: (1) AAC (Advanced Audio Coding), the audio decoder is aac; (2) MP3 (MPEG-1 Audio Layer 3), the audio decoder is libmp3lame; (3) Vorbis (Ogg Vorbis), the audio decoder is libvorbis; (4) Opus (Audio Codec), the audio decoder is libopus; (5) FLAC (Free Lossless Audio Codec), the audio decoder is flac; (6) ALAC (Apple Lossless Audio Code), the audio decoder is alac; (7) AC3 (Audio Codec 3), the audio decoder is ac3; (8) WMA (Windows Media Audio), the audio decoder is wmav2; (9) WAV (Waveform Audio File Format), the audio decoder is the built-in WAV decoder; (10) AMR (Adaptive Audio File Format), the audio decoder is the built-in WAV decoder; Multi-Rate) encoder: the audio decoder is libopencore_amrnb (NB) and libopencore_amrwb (WB); (11) PCM (Pulse Code Modulation), the audio decoder is pcm_s16le, pcm_s24le, etc.
[0025] In the video encoding and decoding process, the basic process is as follows: in the encoder, a frame of image is divided into blocks, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, the original block of the current block minus the prediction block to obtain the residual block, the residual block is transformed and quantized to obtain the quantization coefficient matrix, and the quantization coefficient matrix is entropy encoded and output to the bitstream. In the decoder, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block, on the other hand, the bitstream is decoded to obtain the quantization coefficient matrix, the quantization coefficient matrix is inversely quantized and inversely transformed to obtain the residual block, and the prediction block and the residual block are added to obtain the reconstructed block. The reconstructed blocks constitute the reconstructed image, and the reconstructed image is loop-filtered based on the image or block to obtain the decoded image. The encoder also needs similar operations as the decoder to obtain the decoded image. The decoded image can be used as a reference frame for inter-frame prediction for subsequent frames. The block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information determined by the encoder need to be output to the bitstream if necessary; then the decoder determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information as the encoder by parsing and analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder. The decoded image obtained by the encoder is also usually called a reconstructed image. The current block can be divided into prediction units during prediction, and the current block can be divided into transformation units during transformation. The division of prediction units and transformation units can be different. The above is the basic process of the video encoder and decoder under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The embodiment of the present application is applicable to the basic process under the block-based hybrid coding framework, but is not limited to the framework and process.
[0026] At present, the common video coding and decoding standards all adopt the block-based hybrid coding framework. Each frame in the video image is divided into square LCUs of the same size (such as 128×128, 64×64, etc.), and each LCU can also be divided into rectangular CUs according to rules; and CUs may also be divided into smaller PUs, TUs, etc. Specifically, the coding framework may include steps such as prediction, transform, quantization, entropy coding, and in-loop filtering. Among them, prediction can be divided into intra-frame prediction and inter-frame prediction, and inter-frame prediction can include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the use of intra-frame prediction in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels; however, since there is also a strong similarity between adjacent frames in a video image, the use of inter-frame prediction in video coding and decoding technology can eliminate the temporal redundancy between adjacent frames, thereby improving the coding and decoding efficiency.
[0027] Videos are composed of multiple images. In order to make the video look smooth, each second of video contains dozens or even hundreds of frames, such as 24 frames per second, 30 frames per second, 50 frames per second, 60 frames per second or 120 frames per second. Therefore, there is a lot of temporal redundancy in the video, or in other words, there is a lot of temporal correlation in the video. Inter-frame prediction uses this temporal correlation to improve compression efficiency. Inter-frame prediction often uses "motion" to exploit temporal correlation. A very simple "motion" model is that an object is at a certain position on the image corresponding to a certain moment, and after a certain period of time, it moves to another position on the image corresponding to the current moment. This is the most basic and commonly used translation motion in video coding and decoding. Inter-frame prediction uses motion information to represent "motion". Basic motion information includes the information of the reference frame and the information of the MV; the reference frame can also be understood as the reference picture. The codec determines the reference frame / reference picture based on the information of the reference frame / reference picture, and determines the coordinates of the reference block based on the information of the MV and the coordinates of the current block. The reference block is determined by the coordinates of the reference block in the reference picture. Using a determined reference block as a prediction block is the most basic prediction method for inter-frame prediction.
[0028] Figure 1FIG. 1 is a flow chart of a method for detecting audio and video synchronization according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:
[0029] S101, extracting image data and audio data of a video segment of a target length.
[0030] It should be noted that the execution subject of the audio-video synchronization detection method of the embodiment of the present disclosure may be a hardware device with data audio-video synchronization detection capability and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc.
[0031] The video segment is any video segment that needs to be detected for audio and video synchronization after being decoded.
[0032] A video segment generally includes image data and audio data, and the image data and the audio data are aligned on the time axis.
[0033] It should be noted that the present disclosure does not limit the specific method for extracting the image data and audio data of the video segment of the target length, and it can be selected according to the actual situation.
[0034] Optionally, audio frames within a video segment of a target length are extracted, and an audio data list is generated based on the audio frames; image frames within a video segment of a target length are extracted, and an image data list is generated based on the image frames.
[0035] Optionally, a preset time interval may be obtained, and image frames {L1, L2, ..., L1, L2, L2, L3, L4, L5, L6, L7, L8, L9, L10, L11, L12, L13, L14, L15, L16, L17, L18, L19, L20, L21, L30, L41, L52, L63, L7, L8, L9, L n}, according to the image frame {L1, L2, ...L n}Generate an image data list {L}, extract audio frames {A1, A2, ...A} in a video segment of target length according to the time interval from the video segment n} to generate an audio data list {A} based on the audio frames.
[0036] For example, a video frame extraction tool can be used to extract image frames within a video segment of a target length according to a time interval, and an audio extraction tool can be used to extract audio frames within a video segment of a target length according to a time interval.
[0037] S102, performing face detection and tracking according to the extracted image data to divide it into a plurality of face image lists.
[0038] In the embodiments of the present disclosure, image frames in an image data list can be detected to obtain one or more face images in which the face area in the image frames is greater than a preset threshold, the face images in each image frame are tracked according to the face features, and one or more face identifiers are obtained, and one or more face lists are generated according to the face images and the face identifiers.
[0039] It should be noted that in order to improve the efficiency of audio and video detection, the face area in the image frame can be obtained, and the faces less than or equal to the set value (face area threshold) can be eliminated to obtain the final face image.
[0040] Optionally, after the final face image is obtained, the face image in each image frame may be tracked according to the face features, and one or more face identifications (Identity Document, ID for short) may be obtained.
[0041] In the embodiment of the present disclosure, after the face identifier is obtained, the face identifier can be marked in the image frame, and one or more face lists {F1, F2, ...F i}.
[0042] The number of image lists is the number of different face identifiers.
[0043] For example, for face identifier ID1, image frames containing face identifier ID1 can be determined from image frames, and the image frames containing face identifier ID1 can be divided into a group to obtain image list F1. For face identifier ID2, image frames containing face identifier ID2 can be determined from image frames, and the image frames containing face identifier ID2 can be divided into a group to obtain image list F2.
[0044] S103, extracting mouth features corresponding to each face image list according to the traversal result of each face image list, wherein the mouth features are used to characterize changes in lips.
[0045] In the disclosed embodiment, the face images in each face list may be traversed, and the mouth features of each face image may be extracted, and a mouth feature list of each face image may be generated based on the mouth features.
[0046] It should be noted that the present disclosure does not limit the specific method for extracting the mouth features of each face image, and the method can be selected according to actual conditions.
[0047] Optionally, the image frames in the face image list can be input into a pre-trained facial feature extraction model, and the pre-trained facial feature extraction model can perform feature extraction on the image frames to extract the mouth features corresponding to each face image list. The above process can generate a mouth feature list for each face image.
[0048] S104: confirm the synchronization result of the video segment according to the audio data and the mouth features.
[0049] In the embodiments of the present disclosure, the lip-sound similarity corresponding to the image list can be obtained based on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list. Based on the lip-sound similarity corresponding to the image list, the video segment can be subjected to audio-visual synchronization detection to confirm the synchronization result of the video segment.
[0050] Optionally, a similarity threshold may be preset, a statistical number of image lists having lip sound similarity greater than or equal to the similarity threshold may be obtained, and audio-visual synchronization detection may be performed on the video according to the statistical number.
[0051] For example, a quantity threshold may be preset, and in response to the statistical quantity being greater than or equal to the quantity threshold, it is determined that the audio and video of the video segment are synchronized; in response to the statistical quantity being less than the quantity threshold, it is determined that the audio and video of the video segment are not synchronized.
[0052] The audio and video synchronization detection method proposed in the present disclosure extracts image data and audio data of a video segment of a target length, performs face detection and tracking based on the extracted image data to divide the segment into multiple face image lists, and extracts mouth features corresponding to each face image list based on the traversal results of each face image list, wherein the mouth features are used to characterize the changes in the lips, and the synchronization results of the video segment are confirmed based on the audio data and the mouth features. Therefore, the present disclosure solves the problem of performing audio and video synchronization detection in a scene with multiple characters, performs audio and video synchronization detection on the segment through audio data and mouth features, and improves the accuracy and reliability of obtaining the audio and video synchronization detection results of the video segment.
[0053] Figure 2 It is a flowchart of the audio-video synchronization detection method according to the second embodiment of the present disclosure.
[0054] like Figure 2 As shown, in Figure 2 Based on the illustrated embodiment, the audio and video synchronization detection method of the embodiment of the present disclosure may specifically include the following steps:
[0055] The step S101 of “extracting image data and audio data of a video segment of target length” in the above embodiment may specifically include the following steps S201 and S202 .
[0056] S201, extracting audio frames in a video segment of a target length, and generating an audio data list according to the audio frames.
[0057] Optionally, an audio extraction tool may be used to extract audio frames {A1, A2, ...A n}, and after obtaining the audio frame, store the audio frame in the audio data list {A} to generate an audio data list.
[0058] S202, extracting image frames within a video segment of a target length, and generating an image data list according to the image frames.
[0059] It should be noted that, in order to ensure the synchronization of the image frames and the audio frames on the time axis, the time interval for extracting the image frames is the same as the time interval for extracting the audio frames.
[0060] Optionally, a video frame extraction tool can be used to extract image frames {L1, L2, ...L n}, and after acquiring the video frame, store the video frame in the image data list {L} to generate the image data list.
[0061] In the above embodiment, step S102 of “performing face detection and tracking according to the extracted image data to divide into a plurality of face image lists” may specifically include the following steps S203 and S205 .
[0062] S203, detecting image frames in the image data list to obtain one or more face images in which the face areas of the image frames are greater than a preset threshold.
[0063] It should be noted that the present disclosure does not limit the specific method of detecting the image frames in the image data list, and it can be selected according to actual conditions.
[0064] Optionally, the image frames in the image list can be input into a pre-trained face detection model in sequence, and the face detection model performs face detection on the image frames to obtain face detection results, and obtains one or more face images with a face area greater than a preset threshold from the face detection results.
[0065] Among them, the preset threshold S a That is, the face detection area threshold. The present disclosure does not limit the set value and it can be set according to actual conditions.
[0066] For example, for image frame A t1 , the image frame A can be t1 Input the pre-trained face detection model, and the face detection model detects the image frame A t1 Perform face detection, obtain face detection results, and determine image frame A from the face detection results t1Contains face 1, face 2, and face 3. The face detection area of face 1 is S1, the face detection area of face 2 is S2, and the face detection area of face 3 is S3. a 、S2>S a 、S3>S a , then image frame A t1 The candidate faces are face 2 and face 3.
[0067] S204: Track the face image in each image frame according to the face features, and obtain one or more face identifiers.
[0068] In the disclosed embodiment, after acquiring one or more facial images, facial features can be extracted from the facial images to acquire facial features of the facial images, the facial images in each image frame can be tracked based on the facial features, and one or more facial identifications (Identity Document, ID for short) can be acquired.
[0069] Optionally, after the face identifier is acquired, the face identifier may be marked in the image frame.
[0070] For example, for image frame L1, if the candidate human faces of image frame L1 are face 2 and face 3, the facial identifier of face 2 is ID2 and the facial identifier of face 3 is ID3, then ID2 and ID3 are marked in image frame L1; for image frame L2, the candidate human faces of image frame L2 are face 3 and face 4, the facial identifier of face 3 is ID3 and the facial identifier of face 4 is ID4, then ID3 and ID4 are marked in image frame L2.
[0071] S205: Generate one or more face lists according to the face image and the face identifier.
[0072] In the embodiment of the present disclosure, after the face identification is obtained, the face image can be divided according to the face identification to generate one or more face lists.
[0073] S206: extracting the mouth features corresponding to each face image list according to the traversal result of each face image list.
[0074] For the relevant contents of step S206, please refer to the above embodiment, which will not be repeated here.
[0075] In the above embodiment, step S104 “confirming the synchronization result of the video segment according to the audio data and the mouth features” may specifically include the following steps S207 and S208 .
[0076] S207, obtaining the lip sound similarity corresponding to the image list according to the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list.
[0077] Optionally, the mouth key points may be extracted from the mouth feature image in the mouth feature list, and the mouth key points may be tracked to obtain the movement trajectory of the key points, and whether there is an opening and closing change of the mouth may be determined based on the movement trajectory.
[0078] In the disclosed embodiment, lip movement feature extraction may be performed on the mouth feature list to determine the mouth feature list containing mouth features with opening and closing changes.
[0079] Optionally, the mouth feature list may be input into a pre-trained lip movement detection model, and the lip movement detection model may extract the lip movement features to obtain a mouth feature list of mouth features with opening and closing changes.
[0080] In the disclosed embodiment, the audio frames in the audio data list may be sorted according to their timestamps to obtain an audio frame sequence, and audio features may be extracted from the audio frame sequence to obtain an audio feature sequence.
[0081] For example, after obtaining the audio frames {A1, A2, ...A n}, the audio frames can be sorted according to the timestamps of the audio frames to obtain the audio frame sequence {A t1 , A i2 , …A tn}, extract the audio features of the audio frame sequence to obtain the audio feature sequence {A′ t1 , A′ t2 …A′ tn}.
[0082] In an embodiment of the present disclosure, a mouth feature list of mouth features with opening and closing changes and an audio feature sequence are input into a pre-trained lip synchronization detection model, and the lip synchronization detection model performs cross-modal similarity calculation on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence to obtain the lip similarity corresponding to the image list.
[0083] S208: Performing audio-video synchronization detection on the video segment according to the lip-sound similarity corresponding to the image list to confirm the synchronization result of the video segment.
[0084] like Figure 3 As shown, step S208 in the above embodiment, "performing audio-visual synchronization detection on the video segment according to the lip sound similarity corresponding to the image list to confirm the synchronization result of the video segment" may specifically include the following steps S301 and S304.
[0085] S301: Determine a preset similarity threshold.
[0086] It should be noted that the present disclosure does not limit the setting of the similarity threshold, which can be set according to actual conditions.
[0087] S302, identifying whether the lip sound similarity corresponding to the image list is greater than a similarity threshold.
[0088] S303: Obtain the statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold.
[0089] In the embodiment of the present disclosure, after obtaining the lip sound similarity corresponding to the image list and the preset similarity threshold, it is possible to identify whether the lip sound similarity corresponding to the image list is greater than the similarity threshold, and based on the recognition result, obtain the statistical number of image lists whose lip sound similarity is greater than or equal to the similarity threshold.
[0090] S304: Perform audio and video synchronization detection on the video segment according to the statistical quantity to confirm the synchronization result of the video segment.
[0091] In the embodiment of the present disclosure, a preset quantity threshold may be determined, and audio and video synchronization detection may be performed on the video based on the statistical quantity and the preset quantity threshold.
[0092] It should be noted that the present disclosure does not limit the setting of the preset quantity threshold N. For example, the quantity threshold N can be preset to 1.
[0093] Optionally, in response to the statistical quantity being greater than or equal to a quantity threshold, it is determined that the video audio and picture are synchronized.
[0094] For example, if the statistical quantity n is 2 and the preset quantity threshold N is 1, that is, the statistical quantity is greater than the quantity threshold, then it is determined that the video audio and picture are synchronized.
[0095] Optionally, in response to the statistical quantity being less than a quantity threshold, it is determined that the video audio and picture are not synchronized.
[0096] For example, if the statistical quantity n is 0 and the preset quantity threshold N is 1, that is, the statistical quantity is less than the quantity threshold, it is determined that the video audio and picture are not synchronized.
[0097] In summary, the audio-visual synchronization detection method proposed in the present disclosure extracts audio frames in a video segment of a target length and generates an audio data list based on the audio frames, extracts audio frames in a video segment of a target length and generates an audio data list based on the audio frames, detects image frames in an image data list to obtain one or more face images in the image frame whose face area is greater than a preset threshold, tracks the face image in each image frame according to face features, and obtains one or more face identifiers, generates one or more face lists according to the face images and face identifiers, respectively, and generates a mouth feature list and a mouth feature list according to the mouth features with opening and closing changes. The audio feature sequence of the audio data list is used to obtain the lip sound similarity corresponding to the image list, and the audio and video synchronization detection is performed on the video segment according to the lip sound similarity corresponding to the image list to confirm the synchronization result of the video segment. Therefore, the present invention generates one or more face lists according to the face image and the face identification respectively, which solves the problem of audio and video asynchronous detection in a multi-person scenario, and performs audio and video synchronization detection on the video according to the lip sound similarity corresponding to the image list, avoiding the problem of missed detection in the audio and video synchronization detection process, improving the accuracy and reliability of the audio and video synchronization detection result of the video, and improving the user's experience of watching the video.
[0098] The specific process of the audio and video synchronization detection method proposed in the embodiment of the present disclosure is explained below.
[0099] For example, Figure 4 As shown, the specific process of the audio-visual synchronization detection method may include the following steps:
[0100] S401, obtaining a video segment with a video length of T, extracting image frames from the video at time intervals, storing them in an image list, and extracting audio frames from the video at time intervals, storing them in an audio list.
[0101] S402, performing face detection on the image frames in the image list, obtaining faces whose face detection area is larger than a set value, tracking the faces according to face features, grouping the image frames according to face identifiers, and obtaining multiple image lists.
[0102] S403, traversing each image list to determine a mouth area picture sequence corresponding to each image list.
[0103] S404, determining whether the mouth is in a lip movement state.
[0104] It should be noted that the mouth key points can be extracted from the mouth area images in the mouth area image sequence, and the mouth key points can be tracked to obtain the movement trajectory of the key points. In response to the movement trajectory indicating that the mouth has opened and closed changes, it is determined that the mouth is in a lip movement state.
[0105] It should be noted that by judging whether the mouth opens and closes, that is, whether the person is speaking, and determining whether the mouth is in lip movement state before executing subsequent steps, it is suitable for the scenario of synchronous audio and video detection of multiple characters.
[0106] If it is determined that the mouth is opening and closing, that is, the mouth is in a lip movement state, step S405 is executed.
[0107] If it is determined that there is no opening or closing of the mouth, that is, the mouth is not in a lip movement state, step S403 is executed.
[0108] S405, extract lip movement features from the mouth area pictures corresponding to the image list to obtain a lip movement feature sequence corresponding to the image list, input the lip movement feature sequence and the audio feature sequence corresponding to the image list into a pre-trained lip-sound synchronization detection model, and use the lip-sound synchronization detection model to perform cross-modal similarity calculation on the lip movement feature sequence and the audio feature sequence to obtain the lip-sound similarity corresponding to the image list.
[0109] It should be noted that the lip synchronization detection model is used to calculate the cross-modal similarity of the lip movement feature sequence and the audio feature sequence, and the audio-visual synchronization detection is realized according to the lip synchronization similarity, thereby improving the accuracy of the audio-visual synchronization detection results.
[0110] S406, performing audio and video synchronization detection on the video according to the lip sound similarity corresponding to the image list, that is, determining whether the audio and video of the video are synchronized.
[0111] S407: In response to the fact that the statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold is greater than or equal to the number threshold, it is determined that the video audio and picture are synchronized.
[0112] S408: In response to the statistical number of the image list having lip sound similarity greater than or equal to the similarity threshold being less than the number threshold, it is determined that the audio and picture of the video are not synchronized.
[0113] To sum up, the audio and video synchronization detection method proposed in the present invention groups image frames by face identification to obtain multiple image lists, which solves the problem of audio and video asynchronous detection in multi-person scenarios, and performs audio and video synchronization detection on the video according to the lip sound similarity corresponding to the image list, thereby avoiding the problem of missed detection in the audio and video synchronization detection process, and preventing the problem of secondary detection due to missed detection in the audio and video synchronization detection process, reducing the cost in the audio and video synchronization detection process, improving the efficiency of audio and video synchronization detection, and at the same time ensuring the accuracy and reliability of the audio and video synchronization detection results of the video, thereby improving the user's experience of watching the video.
[0114] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0115] According to an embodiment of the present disclosure, the present disclosure further provides an audio-video synchronization detection device, which is used to implement the above-mentioned audio-video synchronization detection method.
[0116] Figure 5 It is a block diagram of an audio-video synchronization detection device according to an embodiment of the present disclosure.
[0117] like Figure 5 As shown, the audio-video synchronization detection device 500 includes: a first extraction module 501 , a division module 502 , a second extraction module 503 and a confirmation module 504 .
[0118] A first extraction module 501 is used to extract image data and audio data of a video segment of a target length;
[0119] A division module 502, used for performing face detection and tracking according to the extracted image data to divide it into a plurality of face image lists;
[0120] The second extraction module 503 is used to extract the mouth features corresponding to each face image list according to the traversal result of each face image list, wherein the mouth features are used to characterize the changes of lips;
[0121] The confirmation module 504 is used to confirm the synchronization result of the video segment according to the audio data and the mouth feature.
[0122] In one embodiment of the present disclosure, the first extraction module 501 is used to: extract audio frames within the video segment of the target length, and generate an audio data list based on the audio frames; extract image frames within the video segment of the target length, and generate an image data list based on the image frames.
[0123] In one embodiment of the present disclosure, the division module 502 is used to: detect the image frames in the image data list to obtain one or more face images in the image frames whose face areas are greater than a preset threshold; track the face images in each image frame according to face features and obtain one or more face identifiers; and generate one or more face lists according to the face images and the face identifiers.
[0124] In one embodiment of the present disclosure, the second extraction module 503 is used to: traverse the face images in each face list and extract the mouth features of each face image; and generate a mouth feature list for each face image based on the mouth features.
[0125] In one embodiment of the present disclosure, the confirmation module 504 is used to: obtain the lip-sound similarity corresponding to the image list based on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list; and perform audio-visual synchronization detection on the video segment based on the lip-sound similarity corresponding to the image list to confirm the synchronization result of the video segment.
[0126] In one embodiment of the present disclosure, the process of determining a mouth feature list of mouth features with opening and closing changes includes: performing lip movement feature extraction on the mouth feature list to determine the mouth feature list of mouth features with opening and closing changes.
[0127] In one embodiment of the present disclosure, the process of determining the audio feature sequence includes: sorting the audio frames according to the timestamps of the audio frames in the audio data list to obtain an audio frame sequence; and extracting audio features from the audio frame sequence to obtain the audio feature sequence.
[0128] In one embodiment of the present disclosure, the confirmation module 504 is used to: input the mouth feature list of mouth features with opening and closing changes and the audio feature sequence into a pre-trained lip synchronization detection model; and the lip synchronization detection model performs cross-modal similarity calculation on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence to obtain the lip similarity corresponding to the image list.
[0129] In one embodiment of the present disclosure, the confirmation module 504 is used to: determine a preset similarity threshold; identify whether the lip sound similarity corresponding to the image list is greater than the similarity threshold; obtain a statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold; and perform audio-visual synchronization detection on the video segment based on the statistical number to confirm the synchronization result of the video segment.
[0130] The audio-visual synchronization detection device proposed in the present disclosure extracts image data and audio data of a video segment of a target length, performs face detection and tracking based on the extracted image data to divide the video segment into multiple face image lists, and extracts mouth features corresponding to each face image list based on the traversal results of each face image list, wherein the mouth features are used to characterize the changes in the lips, and the synchronization results of the video segment are confirmed based on the audio data and the mouth features. Therefore, the present disclosure solves the problem of performing audio-visual synchronization detection in a scene with multiple people, performs audio-visual synchronization detection on the video segment through audio data and mouth features, and improves the accuracy and reliability of obtaining the audio-visual synchronization detection results of the video segment.
[0131] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a terminal and a computer program product.
[0132] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0133] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0134] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0135] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the audio-visual synchronization detection method. For example, in some embodiments, the audio-visual synchronization detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the audio-visual synchronization detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the audio-visual synchronization detection method in any other appropriate manner (e.g., by means of firmware).
[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0137] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0139] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).
[0140] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0141] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0142] According to an embodiment of the present disclosure, the present disclosure also provides a terminal, such as Figure 7 As shown, the terminal 700 includes the above-mentioned electronic device 600.
[0143] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the audio-visual synchronization detection method described in the above embodiment of the present disclosure are implemented.
[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0145] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for detecting synchronization between audio and video, in, The method comprises: Extracting image data and audio data of a video segment of target length; Performing face detection and tracking according to the extracted image data to divide it into a plurality of face image lists; Extracting mouth features corresponding to each face image list according to the traversal result of each face image list, wherein the mouth features are used to characterize changes in lips; A synchronization result of the video segment is confirmed according to the audio data and the mouth feature.
2. The method according to claim 1, in, The step of extracting image data and audio data of a video segment of a target length includes: Extracting audio frames in the video segment of the target length, and generating an audio data list according to the audio frames; Image frames within the video segment of the target length are extracted, and an image data list is generated according to the image frames.
3. The method according to claim 2, in, The method of performing face detection and tracking based on the extracted image data to divide the image data into multiple face image lists includes: Detecting the image frames in the image data list to obtain one or more face images in the image frames whose face areas are greater than a preset threshold; Tracking the face image in each image frame according to the face features, and obtaining one or more face identifiers; One or more face lists are generated according to the face image and the face identifier.
4. The method according to claim 3, in, The extracting the mouth features corresponding to each face image list according to the traversal result of each face image list includes: Traverse the face images in each face list and extract the mouth features of each face image; A mouth feature list of each face image is generated according to the mouth features.
5. The method according to claim 4, in, The step of confirming the synchronization result of the video segment according to the audio data and the mouth feature includes: Acquire the lip sound similarity corresponding to the image list according to the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list; According to the lip sound similarity corresponding to the image list, the video segment is subjected to audio and video synchronization detection to confirm the synchronization result of the video segment.
6. The method according to claim 5, in, The process of determining a mouth feature list of mouth features with opening and closing changes includes: Lip movement features are extracted from the mouth feature list to determine the mouth feature list of mouth features with opening and closing changes.
7. The method according to claim 5, in, The process of determining the audio feature sequence includes: Sort the audio frames in the audio data list according to the timestamps of the audio frames to obtain an audio frame sequence; Audio features are extracted from the audio frame sequence to obtain the audio feature sequence.
8. The method according to claim 5, in, The obtaining the lip sound similarity corresponding to the image list according to the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list includes: Inputting the mouth feature list of mouth features with opening and closing changes and the audio feature sequence into a pre-trained lip synchronization detection model; The lip-sync detection model performs cross-modal similarity calculation on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence to obtain the lip-sync similarity corresponding to the image list.
9. The method according to claim 5, in, The performing audio-video synchronization detection on the video segment according to the lip sound similarity corresponding to the image list to confirm the synchronization result of the video segment includes: Determining a preset similarity threshold; Identify whether the lip sound similarity corresponding to the image list is greater than the similarity threshold; Obtaining a statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold; According to the statistical quantity, audio and video synchronization detection is performed on the video segment to confirm the synchronization result of the video segment.
10. The method according to claim 6, in, The performing audio and video synchronization detection on the video segment according to the statistical quantity to confirm the synchronization result of the video segment includes: Determining a preset quantity threshold; In response to the statistical quantity being greater than or equal to the quantity threshold, determining that the audio and video of the video segment are synchronized; In response to the statistical quantity being less than the quantity threshold, it is determined that the audio and picture of the video segment are not synchronized.
11. The method according to claim 6, in, Before determining the audio feature sequence, the method further includes: Extracting mouth key points from the mouth feature image in the mouth feature list, and tracking the mouth key points to obtain movement trajectories of the key points; According to the movement trajectory, determine whether there is any opening and closing change of the mouth.
12. An audio and video synchronization detection device, in, The device comprises: A first extraction module, used for extracting image data and audio data of a video segment of a target length; A division module, used for performing face detection and tracking according to the extracted image data, so as to divide the image data into a plurality of face image lists; A second extraction module is used to extract mouth features corresponding to each face image list according to the traversal result of each face image list, wherein the mouth features are used to characterize the changes of lips; A confirmation module is used to confirm the synchronization result of the video segment according to the audio data and the mouth feature.
13. The device according to claim 12, in, The extraction module is used to: Extracting audio frames in the video segment of the target length, and generating an audio data list according to the audio frames; Image frames within the video segment of the target length are extracted, and an image data list is generated according to the image frames.
14. The device according to claim 13, in, The partitioning module is used for: Detecting the image frames in the image data list to obtain one or more face images in the image frames whose face areas are greater than a preset threshold; Tracking the face image in each image frame according to the face features, and obtaining one or more face identifiers; One or more face lists are generated according to the face image and the face identifier.
15. The device according to claim 14, in, The second extraction module is used to: Traverse the face images in each face list and extract the mouth features of each face image; A mouth feature list of each face image is generated according to the mouth features.
16. The device according to claim 15, in, The confirmation module is used to: Acquire the lip sound similarity corresponding to the image list according to the mouth feature list of mouth features with opening and closing changes and the audio feature sequence of the audio data list; According to the lip sound similarity corresponding to the image list, the video segment is subjected to audio and video synchronization detection to confirm the synchronization result of the video segment.
17. The device according to claim 15, in, The process of determining a mouth feature list of mouth features with opening and closing changes includes: Lip movement features are extracted from the mouth feature list to determine the mouth feature list of mouth features with opening and closing changes.
18. The device according to claim 16, in, The process of determining the audio feature sequence includes: Sort the audio frames in the audio data list according to the timestamps of the audio frames to obtain an audio frame sequence; Audio features are extracted from the audio frame sequence to obtain the audio feature sequence.
19. The device according to claim 16, in, The confirmation module is used to: Inputting the mouth feature list of mouth features with opening and closing changes and the audio feature sequence into a pre-trained lip synchronization detection model; The lip-sync detection model performs cross-modal similarity calculation on the mouth feature list of mouth features with opening and closing changes and the audio feature sequence to obtain the lip-sync similarity corresponding to the image list.
20. The device according to claim 16, in, The confirmation module is used to: Determining a preset similarity threshold; Identify whether the lip sound similarity corresponding to the image list is greater than the similarity threshold; Obtaining a statistical number of the image list whose lip sound similarity is greater than or equal to the similarity threshold; According to the statistical quantity, audio and video synchronization detection is performed on the video segment to confirm the synchronization result of the video segment.
21. An electronic device, It is characterized in that including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 11.
22. A terminal, It is characterized in that Comprising the electronic device as claimed in claim 21.
23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
Citation Information
Cited By
Live broadcast abnormal behavior detection method and system
CN121415316A
A method and system for detecting abnormal behavior in live streaming
CN121415316B
