Lip sound synchronous detection method and device, computer equipment and storage medium
By performing time stamp alignment and segmentation of audio and video, and using short- and long-term models to extract features, the misjudgment problem of lip synchronous detection in the prior art under network delay and frame drops is solved, and the accuracy of detection is improved and robustness.
Patent Information
- Application Number
- CN202411084949.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-12
AI Technical Summary
The existing lip synchronous detection algorithm faces frequent misjudgment in network delay and frame dropping, and it is difficult to accurately detect lip synchronous in network video environments, resulting in low efficiency and high misjudgment rate.
By analyzing the video timestamps and audio timestamps of audio and videos, the audio and videos are divided into short segments, the video and audio features are extracted using the short-time model, and the lip sound synchronization detection is performed through the long-time model by integrating cross-modal long-time context information through the long-time model. .
提高了唇音同步检测的准确性和鲁棒性,有效应对网络延时和丢帧导致的同步问题,减少了音视频错位,提高了检测的细粒度和长时间内的同步关系捕捉能力。
Smart Images

Figure CN118945393A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a lip synchronization detection method, device, computer equipment and storage medium. Background Art
[0002] In dual-recording scenarios related to finance and insurance businesses, the non-compliant behavior of bystander answering calls mainly relies on manual viewing and listening to videos to judge. However, this method is inefficient, easily causes eye fatigue, and is difficult to cope with large-scale data screening. In addition, there are some lip detection algorithms to detect the synchronization of the speaker's lips and voice, thereby realizing automatic detection of bystander answering calls. However, the existing lip synchronization detection algorithms face huge challenges when problems such as network delay and frame loss occur, resulting in frequent misjudgments. Especially in the network video environment, the audio and video synchronization problem is more complicated, and there may be problems including video freezes caused by network delays, timestamp imbalance caused by variable frame rate transmission, screen distortion and block effects caused by network frame loss, and misalignment of the audio and video itself. These problems will make it difficult to accurately detect lip synchronization. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a lip synchronization detection method, apparatus, computer equipment and storage medium to solve the problem of low accuracy of lip synchronization detection.
[0004] In order to solve the above technical problems, the embodiment of the present application provides a lip synchronization detection method, which adopts the following technical solution:
[0005] Get the audio and video to be detected;
[0006] Parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp;
[0007] Splitting the audio and video into multiple short clips;
[0008] Input each short segment into a lip sync detection model, and output the video features and audio features of each short segment through a short-time model in the lip sync detection model;
[0009] Inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video;
[0010] Lip synchronization detection is performed according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0011] In order to solve the above technical problems, the embodiment of the present application further provides a lip synchronization detection device, which adopts the following technical solution:
[0012] An audio and video acquisition module, used to acquire the audio and video to be detected;
[0013] An audio and video alignment module, used for parsing the video timestamp and the audio timestamp of the audio and video, so as to align the video and audio in the audio and video according to the video timestamp and the audio timestamp;
[0014] An audio and video segmentation module, used to segment the audio and video into multiple short segments;
[0015] A short-time processing module, used for inputting each short segment into a lip-sync detection model, and outputting video features and audio features of each short segment through the short-time model in the lip-sync detection model;
[0016] A long-term processing module, used for inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video;
[0017] The synchronization detection module is used to perform lip synchronization detection according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0018] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0019] Get the audio and video to be detected;
[0020] Parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp;
[0021] Splitting the audio and video into multiple short clips;
[0022] Input each short segment into a lip sync detection model, and output the video features and audio features of each short segment through a short-time model in the lip sync detection model;
[0023] Inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video;
[0024] Lip synchronization detection is performed according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0026] Get the audio and video to be detected;
[0027] Parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp;
[0028] Splitting the audio and video into multiple short clips;
[0029] Input each short segment into a lip sync detection model, and output the video features and audio features of each short segment through a short-time model in the lip sync detection model;
[0030] Inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video;
[0031] Lip synchronization detection is performed according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0032] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: parsing the video timestamp and audio timestamp of the audio and video, aligning the video and audio in the audio and video according to the video timestamp and audio timestamp, and reducing the audio and video misalignment problem caused by timestamp mismatch; dividing the audio and video into multiple short clips, and accurately extracting the video features and audio features of each short clip through the short-time model in the lip synchronization detection model, thereby ensuring the accuracy of lip synchronization detection from a fine-grained level; inputting the video features and audio features of each short clip into the long-time model in the lip synchronization detection model, the long-time model integrates the information of multiple short clips, captures the synchronization relationship and interaction of audio and video over a long period of time, and obtains cross-modal long-term context information, which can effectively deal with synchronization problems caused by network delays, frame loss, etc., thereby improving the robustness and accuracy of lip synchronization detection based on cross-modal long-term context information. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0034] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0035] Figure 2 is a flow chart of an embodiment of a lip synchronization detection method according to the present application;
[0036] Figure 3 is a structural schematic diagram of an embodiment of a lip synchronization detection device according to the present application;
[0037] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0039] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0041] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0042] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0043] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0044] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0045] It should be noted that the lip synchronization detection method provided in the embodiment of the present application is generally executed by a server, and accordingly, the lip synchronization detection device is generally arranged in the server.
[0046] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0047] Continue to refer Figure 2 , shows a flow chart of an embodiment of a lip sync detection method according to the present application. The lip sync detection method comprises the following steps:
[0048] Step S201, obtaining the audio and video to be detected.
[0049] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.
[0050] Specifically, the audio and video to be tested are obtained. The audio and video files can be double audio and video (dual recording) files involved in financial and insurance related businesses to ensure transaction transparency and compliance. When customers handle related businesses, it is often necessary to record the audio and video of the customers simultaneously to ensure that the customers fully understand and agree to all terms and conditions to prevent disputes.
[0051] Step S202: parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp.
[0052] Specifically, the audio and video are parsed to extract the video timestamp of the video in the audio and video and the audio timestamp of the audio in the audio and video, where the video timestamp and the audio timestamp respectively reflect the time position of the video frame and the audio frame. The video timestamp and the audio timestamp are aligned to ensure the synchronization of the video and the audio.
[0053] Step S203, dividing the audio and video into multiple short segments.
[0054] Specifically, after aligning the audio and video, they are segmented into multiple short segments, which can be small segments of several hundred milliseconds in length.
[0055] Step S204: input each short segment into a lip synchronization detection model, and output the video features and audio features of each short segment through a short-time model in the lip synchronization detection model.
[0056] Specifically, the present application detects lip synchronization through a lip synchronization detection model. The lip synchronization detection model can be a composite model, which can include a short-time model and a long-time model. Among them, the short-time model is used to process short segments, and is processed from a shorter time dimension. The long-time model is processed based on the output of the short-time model, and is processed from a longer time dimension.
[0057] The short-term model contains independent video encoders and audio encoders. The video encoder encodes the video in the short clip to obtain video features, and the audio encoder encodes the audio in the short clip to obtain audio features.
[0058] Step S205 , input the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video.
[0059] Specifically, the video and audio features of each short clip extracted by the short-term model are input into the long-term model. The long-term model combines the features of multiple short clips to generate cross-modal long-term context information of audio and video. Cross-modal long-term context information integrates and associates feature information from different modalities (audio and video) over a long period of time to capture the temporal relationship and interaction between them; it takes into account the features of audio and video in short clips, and also combines the continuity and correlation of these short clips in time series, thereby providing a richer and more comprehensive basis for synchronization detection, and can more accurately judge the synchronization of audio and video.
[0060] In one embodiment, the long-term model receives the video features and audio features of each short clip, and models the changes and associations of these features in time series, such as using a recurrent neural network (RNN), a long short-term memory network (LSTM), or a transformer, which can effectively capture temporal dependencies over a long period of time. The video features and audio features are then fused to establish a cross-modal relationship between them. The long-term model not only processes the features of the current short clip, but also considers the information of the previous and subsequent short clips. This integration of contextual information enables the model to recognize more complex patterns, such as persistent audio and video asynchrony, rather than just instantaneous synchronization or asynchrony.
[0061] Step S206: performing lip synchronization detection according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0062] Specifically, the long-term model performs lip sync detection based on cross-modal long-term context information. The model determines whether the video features and audio features remain consistent over a long period of time, thereby determining whether the lip movement in the video is synchronized with the sound in the audio. The lip sync detection result can show whether the lips are synchronized or not.
[0063] In this embodiment, the video timestamp and audio timestamp of the audio and video are parsed, and the video and audio in the audio and video are aligned according to the video timestamp and audio timestamp to reduce the audio and video misalignment problem caused by timestamp mismatch; the audio and video are divided into multiple short clips, and the video features and audio features of each short clip are accurately extracted through the short-time model in the lip synchronization detection model, thereby ensuring the accuracy of lip synchronization detection from a fine-grained level; the video features and audio features of each short clip are input into the long-time model in the lip synchronization detection model, and the long-time model integrates the information of multiple short clips, captures the synchronization relationship and interaction of audio and video over a long period of time, and obtains cross-modal long-term context information, which can effectively deal with synchronization problems caused by network delays, frame loss, etc., thereby improving the robustness and accuracy of lip synchronization detection based on cross-modal long-term context information.
[0064] Furthermore, the above step S202 may include: extracting the video timestamp of the video in the audio and video, and the audio timestamp of the audio in the audio and video through ffmpeg; comparing the video timestamp and the audio timestamp to obtain the asynchronization information of the video and audio; adjusting the video and audio according to the asynchronization information to align the video timestamp and the audio timestamp.
[0065] Specifically, the audio and video are parsed by the ffmpeg tool, and the video timestamp of each video frame and the audio timestamp of each audio frame in the audio and video are extracted respectively. ffmpeg is an open source multimedia framework for processing audio, video and other multimedia files and streams. Timestamps are key metadata in audio and video, recording the capture or playback time of each frame. The timestamp of a video frame can indicate the position of each frame in the video (such as the frame at the 10th second), and the timestamp of an audio frame can indicate the position of the corresponding audio data on the timeline.
[0066] The video timestamp and the audio timestamp are compared, specifically, the video timestamp of each video frame is matched with the audio timestamp of the corresponding audio frame to find out the time difference between them; if there is a significant difference between the video timestamp and the audio timestamp, it means that the audio and video are not synchronized, and then the video and audio asynchrony information is generated.
[0067] The out-of-sync information includes the time difference or offset between the video and audio, indicating the specific out-of-sync situation of the video and audio, such as how many milliseconds the video may be ahead of or behind the audio. According to the out-of-sync information, the video and audio are adjusted accordingly to align the video timestamp and the audio timestamp. The adjustment methods may include: increasing or decreasing the timestamp of the video frame to align it with the audio timestamp; adjusting the audio playback speed or adding audio frames to align it with the video timestamp; inserting video frames or deleting redundant frames to ensure that the video and audio are synchronized on the timeline.
[0068] In this embodiment, ffmpeg is used to extract the video timestamp of the video in the audio and video, and the audio timestamp of the audio in the audio and video; the video timestamp and the audio timestamp are compared to obtain the asynchrony information of the video and the audio, which can effectively identify the slight time deviation between the audio and the video; the video and the audio are adjusted according to the asynchrony information, and the video timestamp and the audio timestamp are aligned, which can correct the time deviation between the audio and the video, ensure that the data input into the lip synchronization detection model is synchronized, can effectively deal with various abnormal situations in network transmission, and ensure the accuracy of subsequent model detection.
[0069] Furthermore, after the above-mentioned step of obtaining the lip synchronization detection result, it can also include: when it is determined that lip synchronization asynchrony exists according to the lip synchronization detection result, for each short segment, calculating the similarity between the video features of the short segment and the audio features of all the short segments; selecting the audio features of the short segment corresponding to the highest similarity, and calculating the offset of the short segment according to the video features of the short segment and the timestamp of the audio features of the selected short segment; generating audio and video offset information according to the offset of each short segment, and the offset information is used to update the lip synchronization detection result.
[0070] Specifically, if the lip synchronization detection result shows that there is an asynchronous phenomenon between the audio and video, that is, the lip sounds are not synchronized, the similarity between the video features of each short segment and the audio features of all short segments is calculated, for example, the similarity is calculated by cosine similarity or Euclidean distance.
[0071] The audio feature of the short clip corresponding to the highest similarity is selected. The audio feature with the highest similarity represents the audio that is most likely to be synchronized with the video feature of the short clip. Then, the offset of the short clip is calculated based on the timestamp of the selected audio feature and the video feature of the short clip. The offset represents the degree of misalignment of the video and audio on the timeline.
[0072] After the above processing is performed on all short segments, the offset information of the entire audio and video is generated according to the offset of each short segment. The offset information may include the specific offset of each short segment, or may include the comprehensive offset of the entire audio and video. The comprehensive offset may be the average value of the offsets of each short segment.
[0073] The offset information can be used to update the lip synchronization detection result. The updated detection result can display the offset of each short segment and the overall offset, thereby more accurately reflecting the synchronization status of the audio and video.
[0074] In one embodiment, the offset and / or the comprehensive offset may also be used to adjust and correct the audio and video to ensure synchronization.
[0075] In this embodiment, when lip synchronization is present, the similarity between the video features of each short clip and the audio features of all the short clips is calculated; the audio features of the short clip corresponding to the highest similarity are selected, and the offset of the short clip is calculated based on the video features of the short clip and the timestamp of the audio features of the selected short clip, so as to accurately evaluate the synchronization of the short clips; the offset information of the audio and video is generated based on the offset of each short clip, and then the lip synchronization detection result is updated to reflect the synchronization of the entire audio and video in more detail.
[0076] Furthermore, before the above-mentioned step S201, it may also include: obtaining the first audio and video for model training; aligning the video and audio in the first audio and video; dividing the aligned first audio and video into multiple first short clips; inputting each first short clip into an initial lip synchronization detection model, and outputting the video features, audio features and offset of each first short clip through the short-time model in the initial lip synchronization detection model; inputting the video features and audio features of each first short clip into the long-time model in the initial lip synchronization detection model to obtain a lip synchronization prediction result; calculating the model loss based on the offset of each first short clip and the lip synchronization prediction result, and adjusting the initial lip synchronization detection model according to the model loss until the model loss meets the training stop condition, thereby obtaining the lip synchronization detection model.
[0077] Specifically, the present application requires generating a lip sync detection model through model training in advance. Obtain a first audio and video for model training, the first audio and video is a labeled sample, and the label can show whether the audio and video of the first audio and video are synchronized. For unsynchronized segments, the label can also show the offset between the audio and video.
[0078] The video and audio in the first audio and video are aligned, and the time axes of the two are adjusted by parsing the timestamps of the video and audio to ensure accuracy in subsequent processing. The alignment process can refer to the description related to step S202 above, which will not be repeated here. The aligned first audio and video are divided into multiple short clips, and each short clip is input into the initial lip synchronization detection model. The initial lip synchronization detection model includes a short-time model and a long-time model. The short-time model first processes these short clips to extract their video features, audio features and offsets. The extracted video features and audio features are then input into the long-time model for further processing, and the long-time model will output the lip synchronization prediction results.
[0079] Based on the offset of the short-term model and the prediction results of the long-term model, the model loss can be calculated. The model loss is the weighted sum of multiple loss functions, which can include synchronization loss and time offset loss; synchronization loss represents the loss of lip synchronization prediction, which can be a binary cross entropy loss, calculated based on the lip synchronization prediction results; time offset loss represents the loss related to the offset, which can be measured by mean square error, calculated based on the offset of the short-term model.
[0080] According to the model loss, the parameters of the initial lip sync detection model are adjusted by optimization methods such as gradient descent. The training process is repeated until the loss function value reaches a preset training stop condition, and the final lip sync detection model is obtained, where the training stop condition can be that the model loss is less than a preset loss threshold.
[0081] In one embodiment, the (initial) lip sync detection model may also include a face quality detection model, which is used to detect faces in video frames, and to judge the quality of video frames containing faces, and to filter out video frames with poor face display. In this case, the loss function may also include face quality loss, which may be measured using cross entropy loss. At this time, the loss is obtained by weighting face quality loss, synchronization loss, and time offset loss.
[0082] In this embodiment, the video and audio in the first audio and video for training are aligned to ensure the data quality of the input model; the aligned first audio and video are divided into multiple first short clips; each first short clip is input into an initial lip synchronization detection model, and the video features, audio features and offset of each first short clip are output through the short-time model in the initial lip synchronization detection model; the video features and audio features of each first short clip are input into the long-time model in the initial lip synchronization detection model to obtain the lip synchronization prediction result; the model loss is calculated based on the offset of each first short clip and the lip synchronization prediction result, the feature processing and prediction quality of the model are comprehensively evaluated, and the initial lip synchronization detection model is adjusted according to the model loss, so as to obtain a high-accuracy lip synchronization detection model.
[0083] Furthermore, before the above-mentioned step of obtaining the first audio and video for model training, it can also include: obtaining the initial audio and video for model training; randomly adjusting the video frame rate of the initial audio and video to obtain the first audio and video.
[0084] Specifically, the initial audio and video are obtained. The initial audio and video can be unprocessed data collected from actual scenes, which may have different frame rates and qualities. The video frame rate of the initial audio and video is randomly adjusted to artificially introduce frame rate changes. For example, assuming that the expected video frame rate is 30 frames per second, the video frame rate of some clips can be reduced to 15 frames per second, or increased to 45 frames per second, so as to simulate various situations that may occur in the network environment, such as frame rate fluctuations, frame loss, etc., to ensure that the model adapts to different frame rates during training, and generate the first audio and video.
[0085] In this embodiment, initial audio and video for model training are obtained, the video frame rate of the initial audio and video is randomly adjusted, and frame rate changes are artificially introduced so that the model can better cope with the challenges brought by frame rate changes and improve the accuracy of lip synchronization detection.
[0086] Furthermore, after the above-mentioned step of obtaining the first audio and video for model training, it can also include: extracting the video timestamp of the video in the first audio and video through ffmpeg; when it is determined according to the video timestamp that there is a frame rate imbalance in the first audio and video, adjusting the video frame rate of the video in the first audio and video to the standard frame rate through an interpolation algorithm.
[0087] Specifically, the video timestamp of the video in the first audio and video is extracted through ffmpeg. The video timestamp indicates the playback time of each video frame and can be used to analyze the stability of the video frame rate. Compare the timestamps of each video to check whether there is a frame rate imbalance. Among them, frame rate imbalance refers to inconsistent time intervals between video frames, which may be caused by delays, frame losses, etc. in network transmission. If frame rate imbalance is detected, it indicates that the frame rate of the video fluctuates greatly at certain moments.
[0088] An interpolation algorithm (for example, a bilinear interpolation algorithm) can be used to adjust the video frame rate to a standard frame rate. The interpolation algorithm generates new frames between existing frames to make the time intervals between frames consistent, thereby restoring the smooth playback effect of the video. The adjusted video has a balanced frame rate, which can reduce the audio and video synchronization problem caused by uneven frame rates, thereby improving the training effect and detection accuracy of the lip synchronization detection model.
[0089] In this embodiment, the video timestamp of the video in the first audio and video is extracted; when it is determined that the first audio and video has a frame rate imbalance according to the video timestamp, the video frame rate of the video in the first audio and video is adjusted to the standard frame rate through an interpolation algorithm to eliminate false detection caused by inconsistent frame rates and improve the accuracy of subsequent lip synchronization detection.
[0090] Furthermore, the above-mentioned lip synchronization detection method can also include: inputting the first audio and video into a face quality detection model to obtain a weak face video frame in the first audio and video; deleting the weak face video frame from the first audio and video, and complementing the first audio and video through an interpolation algorithm; and randomly deleting video frames in the first audio and video, and complementing the first audio and video through an interpolation algorithm.
[0091] Specifically, the first audio and video may also be input into a face quality detection model in the initial lip synchronization detection model. The face quality detection model identifies weak face video frames by analyzing face features in video frames, where weak face video frames refer to video frames where face features are unclear due to blur, occlusion, poor lighting, etc.
[0092] The detected weak face video frame is deleted from the first audio and video. In order to maintain the continuity and frame rate consistency of the video, an interpolation algorithm is used to fill the first audio and video after deleting the weak face video frame. The interpolation algorithm generates a new frame at the position of the deleted frame to ensure smooth video playback and time continuity.
[0093] The present application can also randomly delete other video frames in the first audio and video (mask can be used to filter some video frames) to simulate the frame loss that may occur in the actual network environment. Subsequently, the same interpolation algorithm is used to fill in the first audio and video after the random deletion of the video frame to ensure the balance of the video frame rate and the smoothness of the playback.
[0094] During training, the performance of the model in different situations is enhanced by simulating various possibilities in actual application environments (such as frame rate changes, poor face quality, video frame loss, etc.).
[0095] In this embodiment, the first audio and video is input into a face quality detection model to obtain a weak face video frame in the first audio and video, where the weak face video frame refers to a video frame that contains a face but is not clear; the weak face video frame is deleted from the first audio and video, and the first audio and video is complemented by an interpolation algorithm; the video frames in the first audio and video can also be randomly deleted, and the first audio and video are complemented by an interpolation algorithm, simulating various possibilities in an actual application environment, and ensuring the temporal continuity and frame rate consistency of the video through the interpolation algorithm, thereby enhancing the prediction performance of the model under different situations.
[0096] It should be emphasized that in order to further ensure the privacy and security of the above audio and video, the above audio and video can also be stored in a blockchain node.
[0097] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0098] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0099] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0101] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0102] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a lip synchronization detection device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0103] like Figure 3 As shown, the lip synchronization detection device 300 described in this embodiment includes: an audio and video acquisition module 301, an audio and video alignment module 302, an audio and video segmentation module 303, a short-time processing module 304, a long-time processing module 305 and a synchronization detection module 306, wherein:
[0104] The audio and video acquisition module 301 is used to acquire the audio and video to be detected.
[0105] The audio and video alignment module 302 is used to parse the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp.
[0106] The audio and video segmentation module 303 is used to segment the audio and video into multiple short segments.
[0107] The short-time processing module 304 is used to input each short segment into the lip synchronization detection model, and output the video features and audio features of each short segment through the short-time model in the lip synchronization detection model.
[0108] The long-term processing module 305 is used to input the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of audio and video.
[0109] The synchronization detection module 306 is used to perform lip synchronization detection according to the cross-modal long-term context information to obtain a lip synchronization detection result.
[0110] In this embodiment, the video timestamp and audio timestamp of the audio and video are parsed, and the video and audio in the audio and video are aligned according to the video timestamp and audio timestamp to reduce the audio and video misalignment problem caused by timestamp mismatch; the audio and video are divided into multiple short clips, and the video features and audio features of each short clip are accurately extracted through the short-time model in the lip synchronization detection model, thereby ensuring the accuracy of lip synchronization detection from a fine-grained level; the video features and audio features of each short clip are input into the long-time model in the lip synchronization detection model, and the long-time model integrates the information of multiple short clips, captures the synchronization relationship and interaction of audio and video over a long period of time, and obtains cross-modal long-term context information, which can effectively deal with synchronization problems caused by network delays, frame loss, etc., thereby improving the robustness and accuracy of lip synchronization detection based on cross-modal long-term context information.
[0111] In some optional implementations of this embodiment, the audio and video alignment module 302 may include: a timestamp extraction submodule, a timestamp comparison submodule, and an audio and video adjustment submodule, wherein:
[0112] The timestamp extraction submodule is used to extract the video timestamp of the video in the audio and video, and the audio timestamp of the audio in the audio and video through ffmpeg.
[0113] The timestamp comparison submodule is used to compare the video timestamp and the audio timestamp to obtain the asynchrony information of the video and audio.
[0114] The audio and video adjustment submodule is used to adjust the video and audio according to the asynchronous information to align the video timestamp and the audio timestamp.
[0115] In this embodiment, ffmpeg is used to extract the video timestamp of the video in the audio and video, and the audio timestamp of the audio in the audio and video; the video timestamp and the audio timestamp are compared to obtain the asynchrony information of the video and the audio, which can effectively identify the slight time deviation between the audio and the video; the video and the audio are adjusted according to the asynchrony information, and the video timestamp and the audio timestamp are aligned, which can correct the time deviation between the audio and the video, ensure that the data input into the lip synchronization detection model is synchronized, can effectively deal with various abnormal situations in network transmission, and ensure the accuracy of subsequent model detection.
[0116] In some optional implementations of this embodiment, the lip synchronization detection device 300 may further include: a similarity calculation module, an offset calculation module and a result update module, wherein:
[0117] The similarity calculation module is used to calculate the similarity between the video features of each short segment and the audio features of all the short segments when it is determined that there is lip-sync asynchrony according to the lip-sync detection result.
[0118] The offset calculation module is used to select the audio feature of the short segment corresponding to the highest similarity, and calculate the offset of the short segment according to the video feature of the short segment and the timestamp of the audio feature of the selected short segment.
[0119] The result updating module is used to generate the offset information of the audio and video according to the offset of each short segment, and the offset information is used to update the lip synchronization detection result.
[0120] In this embodiment, when lip synchronization is present, the similarity between the video features of each short clip and the audio features of all the short clips is calculated; the audio features of the short clip corresponding to the highest similarity are selected, and the offset of the short clip is calculated based on the video features of the short clip and the timestamp of the audio features of the selected short clip, so as to accurately evaluate the synchronization of the short clips; the offset information of the audio and video is generated based on the offset of each short clip, and then the lip synchronization detection result is updated to reflect the synchronization of the entire audio and video in more detail.
[0121] In some optional implementations of this embodiment, the lip synchronization detection device 300 may further include: a training acquisition module, a first alignment module, a first segmentation module, a first input module, a synchronization prediction module, and a model adjustment module, wherein:
[0122] The training acquisition module is used to acquire the first audio and video for model training.
[0123] The first alignment module is used to align the video and audio in the first audio and video.
[0124] The first segmentation module is used to segment the aligned first audio and video into multiple first short segments.
[0125] The first input module is used to input each first short segment into an initial lip synchronization detection model, and output the video features, audio features and offset of each first short segment through a short-time model in the initial lip synchronization detection model.
[0126] The synchronization prediction module is used to input the video features and audio features of each first short clip into the long-term model in the initial lip synchronization detection model to obtain a lip synchronization prediction result.
[0127] The model adjustment module is used to calculate the model loss based on the offset of each first short segment and the lip synchronization prediction result, and adjust the initial lip synchronization detection model according to the model loss until the model loss meets the training stop condition to obtain the lip synchronization detection model.
[0128] In this embodiment, the video and audio in the first audio and video for training are aligned to ensure the data quality of the input model; the aligned first audio and video are divided into multiple first short clips; each first short clip is input into an initial lip synchronization detection model, and the video features, audio features and offset of each first short clip are output through the short-time model in the initial lip synchronization detection model; the video features and audio features of each first short clip are input into the long-time model in the initial lip synchronization detection model to obtain the lip synchronization prediction result; the model loss is calculated based on the offset of each first short clip and the lip synchronization prediction result, the feature processing and prediction quality of the model are comprehensively evaluated, and the initial lip synchronization detection model is adjusted according to the model loss, so as to obtain a high-accuracy lip synchronization detection model.
[0129] In some optional implementations of this embodiment, the lip synchronization detection device 300 may further include: an initial acquisition module and a frame rate adjustment module, wherein:
[0130] The initial acquisition module is used to obtain the initial audio and video for model training.
[0131] The frame rate adjustment module is used to randomly adjust the video frame rate of the initial audio and video to obtain the first audio and video.
[0132] In this embodiment, initial audio and video for model training are obtained, the video frame rate of the initial audio and video is randomly adjusted, and frame rate changes are artificially introduced so that the model can better cope with the challenges brought by frame rate changes and improve the accuracy of lip synchronization detection.
[0133] In some optional implementations of this embodiment, the lip synchronization detection device 300 may further include: an acquisition module and a frame rate adjustment module, wherein:
[0134] The acquisition module is used to extract the video timestamp of the video in the first audio and video through ffmpeg.
[0135] The frame rate adjustment module is used to adjust the video frame rate of the video in the first audio and video to a standard frame rate through an interpolation algorithm when it is determined that the first audio and video has a frame rate imbalance according to the video timestamp.
[0136] In this embodiment, the video timestamp of the video in the first audio and video is extracted; when it is determined that the first audio and video has a frame rate imbalance according to the video timestamp, the video frame rate of the video in the first audio and video is adjusted to the standard frame rate through an interpolation algorithm to eliminate false detection caused by inconsistent frame rates and improve the accuracy of subsequent lip synchronization detection.
[0137] In some optional implementations of this embodiment, the lip synchronization detection device 300 may further include: a face detection module, a deletion and filling module, and a random deletion module, wherein:
[0138] The face detection module is used to input the first audio and video into the face quality detection model to obtain a weak face video frame in the first audio and video.
[0139] The deletion and completion module is used to delete the weak face video frame from the first audio and video, and complete the first audio and video through an interpolation algorithm.
[0140] The random deletion module is used to randomly delete video frames in the first audio and video, and to complete the first audio and video by an interpolation algorithm.
[0141] In this embodiment, the first audio and video is input into a face quality detection model to obtain a weak face video frame in the first audio and video, where the weak face video frame refers to a video frame that contains a face but is not clear; the weak face video frame is deleted from the first audio and video, and the first audio and video is complemented by an interpolation algorithm; the video frames in the first audio and video can also be randomly deleted, and the first audio and video are complemented by an interpolation algorithm, simulating various possibilities in an actual application environment, and ensuring the temporal continuity and frame rate consistency of the video through the interpolation algorithm, thereby enhancing the prediction performance of the model under different situations.
[0142] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0143] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0144] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0145] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the lip synchronization detection method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.
[0146] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the lip synchronization detection method.
[0147] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0148] The computer device provided in this embodiment can execute the above lip synchronization detection method. Here, the lip synchronization detection method can be the lip synchronization detection method of each of the above embodiments.
[0149] In this embodiment, the video timestamp and audio timestamp of the audio and video are parsed, and the video and audio in the audio and video are aligned according to the video timestamp and audio timestamp to reduce the audio and video misalignment problem caused by timestamp mismatch; the audio and video are divided into multiple short clips, and the video features and audio features of each short clip are accurately extracted through the short-time model in the lip synchronization detection model, thereby ensuring the accuracy of lip synchronization detection from a fine-grained level; the video features and audio features of each short clip are input into the long-time model in the lip synchronization detection model, and the long-time model integrates the information of multiple short clips, captures the synchronization relationship and interaction of audio and video over a long period of time, and obtains cross-modal long-term context information, which can effectively deal with synchronization problems caused by network delays, frame loss, etc., thereby improving the robustness and accuracy of lip synchronization detection based on cross-modal long-term context information.
[0150] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the lip synchronization detection method as described above.
[0151] In this embodiment, the video timestamp and audio timestamp of the audio and video are parsed, and the video and audio in the audio and video are aligned according to the video timestamp and audio timestamp to reduce the audio and video misalignment problem caused by timestamp mismatch; the audio and video are divided into multiple short clips, and the video features and audio features of each short clip are accurately extracted through the short-time model in the lip synchronization detection model, thereby ensuring the accuracy of lip synchronization detection from a fine-grained level; the video features and audio features of each short clip are input into the long-time model in the lip synchronization detection model, and the long-time model integrates the information of multiple short clips, captures the synchronization relationship and interaction of audio and video over a long period of time, and obtains cross-modal long-term context information, which can effectively deal with synchronization problems caused by network delays, frame loss, etc., thereby improving the robustness and accuracy of lip synchronization detection based on cross-modal long-term context information.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0153] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A lip synchronization detection method, characterized in that: The steps include: Get the audio and video to be detected; Parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp; Splitting the audio and video into multiple short clips; Input each short segment into a lip sync detection model, and output the video features and audio features of each short segment through a short-time model in the lip sync detection model; Inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video; Lip synchronization detection is performed according to the cross-modal long-term context information to obtain a lip synchronization detection result.
2. The lip synchronization detection method according to claim 1, characterized in that: The step of parsing the video timestamp and the audio timestamp of the audio and video to align the video and audio in the audio and video according to the video timestamp and the audio timestamp includes: Extract the video timestamp of the video in the audio and video and the audio timestamp of the audio in the audio and video through ffmpeg; Comparing the video timestamp with the audio timestamp to obtain asynchronism information between the video and the audio; The video and audio are adjusted according to the out-of-sync information to align the video timestamp and the audio timestamp.
3. The lip synchronization detection method according to claim 1, characterized in that: After the step of obtaining the lip synchronization detection result, the method further includes: When it is determined according to the lip-sync detection result that there is lip-sync asynchrony, for each short segment, calculating the similarity between the video feature of the short segment and the audio features of all the short segments; Selecting the audio feature of the short segment corresponding to the highest similarity, and calculating the offset of the short segment according to the video feature of the short segment and the timestamp of the audio feature of the selected short segment; The offset information of the audio and video is generated according to the offset of each short segment, and the offset information is used to update the lip synchronization detection result.
4. The lip synchronization detection method according to claim 1, characterized in that: Before the step of obtaining the audio and video to be detected, the method further includes: Obtain the first audio and video for model training; Aligning the video and audio in the first audio and video; Segmenting the aligned first audio and video into a plurality of first short segments; Input each first short segment into an initial lip sync detection model, and output the video feature, audio feature and offset of each first short segment through a short-time model in the initial lip sync detection model; Inputting the video features and audio features of each of the first short clips into the long-term model in the initial lip synchronization detection model to obtain a lip synchronization prediction result; The model loss is calculated based on the offset of each first short segment and the lip synchronization prediction result, and the initial lip synchronization detection model is adjusted according to the model loss until the model loss meets the training stop condition, thereby obtaining the lip synchronization detection model.
5. The lip synchronization detection method according to claim 1, characterized in that: Before the step of obtaining the first audio and video for model training, the method further includes: Get the initial audio and video for model training; The video frame rate of the initial audio and video is randomly adjusted to obtain the first audio and video.
6. The lip synchronization detection method according to claim 4, characterized in that: After the step of obtaining the first audio and video for model training, the method further includes: Extract the video timestamp of the video in the first audio and video by using ffmpeg; When it is determined according to the video timestamp that the first audio and video has a frame rate imbalance, the video frame rate of the video in the first audio and video is adjusted to a standard frame rate through an interpolation algorithm.
7. The lip synchronization detection method according to claim 6, characterized in that: The method further comprises: Inputting the first audio and video into a face quality detection model to obtain a weak face video frame in the first audio and video; Deleting the weak face video frame from the first audio and video, and completing the first audio and video by the interpolation algorithm; and Randomly delete video frames in the first audio and video, and complete the first audio and video by using the interpolation algorithm.
8. A lip sync detection device, characterized in that: include: An audio and video acquisition module, used to acquire the audio and video to be detected; An audio and video alignment module, used for parsing the video timestamp and the audio timestamp of the audio and video, so as to align the video and audio in the audio and video according to the video timestamp and the audio timestamp; An audio and video segmentation module, used to segment the audio and video into multiple short segments; A short-time processing module, used for inputting each short segment into a lip-sync detection model, and outputting video features and audio features of each short segment through the short-time model in the lip-sync detection model; A long-term processing module, used for inputting the video features and audio features of each short clip into the long-term model in the lip synchronization detection model to obtain cross-modal long-term context information of the audio and video; The synchronization detection module is used to perform lip synchronization detection according to the cross-modal long-term context information to obtain a lip synchronization detection result.
9. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the lip synchronization detection method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the lip synchronization detection method according to any one of claims 1 to 7 are implemented.
