Sound and lip synchronization detection method and device, electronic equipment and storage medium

Abstract

The invention provides a sound and lip synchronous detection method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for the field of financial science and technology and the field of medical health. The method comprises the following steps: acquiring target audio and video data, wherein the target audio and video data comprises a target mixed audio and a target face video sequence; performing audio coding on the target mixed audio to obtain a target mixed audio feature; performing voice coding on the target mixed audio through a target voice coding model to obtain multi-modal features of a target speaker; performing visual coding on the target face video sequence to obtain a target visual feature; performing feature fusion according to the target mixed audio feature, the target speaker multi-modal feature and the target visual feature to obtain a target lip feature; and performing lip synchronization classification on the target lip features to obtain lip synchronization categories. According to the invention, the adverse effect of the noise in the audio on the sound lip synchronization detection can be reduced, and the accuracy of the sound lip synchronization detection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, and particularly to a method and apparatus for lip-sync detection, an electronic device, and a storage medium. Background Technology

[0002] Lip-sync detection technology refers to a type of detection task that determines whether the lips (mouth shapes) in a given audio signal and the corresponding video are consistent in time and semantics within a multimodal signal. For example, in online insurance verification scenarios in the fintech field, users upload a short video and read a given text or answer questions during the identity authentication process. Lip-sync detection technology can be used to verify whether the lip shapes in the video match the audio, thereby determining whether the video is from a real person and has not been replaced by a deepfake. Another example is in remote consultation scenarios in the healthcare field, where patients introduce themselves and read a specified text during video calls while simultaneously transmitting audio signals. Lip-sync detection technology can be used to verify whether the lip shapes in the video match the audio content, improving the robustness of determining the authenticity of the patient's identity.

[0003] Current technologies primarily rely on the features of audio signals and the visual features of video to achieve lip-sync detection. However, audio signals often contain noise during acquisition (such as background noise, music, and other people's voices), which affects the accuracy of audio signal features and consequently reduces the accuracy of lip-sync detection. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for lip-sync detection, which can reduce the adverse effects of noise in audio on lip-sync detection and improve the accuracy of lip-sync detection.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for lip-phoneme synchronization detection, the method comprising:

[0006] Acquire target audio and video data, the target audio and video data including target mixed audio and target face video sequence, the target face video sequence including the lips of the target speaker;

[0007] The target mixed audio is encoded to obtain the target mixed audio features;

[0008] The target mixed audio is encoded by a pre-trained target speech coding model to obtain target speaker multimodal features; wherein, the target speaker multimodal features are used to characterize the multimodal features of the speech corresponding to the target speaker;

[0009] Visual encoding is performed on the target face video sequence to obtain target visual features;

[0010] The target lip features are obtained by fusing the target mixed audio features, the target speaker multimodal features, and the target visual features.

[0011] The target lip-phone features are classified for lip-phone synchronization to obtain a lip-phone synchronization category; wherein, the lip-phone synchronization category is used to characterize whether the target mixed audio is synchronized or not synchronized with the lips in the target face video sequence.

[0012] Optionally, before performing speech encoding on the target mixed audio using a pre-trained target speech encoding model to obtain the target speaker's multimodal features, the method further includes:

[0013] Obtain clean speech samples, and add noise to the clean speech samples to obtain mixed audio samples;

[0014] Obtain an initial audio denoising model, the last layer of which is a speech reconstruction unit;

[0015] The sample mixed audio is denoised using the initial audio denoising model to obtain the sample reconstructed speech;

[0016] Loss is calculated based on the clean speech sample and the reconstructed speech sample to obtain the target loss data;

[0017] The parameters of the initial audio denoising model are adjusted based on the target loss data to obtain the target audio denoising model;

[0018] The speech reconstructor in the target audio denoising model is deleted to obtain the target speech coding model.

[0019] Optionally, the initial audio denoising model includes an audio encoder, a speaker extractor, a feature fusion layer, a speaker-aware encoder, and a speech reconstruction unit.

[0020] The step of denoising the sample mixed audio using the initial audio denoising model to obtain the sample reconstructed speech includes:

[0021] The sample mixed audio is encoded using the audio encoder to obtain sample mixed audio features;

[0022] The voiceprint extractor is used to extract the voiceprint features of the sample mixed audio.

[0023] The feature fusion layer performs feature fusion on the sample mixed audio features and the sample voiceprint features to obtain sample fused features;

[0024] The speaker multimodal features of the sample are obtained by using the voiceprint perception encoder to perceive the fused features of the sample.

[0025] The speech reconstruction device reconstructs the speech from the mixed audio features of the sample and the multimodal features of the speaker of the sample.

[0026] Optionally, the step of performing voiceprint perception on the sample fusion features through the voiceprint-aware encoder to obtain the multimodal features of the sample speaker includes:

[0027] The sample fusion features are segmented to obtain feature blocks; wherein, two adjacent feature blocks overlap.

[0028] The first three-dimensional feature is obtained by splicing together the individual feature blocks.

[0029] The feature blocks in the three-dimensional feature are permuted to obtain the first permuted feature;

[0030] The voiceprint-aware encoder extracts features from each feature block in the first permutation feature to obtain a context feature block, and merges the context feature blocks to obtain a second three-dimensional feature.

[0031] The second three-dimensional feature is normalized and then a permutation operation is performed to obtain the second permuted feature;

[0032] The speaker multimodal features of the sample are obtained by extracting features from the second permutation feature using the voiceprint-aware encoder.

[0033] Optionally, the step of adding noise to the clean speech samples to obtain mixed audio samples includes:

[0034] Acquire a reference audio, and generate a reference interference signal based on the reference audio;

[0035] The reference interference signal is superimposed on the clean speech sample to obtain the mixed audio sample.

[0036] Optionally, the step of calculating the loss based on the clean speech sample and the reconstructed speech sample to obtain the target loss data includes:

[0037] The clean speech sample is encoded to obtain clean speech sample features, and the reconstructed speech sample is encoded to obtain reconstructed speech sample features;

[0038] The initial speech interaction features are obtained by performing a vector inner product of the clean speech features and the reconstructed speech features of the samples. The first speech distance feature is obtained by multiplying the initial speech interaction features with the clean speech features of the samples.

[0039] The L2 norm is calculated based on the clean speech features of the sample to obtain the second speech distance feature;

[0040] The third speech distance feature is obtained by calculating the ratio between the first speech distance feature and the second speech distance feature, and the fourth speech distance feature is obtained by calculating the L2 norm based on the third speech distance feature.

[0041] The difference between the reconstructed speech features and the third speech distance features is calculated to obtain the fifth speech distance feature. The L2 norm is calculated based on the fifth speech distance feature to obtain the sixth speech distance feature.

[0042] The target speech distance is obtained by calculating the ratio between the fourth speech distance feature and the sixth speech distance feature, and the target loss data is obtained by performing a logarithmic operation on the target speech distance.

[0043] Optionally, feature fusion is performed based on the target mixed audio features, the target speaker multimodal features, and the target visual features to obtain target lip features, including:

[0044] The target hybrid audio features, the target speaker multimodal features, and the target visual features are concatenated to obtain the initial lip features;

[0045] The initial lip features are subjected to self-attention calculation to obtain the target lip features.

[0046] To achieve the above objectives, a second aspect of this application provides a lip-sound synchronization detection device, the device comprising:

[0047] The data acquisition module is used to acquire target audio and video data, which includes target mixed audio and target face video sequence, and the target face video sequence includes the lips of the target speaker;

[0048] The audio encoding module is used to encode the target mixed audio to obtain the target mixed audio features;

[0049] The speech encoding module is used to encode the target mixed audio using a pre-trained target speech encoding model to obtain target speaker multimodal features; wherein, the target speaker multimodal features are used to characterize the multimodal features of the speech corresponding to the target speaker;

[0050] A visual encoding module is used to perform visual encoding on the target face video sequence to obtain target visual features;

[0051] The feature fusion module is used to perform feature fusion based on the target mixed audio features, the target speaker multimodal features, and the target visual features to obtain target lip features;

[0052] The synchronization classification module is used to classify the target lip features into lip synchronization categories to obtain lip synchronization categories; wherein, the lip synchronization categories are used to characterize whether the target mixed audio is synchronized or not synchronized with the lips in the target face video sequence.

[0053] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the lip-sound synchronization detection method described in the first aspect.

[0054] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the lip-sync detection method described in the first aspect.

[0055] The proposed method, apparatus, electronic device, and storage medium for lip-sound synchronization detection address the technical problem of noise in the target mixed audio affecting the accuracy of lip-sound synchronization detection. Instead of denoising the target mixed audio and then performing lip-sound synchronization detection based on the denoised audio, this application performs audio encoding on the target mixed audio to obtain target mixed audio features and visual encoding on the target face video sequence to obtain target visual features. However, considering the noise in the target mixed audio features, this application also performs speech encoding on the target mixed audio to obtain multimodal features of the speech corresponding to the target speaker, i.e., target speaker multimodal features. Then, feature fusion is performed based on the target mixed audio features, target speaker multimodal features, and target visual features to obtain target lip-sound features. The noise feature components in these target lip-sound features are compressed, while the feature components related to the speech and lips of the target speaker are enhanced. Finally, lip-sound synchronization classification is performed on the target lip-sound features to obtain lip-sound synchronization categories. In summary, this application can reduce the adverse effects of noise in audio on lip-sound synchronization detection and improve the accuracy of lip-sound synchronization detection.

[0056] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0057] Figure 1 This is a flowchart of the lip-sound synchronization detection method provided in the embodiments of this application;

[0058] Figure 2 This is a flowchart of a lip-sound synchronization detection method provided in another embodiment of this application;

[0059] Figure 3 yes Figure 2 The flowchart for step 203 in the document;

[0060] Figure 4 yes Figure 3 The flowchart for step 304 in the document;

[0061] Figure 5 yes Figure 2 The flowchart for step 204 in the document;

[0062] Figure 6 yes Figure 1 The flowchart for step 105 in the document;

[0063] Figure 7 This is a block diagram of the module structure of the lip-sync detection device provided in the embodiments of this application;

[0064] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0070] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0071] With the increasing prevalence of online verification, the importance of lip-sync detection technology is becoming increasingly prominent, especially in applications within secure environments. Environmental issues caused by bystander interference have become a pressing challenge. Furthermore, while advancements in multimodal technologies and models have improved detection performance, interference from non-speech noise (such as background noise and music) remains a significant problem, directly impacting the application prospects of lip-sync detection in complex and dynamic environments. How to further optimize lip-sync detection capabilities and achieve high accuracy in complex and varied environments is a question worthy of in-depth research and exploration.

[0072] The lip-sync detection method provided in this application can be applied to terminals and servers, or it can be software running on the server. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the lip-sync detection method, but it is not limited to the above forms.

[0073] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0074] This application provides a method for detecting lip-phonetic synchronization, a device for detecting lip-phonetic synchronization, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the method for detecting lip-phonetic synchronization in this application is described.

[0075] It should be noted that in each specific implementation of this application, when it is necessary to process data related to the user's identity or characteristics, such as voice data or facial data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0076] Reference Figure 1 , Figure 1 This is an optional flowchart of the lip-sound synchronization detection method provided in the embodiments of this application. The method may include, but is not limited to, steps 101 to 106.

[0077] Step 101: Obtain target audio and video data, which includes target mixed audio and target face video sequence, and the target face video sequence includes the lips of the target speaker;

[0078] Step 102: Perform audio encoding on the target mixed audio to obtain the target mixed audio features;

[0079] Step 103: Encode the target mixed audio using a pre-trained target speech coding model to obtain the target speaker's multimodal features;

[0080] Step 104: Visually encode the target face video sequence to obtain the target visual features;

[0081] Step 105: Perform feature fusion based on the target mixed audio features, the target speaker multimodal features, and the target visual features to obtain the target lip features;

[0082] Step 106: Perform lip-sound synchronization classification on the target lip features to obtain the lip-sound synchronization category.

[0083] In steps 101 to 106 of this application embodiment, when addressing the technical problem of noise in the target mixed audio affecting the accuracy of lip-sync detection, instead of denoising the target mixed audio and then performing lip-sync detection based on the denoised audio, this application performs audio encoding on the target mixed audio to obtain target mixed audio features and visual encoding on the target face video sequence to obtain target visual features. However, considering the noise in the target mixed audio features, this application also performs speech encoding on the target mixed audio to obtain multimodal features of the speech corresponding to the target speaker, i.e., target speaker multimodal features. Then, feature fusion is performed based on the target mixed audio features, target speaker multimodal features, and target visual features to obtain target lip-sync features. The noise feature components in these target lip-sync features are compressed, while the feature components related to the speech and lips of the target speaker are enhanced. Finally, lip-sync classification is performed on the target lip-sync features to obtain lip-sync categories. In summary, this application can reduce the adverse effects of noise in audio on lip-sync detection and improve the accuracy of lip-sync detection.

[0084] In one example, in an online insurance verification scenario in the fintech field, when a user uploads a short video and reads a given text or answers a question during the identity authentication process, the lip-sync technology can be used to verify whether the "lip movements in the video" match the audio, thereby determining whether the video is from a real person and has not been replaced by a deepfake.

[0085] In one example, within the fintech sector's inclusive lending scenario, a user might want to apply for a loan from a bank. In this case, the user can initiate a video conference with the bank's linked terminal or server via a mobile app or web platform. During the video conference, a digital customer service representative will ask the user multiple questions, including identity information, asset status, proof of income, and loan purpose. Lip-sync technology can be used to verify whether the lip movements in the video match the audio, thus determining whether the video is from a real person and has not been replaced by a deepfake.

[0086] In one example, within a remote consultation scenario in the healthcare field, a patient introduces themselves and reads a designated text during a video call, while simultaneously transmitting an audio signal. Lip-sync technology can be used to verify whether the lip movements in the video match the audio content, improving the robustness of verifying the patient's identity.

[0087] In step 101 of some embodiments, target audio and video data is acquired. The target audio and video data includes target mixed audio and a target face video sequence. The target mixed audio includes the speech of the target speaker. If the audio was acquired in a noisy environment, the target mixed audio also includes noise (background noise, music, and other people's voices). The target face video sequence includes multiple target face video frames. Each target face video frame includes the face of the target speaker, including lips (lip movements), i.e., the target face video sequence includes the lips of the target speaker. The target mixed audio can be acquired using a microphone, and the target face video sequence can be acquired using a camera.

[0088] In step 102 of some embodiments, the target mixed audio is audio encoded to obtain target mixed audio features. For example, the target mixed audio is input into an audio encoding network, which converts the temporal audio waveform corresponding to the target mixed audio into a 2D embedding sequence, i.e., the target mixed audio features. The audio encoding network can be a 1D-CNN. This application does not limit the audio encoding network and can set it according to requirements.

[0089] In one embodiment, reference is made to Figure 2 Before step 102, the lip-sync detection method may further include:

[0090] Step 201: Obtain clean speech samples, add noise to the clean speech samples, and obtain mixed audio samples.

[0091] Step 202: Obtain the initial audio denoising model. The last layer of the initial audio denoising model is the speech reconstruction unit.

[0092] Step 203: Denoise the mixed audio samples using the initial audio denoising model to obtain the reconstructed speech samples;

[0093] Step 204: Calculate the loss based on the clean speech sample and the reconstructed speech sample to obtain the target loss data;

[0094] Step 205: Adjust the parameters of the initial audio denoising model based on the target loss data to obtain the target audio denoising model;

[0095] Step 206: Delete the speech reconstructor in the target audio denoising model to obtain the target speech coding model.

[0096] In step 201, the clean speech sample is the speech of a normal speaker without noise. The mixed audio sample is the audio sample clean speech with noise added on top.

[0097] In one embodiment, step 201 may include: acquiring reference audio, generating a reference interference signal based on the reference audio, and superimposing the reference interference signal onto the clean sample speech to obtain mixed sample audio. For example, other video audio or MUSAN noise library signals are randomly selected as reference audio and superimposed onto the clean sample speech according to SNR ∈ [-10, 10] dB to generate mixed sample audio. In this way, by controlling the SNR of the superimposed interference signal to simulate a real noise environment, the generalization ability of the target speech coding model is improved, thereby improving the accuracy of lip-sync detection.

[0098] In step 202, the initial audio denoising model is a neural network model used to recover clean speech from the input mixed audio. The initial audio denoising model consists of multiple layers. The last layer of the initial audio denoising model is a speech reconstruction unit. The last layer reconstructs clean speech based on the output features obtained from mapping the mixed audio by the previous layers.

[0099] In one embodiment, the initial audio denoising model includes an audio encoder, a speaker extractor, a feature fusion layer, a speaker-aware encoder, and a speech reconstruction unit. In this case, the initial audio denoising model comprises five layers, with the fifth layer being the speech reconstruction unit.

[0100] In step 203, the sample mixed audio is input into the initial audio denoising model for denoising, and the sample reconstructed speech can be obtained.

[0101] In one embodiment, reference is made to Figure 3 Step 203 may include:

[0102] Step 301: The sample mixed audio is encoded using an audio encoder to obtain the sample mixed audio features;

[0103] Step 302: Extract the voiceprint from the mixed audio sample using a voiceprint extractor to obtain the voiceprint features of the sample.

[0104] Step 303: The sample mixed audio features and sample speaker features are fused through the feature fusion layer to obtain the sample fused features;

[0105] Step 304: Use a voiceprint-aware encoder to perform voiceprint perception on the sample fusion features to obtain the multimodal features of the sample speaker;

[0106] Step 305: The mixed audio features and multimodal features of the speaker in the sample are reconstructed using a speech reconstruction tool to obtain the reconstructed speech of the sample.

[0107] In step 301, the sample mixed audio is input into an audio encoder. The audio encoder converts the temporal audio waveform corresponding to the sample mixed audio into a 2D embedding sequence, i.e., the sample mixed audio features. The audio encoder can be a 1D-CNN. This application does not impose any restrictions on the audio encoder and it can be set according to requirements.

[0108] In step 302, the sample mixed audio is input into the speaker extractor. The speaker extractor converts the temporal audio waveform corresponding to the sample mixed audio into a 2D embedding sequence, i.e., the sample speaker features. The speaker extractor can be ERes2Net. This application does not limit the speaker extractor and can set it according to requirements.

[0109] In step 303, the feature fusion layer is used to fuse the sample mixed audio features and sample speaker features. Common fusion methods, such as concatenation and cross-attention, can achieve audio and speaker feature fusion. The shape of the sample fusion feature is (D, T), where D is the dimension of the sample fusion feature and T is the time length of the sample fusion feature.

[0110] In step 304, to better extract speech features related to voiceprints, a voiceprint-aware encoder is proposed. The voiceprint-aware encoder can include multiple (e.g., six) cascaded bidirectional long short-term memory networks (BLSTM). Inputting the sample fusion features into the voiceprint-aware encoder yields the multimodal features of the sample speaker. These multimodal features characterize the multimodal features of the speech corresponding to the sample speaker. Multimodal features refer to information beyond the pure speech waveform itself, including associated auxiliary modal information such as text transcription (ASR output), emotion / feeling tags, speaking context, lip / facial features, and vocal tract physiological parameters.

[0111] The goal of BLSTM is to improve the ability to predict the current state by simultaneously utilizing contextual information before and after a time step in sequence modeling. The core idea of ​​BLSTM is to propagate the sequence "bidirectionally" in the time dimension, using a forward LSTM running from front to back and a backward LSTM running from back to front. The hidden states are concatenated or merged at each time step to obtain a richer representation of the current state.

[0112] In one embodiment, reference is made to Figure 4 Step 304 may include:

[0113] Step 401: Segment the sample fusion features to obtain feature blocks; where adjacent feature blocks overlap.

[0114] Step 402: Concatenate the individual feature blocks to obtain the first three-dimensional feature;

[0115] Step 403: Perform a permutation operation on the feature blocks in the three-dimensional features to obtain the first permuted feature;

[0116] Step 404: Using a voiceprint-aware encoder, extract features from each feature block in the first permutation feature to obtain a context feature block, and merge the context feature blocks to obtain the second three-dimensional feature.

[0117] Step 405: Normalize the second three-dimensional feature and then perform a permutation operation to obtain the second permuted feature;

[0118] Step 406: Using a voiceprint-aware encoder, feature extraction is performed on the second permutation feature to obtain the multimodal features of the sample speaker.

[0119] Specifically, the sample fusion features are segmented into multiple feature chunks with a 50% overlap rate. These feature chunks are further concatenated to form the first three-dimensional feature (shape D, K, Q, where Q is the number of chunks, K is the length of a chunk, and D is the feature dimension). The encoder consists of six identical audiovisual extraction blocks (BLSTM) connected in series. In each audiovisual extraction block, a permutation operation is first performed to obtain the first permuted feature. Subsequently, a BLSTM is applied within each feature chunk to integrate knowledge of the local temporal dimension. After passing through a linear layer with group normalization (GN), the features are reordered through a permutation operation, and another BLSTM is applied to integrate the global temporal information between all feature chunks. The final output is a two-dimensional embedding M, which is the sample speaker multimodal feature. The sample speaker multimodal feature contains multimodal features that represent speech signals associated with a given voiceprint, providing an optimized multimodal information foundation for subsequent lip-sync classification tasks.

[0120] The advantage of the embodiments of steps 401 to 406 described above is that they can improve the ability of the multimodal features of the sample speaker to represent the speech of the speaker, which is beneficial to improving the accuracy of lip-sound synchronization detection.

[0121] In step 305, the sample mixed audio features and sample speaker multimodal features are input into the speech reconstruction unit. The speech reconstruction unit performs speech reconstruction on the sample mixed audio features and sample speaker multimodal features to obtain the sample reconstructed speech. For example, the speech reconstruction unit first multiplies the sample speaker multimodal features with the sample mixed audio features, and then inputs the result of the multiplication into a linear layer to obtain the sample reconstructed speech.

[0122] The advantage of the embodiments of steps 301 to 305 above is that by fusing voiceprint features and mixed audio features, and then performing voiceprint perception on the fused features, the ability of the multimodal features of the sample speaker to represent the speech of the speaker can be improved, which is conducive to improving the accuracy of lip-sound synchronization detection.

[0123] In step 204, the target loss data is used to characterize the difference between the clean speech sample and the reconstructed speech sample. The larger the target loss data, the greater the difference, and the more the parameters of the initial audio denoising model need to be adjusted until the target loss data is less than a predetermined loss threshold, or the number of training iterations reaches a predetermined threshold.

[0124] In one embodiment, reference is made to Figure 5 Step 204 may include:

[0125] Step 501: Encode the clean speech sample to obtain clean speech sample features, and encode the reconstructed speech sample to obtain reconstructed speech sample features.

[0126] Step 502: Perform a vector inner product of the clean speech features and the reconstructed speech features to obtain the initial speech interaction features. Multiply the initial speech interaction features with the clean speech features to obtain the first speech distance features.

[0127] Step 503: Calculate the L2 norm based on the clean speech features of the sample to obtain the second speech distance feature;

[0128] Step 504: Calculate the ratio of the first speech distance feature and the second speech distance feature to obtain the third speech distance feature; calculate the L2 norm based on the third speech distance feature to obtain the fourth speech distance feature.

[0129] Step 505: Calculate the difference between the reconstructed speech features and the third speech distance features to obtain the fifth speech distance features; calculate the L2 norm based on the fifth speech distance features to obtain the sixth speech distance features.

[0130] Step 506: Calculate the ratio of the fourth speech distance feature and the sixth speech distance feature to obtain the target speech distance, and perform a logarithmic operation on the target speech distance to obtain the target loss data.

[0131] In step 501, the clean speech sample can be encoded using a speech encoder to obtain clean speech features. Similarly, the reconstructed speech sample can be encoded using the same speech encoder to obtain reconstructed speech features. The more similar or identical the clean speech features and the reconstructed speech features are, the stronger the denoising capability of the initial audio denoising model. The speech encoder can employ neural networks such as 1D-CNN.

[0132] In step 502, the first speech distance feature can be as follows:

[0133] L1 =<s′,s> s,

[0134] Where L1 represents the first speech distance feature, s′ represents the sample reconstructed speech feature corresponding to the sample reconstructed speech, and s represents the sample clean speech feature corresponding to the sample clean speech.

[0135] In step 503, the second speech distance feature can be as follows:

[0136] L2 = |s| 2 ,

[0137] Where L2 represents the second speech distance feature, and s represents the sample clean speech feature corresponding to the sample clean speech.

[0138] In step 504, the fourth speech distance feature can be as follows:

[0139]

[0140] Where L4 represents the fourth speech distance feature, This represents the third speech distance feature, L3.

[0141] In step 505, the sixth speech distance feature can be as follows:

[0142]

[0143] Where L6 represents the sixth speech distance feature, This represents the fifth speech distance feature, L5.

[0144] In step 506, the target loss data can be as follows:

[0145]

[0146] Among them, L pretrain This represents the target loss data.

[0147] The advantage of the embodiments of steps 501 to 506 described above is that the target loss data can be determined by the distance between the sample reconstructed speech and the sample clean speech, which is beneficial to improving the denoising capability of the target audio denoising model, thereby improving the speech coding capability of the target speech coding model, and thus improving the accuracy of lip-sync detection.

[0148] In step 205, the parameters of the initial audio denoising model are adjusted based on the target loss data to obtain the target audio denoising model. Finally, in step 206, the speech reconstructor in the target audio denoising model is removed to obtain the target speech coding model.

[0149] The advantage of the embodiments of steps 201 to 206 described above is that they can improve the denoising capability of the target audio denoising model, thereby improving the speech coding capability of the target speech coding model, and thus improving the accuracy of lip-sync detection.

[0150] In step 103 of some embodiments, the target mixed audio is encoded using a pre-trained target speech coding model to obtain target speaker multimodal features. These target speaker multimodal features are used to characterize the multimodal features of the speech corresponding to the target speaker. The target speech coding model includes an audio encoder, a speaker extractor, a feature fusion layer, and a speaker-aware encoder.

[0151] In one embodiment, step 103 may include: encoding the target mixed audio using an audio encoder to obtain target mixed audio features; extracting the target mixed audio using a voiceprint extractor to obtain target voiceprint features; fusing the target mixed audio features and target voiceprint features using a feature fusion layer to obtain target fused features; and performing voiceprint perception on the target fused features using a voiceprint-aware encoder to obtain target speaker multimodal features. Thus, by fusing voiceprint features and mixed audio features, and then performing voiceprint perception on the fused features, the ability of the target speaker multimodal features to represent the speaker's corresponding speech can be improved, which is beneficial for improving the accuracy of lip-sync detection.

[0152] In step 104 of some embodiments, the target face video sequence is visually encoded to obtain target visual features. For example, a visual encoder visually encodes the target face video frames in the target face video sequence to obtain target visual features. Visual encoders include CNN, RNN, BLSTM, Transformer encoders, etc. Target visual features are used to characterize the lip movement features of the target speaker.

[0153] In step 105 of some embodiments, feature fusion is performed based on target mixed audio features, target speaker multimodal features and target visual features to obtain target lip features.

[0154] In one embodiment, reference is made to Figure 6 Step 105 may include:

[0155] Step 601: Concatenate the target hybrid audio features, the target speaker multimodal features, and the target visual features to obtain the initial lip features;

[0156] Step 602: Perform self-attention calculation on the initial lip features to obtain the target lip features.

[0157] Self-attention computation is based on the self-attention mechanism. Self-attention is an attention mechanism that allows the model to directly perform weighted aggregation on all other positions in the same sequence when computing the representation of a certain position, thereby gaining an awareness of the global context. Simply put, it allows each element in the sequence to "pay attention" to other elements in the same sequence to capture global dependencies and relationships.

[0158] The advantage of the embodiments of steps 601 to 602 described above is that by first splicing and then performing self-attention calculation, the noise feature components in the target lip features can be compressed, while the feature components related to the speech and lips of the target speaker can be improved, that is, the representation ability of the target lip features is improved, thereby improving the accuracy of lip-sound synchronization detection.

[0159] In step 106 of some embodiments, the target lip-sync features can be classified using a classifier to obtain a lip-sync category. The lip-sync category is used to characterize whether the lips in the target mixed audio and the target face video sequence are synchronized or not synchronized. The classifier may include a cascaded 1D-CNN and fully connected layers.

[0160] In one embodiment, before step 106, the lip-sound synchronization detection method may further include: fusing sample mixed audio features, sample visual features, and sample speaker multimodal features through a self-attention mechanism, outputting whether the result is lip-sound synchronization after passing through a 1D-CNN and a fully connected layer; and then training a target speech coding model and a classifier according to a loss function to obtain a lip-sound synchronization detection model.

[0161] For example, the loss function is as follows:

[0162] L fintune = p′logp+(1-p′)log(1-p), where p′ is the predicted lip-sync label and p is the actual lip-sync label.

[0163] In one embodiment, the lip-sync detection model can be pre-trained in the following manner:

[0164] Pre-training task definition: Extract clean speech of the target speaker from mixed audio based on voiceprint through self-supervised learning. Specifically, it includes: input: sample mixed audio (where s is the sample clean speech and s is the interference signal) and the voiceprint features of the target speaker; output: sample reconstructed speech. The goal is to minimize the distortion between the sample reconstructed speech and the sample clean speech.

[0165] Basic corpus selection: Single-speaker audio-video synchronization datasets such as VoxCeleb2 were used, with each sample containing a single visible speaking face and its clean speech sample;

[0166] Interference signal superposition: Randomly select other video audio or MUSAN noise library signals as interference sources, and superimpose them onto the clean speech samples according to SNR∈[-10,10]dB to generate mixed audio samples;

[0167] Sample structure: Each training sample contains a triple (x, v, s). This process can generate an infinite amount of self-supervised training data. x is the mixed audio of the sample (including noise), v is the sample face video sequence, and s is the sample clean speech.

[0168] The initial audio denoising model is trained by mixing sample audio and clean sample speech in the training samples (the specific training process can be referred to in steps 201 to 206) to obtain the target speech coding model. Then, the target speech coding model and classifier are trained by the training samples to obtain the lip sample detection model.

[0169] In summary, the present application can achieve at least the following beneficial effects: (1) it is the first to model the target speaker extraction task based on voiceprint and the AV-ASD task in a unified manner; (2) it designs a self-supervised hybrid audio generation method, which simulates the real noise environment by superimposing interference signals with controllable SNR, thereby improving the accuracy of lip-sync detection.

[0170] Please see Figure 7 This application also provides a lip-pronunciation synchronization detection device, which can implement the above-described lip-pronunciation synchronization detection method. Figure 7 This is a block diagram of the module structure of the lip-sync detection device provided in the embodiments of this application. The device includes:

[0171] The data acquisition module 701 is used to acquire target audio and video data, which includes target mixed audio and target face video sequence, and the target face video sequence includes the lips of the target speaker;

[0172] Audio encoding module 702 is used to encode the target mixed audio to obtain the target mixed audio features;

[0173] The speech coding module 703 is used to perform speech coding on the target mixed audio through a pre-trained target speech coding model to obtain the target speaker multimodal features; wherein, the target speaker multimodal features are used to represent the multimodal features of the speech corresponding to the target speaker;

[0174] The visual encoding module 704 is used to visually encode the target face video sequence to obtain the target visual features;

[0175] The feature fusion module 705 is used to fuse the target mixed audio features, the target speaker multimodal features and the target visual features to obtain the target lip features;

[0176] The synchronization classification module 706 is used to classify the lip-phonetic features of the target and obtain the lip-phonetic synchronization category; wherein, the lip-phonetic synchronization category is used to characterize whether the lips of the target mixed audio and the target face video sequence are synchronized or not synchronized.

[0177] In one embodiment, the lip-sync detection device further includes a model training module, configured to: acquire clean sample speech; add noise to the clean sample speech to obtain mixed sample audio; acquire an initial audio denoising model, the last layer of which is a speech reconstructor; denoise the mixed sample audio using the initial audio denoising model to obtain reconstructed sample speech; calculate the loss based on the clean sample speech and the reconstructed sample speech to obtain target loss data; adjust the parameters of the initial audio denoising model based on the target loss data to obtain a target audio denoising model; and remove the speech reconstructor from the target audio denoising model to obtain a target speech coding model.

[0178] It should be noted that the specific implementation of this lip-phonetic synchronization detection device is basically the same as the specific implementation of the above-mentioned lip-phonetic synchronization detection method, and will not be repeated here.

[0179] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned lip-sync detection method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0180] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0181] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0182] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the lip-sync detection method of the embodiments of this application.

[0183] The 803 input / output interface is used to implement information input and output.

[0184] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0185] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0186] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0187] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described lip-sync detection method.

[0188] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0189] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0190] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0193] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0194] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0196] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0198] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for detecting lip-phonetic synchronization, characterized in that, The method includes: Acquire target audio and video data, the target audio and video data including target mixed audio and target face video sequence, the target face video sequence including the lips of the target speaker; The target mixed audio is encoded to obtain the target mixed audio features; The target mixed audio is encoded by a pre-trained target speech coding model to obtain target speaker multimodal features; wherein, the target speaker multimodal features are used to characterize the multimodal features of the speech corresponding to the target speaker; Visual encoding is performed on the target face video sequence to obtain target visual features; The target lip features are obtained by fusing the target mixed audio features, the target speaker multimodal features, and the target visual features. The target lip-phone features are classified for lip-phone synchronization to obtain a lip-phone synchronization category; wherein, the lip-phone synchronization category is used to characterize whether the target mixed audio is synchronized or not synchronized with the lips in the target face video sequence.

2. The method according to claim 1, characterized in that, Before performing speech encoding on the target mixed audio using a pre-trained target speech encoding model to obtain the target speaker's multimodal features, the method further includes: Obtain clean speech samples, and add noise to the clean speech samples to obtain mixed audio samples; Obtain an initial audio denoising model, the last layer of which is a speech reconstruction unit; The sample mixed audio is denoised using the initial audio denoising model to obtain the sample reconstructed speech; Loss is calculated based on the clean speech sample and the reconstructed speech sample to obtain the target loss data; The parameters of the initial audio denoising model are adjusted based on the target loss data to obtain the target audio denoising model; The speech reconstructor in the target audio denoising model is deleted to obtain the target speech coding model.

3. The method according to claim 2, characterized in that, The initial audio denoising model includes an audio encoder, a speaker extractor, a feature fusion layer, a speaker-aware encoder, and a speech reconstruction unit. The step of denoising the sample mixed audio using the initial audio denoising model to obtain the sample reconstructed speech includes: The sample mixed audio is encoded using the audio encoder to obtain sample mixed audio features; The voiceprint extractor is used to extract the voiceprint features of the sample mixed audio. The feature fusion layer performs feature fusion on the sample mixed audio features and the sample voiceprint features to obtain sample fused features; The speaker multimodal features of the sample are obtained by using the voiceprint perception encoder to perceive the fused features of the sample. The speech reconstruction device reconstructs the speech from the mixed audio features of the sample and the multimodal features of the speaker of the sample.

4. The method according to claim 3, characterized in that, The step of performing voiceprint perception on the sample fusion features through the voiceprint perception encoder to obtain the multimodal features of the sample speaker includes: The sample fusion features are segmented to obtain feature blocks; wherein, two adjacent feature blocks overlap. The first three-dimensional feature is obtained by splicing together the individual feature blocks. The feature blocks in the three-dimensional feature are permuted to obtain the first permuted feature; The voiceprint-aware encoder extracts features from each feature block in the first permutation feature to obtain a context feature block, and merges the context feature blocks to obtain a second three-dimensional feature. The second three-dimensional feature is normalized and then a permutation operation is performed to obtain the second permuted feature; The speaker multimodal features of the sample are obtained by extracting features from the second permutation feature using the voiceprint-aware encoder.

5. The method according to any one of claims 2 to 4, characterized in that, The step of adding noise to the clean speech samples to obtain mixed audio samples includes: Acquire a reference audio, and generate a reference interference signal based on the reference audio; The reference interference signal is superimposed on the clean speech sample to obtain the mixed audio sample.

6. The method according to claim 3, characterized in that, The step of calculating the loss based on the clean speech sample and the reconstructed speech sample to obtain the target loss data includes: The clean speech sample is encoded to obtain clean speech sample features, and the reconstructed speech sample is encoded to obtain reconstructed speech sample features; The initial speech interaction features are obtained by performing a vector inner product of the clean speech features and the reconstructed speech features of the samples. The first speech distance feature is obtained by multiplying the initial speech interaction features with the clean speech features of the samples. The L2 norm is calculated based on the clean speech features of the sample to obtain the second speech distance feature; The third speech distance feature is obtained by calculating the ratio between the first speech distance feature and the second speech distance feature, and the fourth speech distance feature is obtained by calculating the L2 norm based on the third speech distance feature. The difference between the reconstructed speech features and the third speech distance features is calculated to obtain the fifth speech distance feature. The L2 norm is calculated based on the fifth speech distance feature to obtain the sixth speech distance feature. The target speech distance is obtained by calculating the ratio between the fourth speech distance feature and the sixth speech distance feature, and the target loss data is obtained by performing a logarithmic operation on the target speech distance.

7. The method according to any one of claims 1 to 4, characterized in that, Based on the target mixed audio features, the target speaker multimodal features, and the target visual features, feature fusion is performed to obtain the target lip features, including: The target hybrid audio features, the target speaker multimodal features, and the target visual features are concatenated to obtain the initial lip features; The initial lip features are subjected to self-attention calculation to obtain the target lip features.

8. A lip-sync detection device, characterized in that, The device includes: The data acquisition module is used to acquire target audio and video data, which includes target mixed audio and target face video sequence, and the target face video sequence includes the lips of the target speaker; The audio encoding module is used to encode the target mixed audio to obtain the target mixed audio features; The speech encoding module is used to encode the target mixed audio using a pre-trained target speech encoding model to obtain target speaker multimodal features; wherein, the target speaker multimodal features are used to characterize the multimodal features of the speech corresponding to the target speaker; A visual encoding module is used to perform visual encoding on the target face video sequence to obtain target visual features; The feature fusion module is used to perform feature fusion based on the target mixed audio features, the target speaker multimodal features, and the target visual features to obtain target lip features; The synchronization classification module is used to classify the target lip features into lip synchronization categories to obtain lip synchronization categories; wherein, the lip synchronization categories are used to characterize whether the target mixed audio is synchronized or not synchronized with the lips in the target face video sequence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images