Audio-visual speech synchronization detection method and device based on cross-modal distillation, equipment and medium

Abstract

The application provides a lip synchronization detection method and device based on cross-modal distillation, equipment and medium, the method comprises: obtaining training data, and preprocessing the training data; based on the preset teacher model and student model, generate training teacher audio feature data, teacher soft label distribution data, training student audio feature data and student soft label distribution data according to the preprocessed data; based on the preset double constraint knowledge distillation algorithm, train the student model according to the feature data and the soft label distribution data; receive the video data to be detected, and separate processing; based on the teacher model and the trained student model, generate feature data according to the separated video data to be detected; based on the preset judgment algorithm, judge two feature data to detect whether the video data to be detected is lip synchronization. It can be applied to the behavior detection scene of financial technology and medical health, and solve the problems of poor robustness, insufficient detection precision and high labeling cost of existing lip synchronization detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and is applied to the fields of financial technology or healthcare. In particular, it relates to a method, apparatus, device, and medium for lip-sound synchronization detection based on cross-modal distillation. Background Technology

[0002] Lip-sync technology aims to determine the consistency between the lip movements of a person in a video and the audio pronunciation in terms of time and content, and has important applications in fintech, healthcare, and other fields. In fintech, services such as remote account opening and online loan dual recording require verification of customer identity; lip-sync can serve as a liveness detection and fraud prevention tool, identifying attacks such as audio-video splicing, AI face-swapping, or dubbing manipulation. In healthcare, during remote consultations, doctors need to observe the patient's pronunciation and lip-sync to aid in the assessment of speech dysfunction rehabilitation; audio-video asynchrony can affect diagnostic judgment.

[0003] However, existing technologies face significant challenges in practical applications: First, they exhibit poor robustness in noisy environments, as audio signals are easily interfered with by background noise such as chatter and horns, disrupting the correlation between audio and video features and leading to misjudgments. Second, they lack precision in detecting subtle lip-sync differences; existing contrastive learning methods typically only assess coarse alignment in the temporal dimension, failing to capture fine-grained differences at the content level, such as the difficulty in distinguishing similar phonemes like the video lip "b" from the audio "p". Finally, they are highly dependent on training data; acquiring massive amounts of precisely labeled synchronized / asynchronous audio and video data is costly, limiting model performance improvement and deployment feasibility. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for lip-sound synchronization detection based on cross-modal distillation, aiming to solve the technical problems of poor robustness, insufficient detection accuracy, and high annotation cost in existing lip-sound synchronization detection methods.

[0005] In a first aspect, embodiments of the present invention provide a method for lip-sync detection based on cross-modal distillation, comprising: acquiring training video data and preprocessing the training video data to generate training audio data, training video frames, and noisy frequency data; generating training teacher audio feature data, teacher soft-label distribution data, training student audio feature data, and student soft-label distribution data based on a preset teacher model and student model according to the training audio data, the training video frames, and the noisy frequency data; training a student model based on a preset dual-constraint knowledge distillation algorithm according to the training teacher audio feature data, the teacher soft-label distribution data, the training student audio feature data, and the student soft-label distribution data; receiving video data to be detected and separating the video data to be detected to generate audio data to be detected and video frames to be detected; generating teacher audio feature data to be detected and student audio feature data to be detected based on the teacher model and the trained student model according to the audio data to be detected and the video frames to be detected; and determining whether the video data to be detected is lip-synced based on the teacher audio feature data to be detected and the student audio feature data to be detected according to a preset judgment algorithm.

[0006] Secondly, embodiments of the present invention also provide a lip-sound synchronization detection device based on cross-modal distillation, which includes a unit for performing the above-described method.

[0007] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0009] This application provides a method, apparatus, device, and medium for lip-phonetic synchronization detection based on cross-modal distillation. Through a dual-constraint knowledge distillation algorithm, it uses the clean audio standard features extracted from a pre-trained speech teacher model with frozen parameters and the distribution of soft labels as distillation benchmarks to train a student model for audio-visual fusion, achieving accurate restoration of clean audio features under noisy frequency and corresponding video frame inputs. Then, during the inference stage, the difference between the restored features of the student model and the benchmark features of the teacher model is used to determine lip-phonetic synchronization. Therefore, this solution does not require additional training of a synchronization classifier, naturally possesses strong noise resistance, and captures deep, fine-grained correlations between lip shape and articulation through soft label distillation, significantly improving the detection accuracy of subtle lip-phonetic synchronization issues and greatly reducing the dependence of model training on high-precision labeled data and the cost of implementation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic flowchart illustrating the lip-sync detection method based on cross-modal distillation provided in this embodiment of the invention;

[0012] Figure 2 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0013] Figure 3 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0014] Figure 4 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0015] Figure 5 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0016] Figure 6 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0017] Figure 7 A schematic diagram of a sub-process of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention;

[0018] Figure 8 A schematic block diagram of a lip-sync detection device based on cross-modal distillation provided in an embodiment of the present invention;

[0019] Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 This is a schematic flowchart of the lip-sound synchronization detection method based on cross-modal distillation provided in this invention. In this application, the lip-sound synchronization detection method based on cross-modal distillation is applied to the field of artificial intelligence, particularly in scenarios such as online face-to-face review for remote account opening in fintech and real-person identity verification for online follow-up consultations in healthcare. For example, in the remote account opening scenario in the fintech field. In this application environment, the system includes a user-end mobile device, a cloud processing server, and a core banking business system. The user-end device is used to collect audio and video data and upload it to the cloud server; the cloud server, as the executing entity in this case, deploys pre-trained student and teacher models, responsible for performing lip-sound synchronization detection; the core banking business system receives the detection results and decides whether to approve the account opening application.

[0025] Users initiate remote account opening via a mobile app, reading aloud a random verification code displayed on the screen. The mobile app records audio and video streams and uploads them to a cloud server. After receiving the video data to be tested, the server performs lip-sync detection on the data sent by the user. If the detection result is lip-sync, it means that visual information has successfully assisted in the reconstruction of audio features, and the audio and video content are consistent. The server then sends an "approved" command to the bank's core system. If the detection result is lip-sync, it means that there is a conflict between the audio and video, which may be a fraudulent activity. The system is judged as out of sync, and an "rejected" command is sent, prompting manual review.

[0026] This method, through a student model trained by dual-constraint knowledge distillation, can accurately distinguish similar phonemes such as "b" and "p" in noisy environments such as background voices in a bank lobby. It solves the problems of poor noise resistance, difficulty in recognizing subtle lip movements, and reliance on a large amount of labeled data in existing technologies, providing a reliable anti-fraud technical guarantee for remote account opening services.

[0027] This application provides a method, apparatus, computer device, and storage medium for lip-sync detection based on cross-modal distillation. The method includes: acquiring training video data and preprocessing the training video data to generate training audio data, training video frames, and noisy frequency data; generating training teacher audio feature data, teacher soft-label distribution data, training student audio feature data, and student soft-label distribution data based on the training audio data, training video frames, and noisy frequency data using a preset teacher model and student model; and generating training teacher audio feature data, teacher soft-label distribution data, training student audio feature data, and student soft-label distribution data based on a preset dual-constraint knowledge distillation algorithm using the training teacher audio feature data, training video frames, and noisy frequency data. The system trains a student model using frequency feature data, teacher soft-label distribution data, training student audio feature data, and student soft-label distribution data; receives video data to be detected and performs separation processing on the video data to be detected to generate audio data to be detected and video frames to be detected; generates teacher audio feature data to be detected and student audio feature data to be detected based on the teacher model and the trained student model, according to the audio data to be detected and the video frames to be detected; and determines whether the video data to be detected is lip-synced based on the teacher audio feature data to be detected and the student audio feature data to be detected using a preset judgment algorithm.

[0028] This application employs a dual-constraint knowledge distillation algorithm. Using the clean audio standard features extracted from a pre-trained speech teacher model with frozen parameters and the distribution of soft labels as distillation benchmarks, it trains a student model for audio-visual fusion, achieving accurate restoration of clean audio features under noisy frequency and corresponding video frame inputs. Then, during the inference phase, the difference between the restored features of the student model and the benchmark features of the teacher model is used to determine lip-phonetic synchronization. Therefore, this scheme eliminates the need for additional synchronization classifier training, inherently possessing strong noise resistance. Simultaneously, by capturing the deep, fine-grained correlation between lip shape and articulation through soft label distillation, it significantly improves the detection accuracy of subtle lip-phonetic asynchrony phenomena, greatly reducing the model's dependence on high-precision labeled data and lowering deployment costs.

[0029] Figure 1 This is a schematic flowchart of the lip-sound synchronization detection method based on cross-modal distillation provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S10-S60.

[0030] S10. Acquire training video data and preprocess the training video data to generate training audio data, training video frames, and noisy frequency data.

[0031] Specifically, the training video data refers to pre-collected, pure audio-video synchronized raw video data, where the lip movements of characters in the video are perfectly aligned with the audio content in time. This training video data is used to train the student model; therefore, the more training video data, the better. Preprocessing refers to a series of operations that parse and transform the raw video data. The training audio data is the raw audio signal separated from the raw video data; the training video frames are image sequences extracted from the raw video data; and the noisy audio data is an audio sample simulating a real environment generated by artificially adding noise to the pure audio (i.e., the training audio data).

[0032] In this embodiment, tens of thousands of clean training video data are first acquired. The training video data is used as training samples, and the number of samples can be expanded according to the model training requirements. The larger the sample size, the better it is to improve the model's generalization performance.

[0033] Subsequently, standardized preprocessing operations were performed on all training video data, extracting the corresponding training audio data and consecutive training video frames from each training video data, and generating accompanying noisy audio data based on the extracted clean training audio data.

[0034] This preprocessing operation can simultaneously acquire clean audio, noisy frequencies, and corresponding video frames as a training dataset, providing a standardized input basis for subsequent model training. Sufficient sample size ensures training data coverage, providing reliable data support for improving the model's noise resistance and detection accuracy.

[0035] In one embodiment, reference is made to Figure 2 Step S10 includes steps S11-S12.

[0036] S11. The training video data is separated to generate the training audio data and the training video frames;

[0037] S12. Add noise with different signal-to-noise ratios to the audio data to form the noisy frequency data.

[0038] Specifically, the separation process refers to parsing and extracting the audio and video streams in the training video data into independent audio files and video frame sequences; the signal-to-noise ratio (SNR) is the ratio of signal power to noise power, used to measure the relative intensity of noise, and is measured in decibels (dB).

[0039] In the implementation of this embodiment, audio and video separation processing is first performed on all the acquired training video data. Through audio and video stream decoding and splitting operations, the corresponding single-channel training audio data and continuous training video frames that are strictly aligned with the audio timing are accurately extracted from each training video data, thus completing the basic splitting and extraction of the original audio and video.

[0040] More specifically, for each training video data segment, a multimedia processing tool (such as FFmpeg) is first used to perform demultiplexing to extract the audio track and save it as the original audio signal in WAV format (i.e., the training audio data). At the same time, the video track is decoded frame by frame at a fixed frame rate (e.g., 25 frames / second) to output an RGB format image frame sequence. The face region is cropped for each frame to obtain training video frames focused on the lip-sync region.

[0041] Subsequently, noise adaptation processing is performed on the pure training audio data obtained from the splitting. Multiple sets of noise with different signal-to-noise ratios are superimposed on the training audio data. The noise includes, but is not limited to, background human voices (such as the noise of a coffee shop), traffic noise (such as car horns and engine sounds), sudden interference sounds (such as the sound of a door closing and keyboard typing), and steady noise (such as Gaussian white noise). Audio processing with multi-gradient noise intensity is completed to generate noisy frequency data that is completely aligned with the time sequence of the original training audio data and corresponds to different noise scenarios.

[0042] Through the above operations, each piece of training video data can generate a set of clean audio (the training audio data), a set of video frame sequences (the training video frames), and dozens of sets of noisy frequency data with different noise types and signal-to-noise ratio combinations.

[0043] This embodiment uses standardized audio and video separation processing to obtain clean audio and video frame data with strict temporal matching, ensuring the basic synchronization of training data and providing the model with a precisely aligned supervision signal. At the same time, by superimposing noise with different signal-to-noise ratios to generate noisy frequencies, a training dataset covering multiple gradient noise intensities can be constructed. This provides input data that fits the real application environment for model training, effectively improving the robustness of the model in complex noisy environments. This lays a solid data foundation for the subsequent high-precision detection of the model, and eliminates the need for additional manual annotation of a large number of noisy samples, effectively reducing data preparation costs.

[0044] S20. Based on the preset teacher model and student model, generate training teacher audio feature data, teacher soft label distribution data, training student audio feature data and student soft label distribution data according to the training audio data, the training video frames and the noisy frequency data;

[0045] Specifically, the teacher model refers to a feature extraction network constructed using a pre-trained speech base model (such as WavLM), whose parameters are frozen during training and used only to extract high-level speech representations from clean audio. The student model refers to a multimodal network comprising an audio front-end, a visual front-end, and a fusion encoder, used to learn robust feature representations from noisy audio and video frames. In this embodiment, the audio front-end of the student model consists of several layers of convolutional neural networks and a temporal modeling module (such as Transformer or LSTM), the visual front-end uses a 3D convolutional neural network to extract lip movement features, and the fusion encoder achieves the alignment and fusion of audio and video features through a cross-modal attention mechanism.

[0046] The training teacher audio feature data refers to the deep feature vectors extracted by the teacher model from the pure training audio, which reflects the semantic information of the audio at multiple levels of abstraction; the teacher soft label distribution data refers to the probability distribution of the speech content category output by the teacher model, which reflects the model's fine-grained judgment of the audio content.

[0047] The training student audio feature data refers to the high-dimensional feature vector calculated by the student model through an internal fusion encoder after receiving noisy frequencies and corresponding video frames. It is a simulation or restoration result of the pure audio features output by the teacher model. The student soft label distribution data refers to the content category probability distribution output by the student model based on noisy frequencies and video frames, reflecting the model's fine-grained judgment of audio content.

[0048] In the specific implementation process, the training audio data is input into the teacher model, and the training teacher audio feature data is generated through feature extraction, while the teacher soft label distribution data is output simultaneously.

[0049] Simultaneously, the noisy frequency data and the training video frames are input into the student model. That is, the noisy frequency data is input into the audio front end of the student model, and the full-face image sequence corresponding to the training video frames aligned with the audio time sequence is input into the visual front end of the student model. The features extracted by the two front ends are sent to the fusion encoder to complete cross-modal fusion calculation, generate training student audio feature data, and synchronously output student soft label distribution data.

[0050] This embodiment provides a highly reliable distillation learning benchmark through a pre-trained speech-based teacher model with parameters frozen throughout the training process, significantly reducing the reliance on high-precision labeled data during student model training. Furthermore, through cross-modal fusion processing of the audiovisual network student model, which includes an audio front-end, a visual front-end, and a fusion encoder, it achieves the simulated restoration of clean audio features in noisy scenes. The high-dimensional feature vector output by the fusion encoder serves as the training student audio feature data, ensuring that the audio features output by the student model are consistent with the teacher audio feature data in the feature space. This effectively improves the noise robustness and fine-grained lip-sync detection accuracy of the student model in complex noisy environments.

[0051] In one embodiment, reference is made to Figure 3 Step S20 includes steps S21-S22.

[0052] S21. The teacher model performs multi-level feature extraction on the training audio data to generate the training teacher audio feature data, and then performs clustering calculation and function transformation on the training teacher audio feature data to generate the teacher soft label distribution data.

[0053] S22. The student model calculates the training student audio feature data based on the noisy frequency data and the training video frames through an internal fusion encoder, and then performs clustering calculation and function transformation on the training student audio feature data to generate the student soft label distribution data.

[0054] In this embodiment, the teacher soft-label distribution data refers to the probability distribution obtained by calculating the distance from each frame feature to each cluster center after K-means clustering of teacher audio features, and then transforming it through the Softmax function. Its essence is the soft attribution representation of audio frames in the feature space. The student soft-label distribution data refers to the probability distribution obtained after performing the same clustering calculation and Softmax transformation on student audio features, reflecting the student model's clustering attribution judgment of audio content.

[0055] In the specific implementation process, the preprocessed clean training audio data is input into the teacher model. The teacher model performs feature extraction layer by layer through its multi-layer Transformer structure and outputs training teacher audio feature data, which contains the feature vector corresponding to each audio frame.

[0056] Subsequently, K-means clustering is performed on the audio feature data of the training teachers in all training data. The number of cluster centers is preset to K (e.g., K=100), resulting in K cluster center vectors. These centers are used in subsequent steps. The cluster centers refer to the representative "semantic benchmark" (similar to standard phonemes) obtained in the pre-training stage by performing K-means algorithm on massive speech features (extracted by WavLM).

[0057] In the specific calculation, for each frame of teacher audio feature vector (i.e. each training teacher audio feature data), the Euclidean distance between it and K cluster centers is calculated to obtain K distance values. The distance values ​​are input into the Softmax function for normalization, which is transformed into a probability distribution with a sum of 1. The closer the distance, the higher the probability. This generates teacher soft label distribution data. This distribution retains the similarity information between phonemes in a probabilistic form. For example, the features of the pronunciation "b" and "p" will be very similar in distribution.

[0058] Furthermore, the noisy audio data is input into the audio front end of the student model, while the training video frame sequence is input into the visual front end. The audio front end extracts preliminary audio features, and the visual front end uses, for example, a 3D convolutional network to extract lip-sync features. The two are cross-modal interacted and aligned in the fusion encoder. The fusion encoder outputs training student audio feature data, and this feature vector is consistent with the teacher's audio features in terms of dimension.

[0059] Then, using the K cluster centers determined in the teacher model clustering stage, the Euclidean distance from the training student audio feature vector of each frame to each cluster center is calculated, and the distance value is input into the Softmax function for normalization to generate student soft label distribution data. This distribution also reflects the student features in the cluster space in a probabilistic form.

[0060] This embodiment preserves the similarity information between phonemes through soft clustering label distribution. Compared with hard labels, soft distribution can better preserve the similarity information between phonemes. For example, the feature distributions of "b" and "p" are similar rather than completely different, enabling the student model to capture subtle mouth shape and pronunciation associations, effectively improving the detection accuracy of fine-grained lip-phoneme synchronization differences. At the same time, the frozen teacher model provides a reliable distillation benchmark, which greatly reduces the dependence and cost of model training on high-precision labeled data. Through cross-modal fusion of the student model, the accurate restoration of clean audio features in noisy scenes is achieved, improving the model's noise robustness and realizing high-precision lip-phoneme synchronization detection.

[0061] S30. Based on the preset dual-constraint knowledge distillation algorithm, train the student model according to the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data;

[0062] This embodiment describes the student model training stage of a lip-sync detection method based on cross-modal distillation. The dual-constraint knowledge distillation algorithm is a model compression and transfer learning method. Pre-defined in the system, it allows a lightweight student model to mimic the output behavior of a large teacher model, thereby transferring knowledge from the teacher model to the student model. This embodiment employs a dual-constraint cross-modal knowledge distillation algorithm that combines feature level and soft label distribution level.

[0063] In this embodiment, based on a preset dual-constraint cross-modal knowledge distillation algorithm, the audio feature data of the training teacher and the audio feature data of the training student are used as inputs to the feature level, and the Euclidean distance between them is calculated as the feature distillation loss; the soft label distribution data of the teacher and the soft label distribution data of the student are used as inputs to the soft label distribution level, and the KL divergence between them is calculated as the soft label distribution distillation loss; the feature distillation loss and the soft label distribution distillation loss are weighted and summed according to preset weights to obtain the total loss function; based on the total loss function, all trainable parameters of the audio front-end, visual front-end and fusion encoder in the student model are updated through the backpropagation algorithm until the total loss function converges to a preset threshold or reaches a preset number of training iterations, thus completing the training of the student model.

[0064] This embodiment uses a dual-constraint distillation mechanism to enable the student model to simultaneously learn the teacher model's ability to extract pure audio features and fine-grained phoneme similarity information. This effectively improves the student model's noise robustness in complex noise environments and the detection accuracy of fine-grained lip-sync differences, significantly reducing the model's dependence on and cost of massive amounts of high-precision labeled data for training.

[0065] In one embodiment, reference is made to Figure 4 Step S30 includes steps S31-S33.

[0066] S31. Calculate the total loss of the student model based on the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data;

[0067] S32. The total loss is transmitted back to the student model through the backpropagation algorithm, and the student model continuously updates its network weights using the optimizer based on the total loss.

[0068] S33. Iteratively execute the steps of calculating the total loss and updating the network weights until the total loss converges to a preset minimum threshold or reaches a preset number of iterations.

[0069] Specifically, the backpropagation algorithm is the mechanism for calculating gradients and updating parameters during neural network training; the optimizer (such as Adam) is used to adjust network weights according to the gradients so that the total loss gradually decreases.

[0070] For example, in application scenarios such as remote online face review in fintech and real-person identity verification in remote medical consultation, this embodiment takes the training teacher audio feature data, teacher soft label distribution data, training student audio feature data, and student soft label distribution data adapted to the business scenario as input. First, it calculates two types of loss according to the preset formula. In this embodiment, it refers to feature regression loss and KL divergence loss. Then, it sums them up according to the preset weights to obtain the total loss.

[0071] Then, the error signal corresponding to the total loss is transmitted back to the student model through the backpropagation algorithm. An optimizer is used, an initial learning rate is set (e.g., 1e-4), and the weights of the student model's fully trainable network are iteratively updated. The total loss calculation and weight update steps are continuously executed iteratively until the total loss converges to the preset minimization threshold (e.g., 1e-6) or reaches the preset maximum number of iterations (e.g., 100,000 rounds), thus completing the training of the student model to achieve the minimization of the total loss.

[0072] This embodiment uses a two-dimensional loss constraint to force the student model output to be highly consistent with the teacher model in terms of feature values ​​and probability distribution. The student model is trained to recover pure speech features with the help of visual information under noise interference, which effectively improves the model's noise robustness and fine-grained detection accuracy in real noisy business scenarios. It also significantly reduces the model training's dependence on massive amounts of accurately labeled data, making it suitable for application in business scenarios with high security requirements.

[0073] In one embodiment, reference is made to Figure 5 The total loss includes feature regression loss and KL divergence loss, and step S31 includes steps S311-S313.

[0074] S311. Calculate the feature regression loss of the student model based on the audio feature data of the training teacher and the audio feature data of the training student.

[0075] S312. Calculate the KL divergence loss of the student model based on the teacher soft label distribution data and the student soft label distribution data;

[0076] S313. The feature regression loss and the KL divergence loss are weighted and summed to generate the total loss.

[0077] Specifically, the total loss includes feature regression loss and KL divergence loss, wherein the feature regression loss This refers to using mean squared error (MSE) to calculate the audio feature data used in training students. Audio feature data of training teachers The loss function, calculated as the squared Euclidean distance between them, is expressed mathematically as follows:

[0078] ,

[0079] The feature regression loss is used to force students to approximate the teacher's feature representation at the numerical level.

[0080] The KL divergence loss This refers to data used to measure the distribution of soft labels on students. Teacher soft label distribution data The loss function for the difference between them is expressed mathematically as follows:

[0081] ,

[0082] The KL divergence loss is used to force students to approximate the teacher's soft-label output at the probability distribution level.

[0083] The weighted summation refers to adding the feature regression loss and the KL divergence loss according to preset weights to form a total loss function. The total loss is the final supervision signal used to jointly optimize the student model.

[0084] In the stage of calculating the total loss, the feature regression loss is first calculated based on the audio feature data of the training teachers and the audio feature data of the training students. That is, the mean square error of the feature vector of each frame is calculated, and the average value is taken after summing frame by frame as the feature regression loss value of the current batch.

[0085] Next, the KL divergence loss is calculated based on the distribution data of teacher soft labels and student soft labels. That is, the KL divergence is calculated for the two probability distributions of each frame, and the average value is taken after summing frame by frame as the KL divergence loss value of the current batch.

[0086] The two losses are then weighted and summed according to preset weights to obtain the total loss.

[0087] After calculating the total loss, the error signal is transmitted back to the student model through the backpropagation algorithm, and the optimizer is used to iteratively update the parameters of the student model with a preset learning rate and a preset batch size.

[0088] In each iteration, repeat the steps of calculating the total loss and updating the weights until the total loss converges to the preset minimum threshold or reaches the preset number of iterations, thereby minimizing the total loss.

[0089] This embodiment employs a dual constraint of feature regression loss and KL divergence loss. During training, the student model is forced to recover clean audio features from noisy input and mimics the teacher's fine-grained partitioning of the phoneme space, thus maintaining high-precision evaluation even under various environmental noise interferences. For example, in remote speech rehabilitation assessment scenarios in the medical and health field, when the monitoring device's prompts mask part of the speech, the student model can still accurately determine whether the patient's pronunciation is "ba" rather than "pa" through visual lip-reading information, ensuring the reliability of the rehabilitation assessment.

[0090] Meanwhile, the training process does not require manual annotation of noisy data, which significantly reduces the cost of data acquisition. This allows student models to quickly adapt to the specific environmental noise characteristics of different hospitals or financial audits, providing strong support for the accuracy and user experience of remote medical services or financial account opening interviews.

[0091] S40. Receive the video data to be detected, and perform separation processing on the video data to be detected to generate audio data to be detected and video frames to be detected.

[0092] Specifically, the video data to be detected refers to video files or video streams collected from actual application scenarios that require lip-sync judgment. These may contain complex factors such as environmental noise and compression artifacts, or they may be pure audio and video data. The separation processing refers to using multimedia processing tools to parse and extract the audio and video tracks in the video file into independent audio files and video frame sequences. The audio data to be detected refers to the original audio signal separated from the video data to be detected, which will be used as the audio input to the trained student model. The video frames to be detected refer to the key image sequences extracted from the video data to be detected, which will be used as the visual input to the trained student model.

[0093] After receiving the video data to be detected, the system first performs a separation operation to generate audio data and video frames to be detected.

[0094] For example, in a remote speech rehabilitation assessment scenario in the healthcare field, patients record pronunciation training videos using a rehabilitation app and upload them to a cloud server. The server receives these videos as data to be tested. The videos may contain ambient noise from hospital monitoring equipment, hallway conversations, or family members talking. The server-side multimedia processing module first separates the audio and video data, extracting the audio data and converting it to the format required by the model. Simultaneously, video frames are extracted at the same frame rate, and the mouth region is cropped using a pre-trained facial landmark detection model to ensure that the image size of each frame is consistent with that during the training phase (e.g., 96×96 pixels). The processed audio data and video frames to be tested are stored in chronological order, ready to be input into the trained student model for synchronization judgment.

[0095] For example, in remote account opening scenarios within the fintech field, when a user initiates a remote account opening application via a mobile app, the system captures a video stream of the user reading a random verification code in real time as the video data to be detected. This video data may contain environmental interference such as background noise from the bank lobby, keyboard clicks, or air conditioning hum. The system calls a multimedia processing module (such as FFmpeg) to demultiplex the video data to be detected, extracting the audio track and saving it as a 16kHz sampling rate, mono WAV file as the audio data to be detected. Simultaneously, the video track is decoded frame by frame at a fixed frame rate (e.g., 25 frames / second), outputting an RGB format image frame sequence. A face detection algorithm is then used to locate the face region in each frame, and image blocks containing the mouth are cropped out as the video frames to be detected. The cropping operation removes background interference, focusing on the area of ​​lip movement changes, improving the accuracy of subsequent detection.

[0096] This embodiment standardizes and preprocesses the video to be detected, ensuring that the input data format is consistent with that of the training phase. This eliminates format differences caused by different acquisition devices, laying the foundation for subsequent high-precision detection. Furthermore, in real-world scenarios such as fintech and healthcare, this step serves as the entry point for the inference process, guaranteeing the reliability and real-time performance of the lip-sync detection system in complex real-world environments, and providing standardized input data for subsequent fraud detection or rehabilitation assessment.

[0097] S50. Based on the teacher model and the trained student model, generate audio feature data of the teacher to be detected and audio feature data of the student to be detected according to the audio data to be detected and the video frame to be detected.

[0098] Specifically, the teacher audio feature data to be detected refers to the sequence of high-dimensional feature vectors extracted by the teacher model from the audio to be detected; the student audio feature data to be detected refers to the high-dimensional feature vectors calculated by the student model through a fusion encoder after receiving the audio to be detected and the corresponding video frame.

[0099] In specific implementation, the audio data to be detected is input into the teacher model. The teacher model first performs frame segmentation and feature transformation on the audio, and then extracts temporal dependent features layer by layer through its multi-layer Transformer structure, finally outputting the teacher audio feature data to be detected. This data is a feature vector corresponding to each audio frame, with the dimension consistent with that of the training phase. Simultaneously, the audio data to be detected and the video frames to be detected are input into the trained student model. The student model's audio front end receives the audio to be detected, extracts local acoustic features through convolutional layers, and inputs them into the temporal module to model the context information, outputting an audio feature representation. The visual front end receives the video frame sequence to be detected, captures the motion features of lip movements changing over time through a 3D convolutional network, and outputs a visual feature representation. The audio and visual features undergo cross-modal interaction and alignment in the fusion encoder. The fusion encoder uses a multi-head attention mechanism to enhance the features of the two modalities, finally outputting the student audio feature data to be detected. This data is also a frame-level feature vector with the dimension consistent with the teacher features. The above two feature data are stored in time alignment for subsequent synchronization judgment.

[0100] This embodiment extracts features in parallel using dual models, providing a two-dimensional comparison basis for subsequent lip-sync detection, effectively avoiding detection errors caused by noise interference, and ensuring fine-grained detection accuracy. In real-person identity verification scenarios such as remote online face-to-face review in fintech and remote medical consultation, it can be directly adapted to business data collection, quickly complete feature extraction, and support accurate detection of subsequent audio and video tampering violations.

[0101] S60. Based on a preset judgment algorithm, determine whether the video data to be detected is lip-synced according to the audio feature data of the teacher to be detected and the audio feature data of the student to be detected.

[0102] Specifically, the teacher audio feature data to be detected refers to the deep feature vector sequence extracted by the teacher model from the audio to be detected, representing a reference benchmark for clean audio; the student audio feature data to be detected refers to the high-dimensional feature vector sequence jointly extracted by the trained student model from the audio and video frames to be detected, incorporating visual lip-sync information to resist noise interference; the preset judgment algorithm refers to the calculation rules used to measure the consistency between teacher features and student features, which usually includes two steps: similarity calculation and threshold comparison. The similarity calculation can use measures such as cosine similarity or Euclidean distance.

[0103] In this embodiment, a preset judgment algorithm is provided in the system to judge the audio feature data of the teacher to be detected and the audio feature data of the student to be detected according to the preset judgment algorithm in order to detect whether the video data to be detected is lip-synced.

[0104] This embodiment effectively captures feature deviations caused by audio-visual asynchrony in noisy environments by comparing the clean audio benchmark provided by the teacher model (i.e., the teacher audio feature data to be detected) with the features output by the student model after fusing visual information (i.e., the student audio feature data to be detected). The teacher audio feature data to be detected represents the "clean sound that should be heard," while the student audio feature data to be detected represents the "sound understood after combining the seen lip movements." When the two are highly consistent, it indicates that the visual information matches the audio content; when there is a significant deviation, it suggests possible misjudgment due to dubbing alteration, lip-syncing errors, or environmental noise. This achieves accurate identification of lip-sync in complex scenarios.

[0105] In one embodiment, reference is made to Figure 6 Step S60 includes steps S61-S62.

[0106] S61. Calculate the distance metric between the audio feature data of the teacher to be detected and the audio feature data of the student to be detected;

[0107] S62. Based on the preset judgment logic, determine whether the video data to be detected is lip-synced according to the distance metric value.

[0108] Specifically, the distance metric refers to a mathematical index used to quantify the difference between two feature vectors. In this embodiment, negative cosine similarity or Euclidean distance is used for calculation. The negative cosine similarity value ranges from [-1, 1] and the larger the value, the greater the difference. The Euclidean distance is a non-negative value and the larger the value, the greater the difference. The preset judgment logic refers to the comparison rule based on the distance metric and the preset threshold, as well as the optional continuous frame statistical rule, used to finally output the judgment result of lip-sync.

[0109] In the specific implementation process, the audio feature data of the teacher to be detected, corresponding to the audio data to be detected, and the audio feature data of the student to be detected aligned on the same time axis are acquired. Both are frame-level feature vector sequences with the same dimension. For each feature pair corresponding to a video frame, the distance metric between the two is first calculated. In this implementation, negative cosine similarity or Euclidean distance is used for calculation. Subsequently, the calculated distance metric is used to determine whether there is lip-sync based on a preset judgment logic.

[0110] This embodiment achieves an objective quantitative judgment of lip-sync state by calculating the distance metric between teacher and student features and applying a preset judgment logic. Teacher features represent an ideal benchmark for pure audio, while student features incorporate visual information to resist noise. In a synchronized state, both should remain close in the feature space; in an asynchronous state, they will deviate significantly. This feature space distance-based judgment method, compared to directly comparing the original signal, can more effectively capture subtle differences caused by dubbing alterations, lip-sync misalignment, or noise interference, significantly improving the accuracy and robustness of the detection.

[0111] In one embodiment, reference is made to Figure 7 Step S62 includes steps S621-S622.

[0112] S621. If the distance metric value is less than or equal to the preset synchronization threshold, then the video data to be detected is determined to be lip-synced.

[0113] S622. If the distance metric value is greater than the preset synchronization threshold, it is determined that the audio and lip movements of the video data to be detected are out of sync.

[0114] Specifically, the preset synchronization threshold refers to a pre-set distance threshold used to distinguish between lip-sync and asynchronous states. Its value is determined on the validation set by statistically analyzing the distance distribution of normal synchronized samples.

[0115] In this embodiment, the distance metric value corresponding to the video data to be detected is first obtained, and a pre-calibrated preset synchronization threshold is retrieved; the distance metric value is then compared with the preset synchronization threshold.

[0116] If the distance metric of the current frame is less than or equal to the preset synchronization threshold, it means that the teacher's features and the student's features are highly consistent in direction, that is, the visual information successfully assisted in the restoration of the audio features, and the audio content matches the lip-sync content. Therefore, the current frame is determined to be lip-synced.

[0117] If the distance metric of the current frame is greater than the preset synchronization threshold, it indicates that there is a significant directional deviation between the teacher's features and the student's features, that is, there is a conflict between visual information and audio information, which causes the student model to be unable to recover features consistent with the teacher from the noisy frequency. This may be due to dubbing alteration, lip-syncing misalignment, or noise interference. Therefore, it is determined that the current frame is out of sync with the audio.

[0118] This embodiment achieves reproducible and quantifiable judgment of lip-sync state through standardized thresholding logic. It can accurately capture fine-grained differences between audio and video, avoid misjudgments caused by environmental noise, and improve the stability and accuracy of detection results. In scenarios such as online face-to-face review for remote account opening in financial technology and real-person identity verification for online medical follow-up visits, the threshold can be flexibly adjusted according to the business security level to achieve accurate and efficient detection of violations such as audio and video tampering and identity theft.

[0119] For example, in a remote account opening scenario in the fintech field, the system collects audio and video of the user reading the verification code in real time. After processing, it obtains the audio feature data of the teacher and the student to be detected, and calculates the distance metric value for each frame. If the distance metric value is consistently less than or equal to a preset synchronization threshold, it is determined that the user's lip-sync is correct, and the liveness detection is passed. If the distance value suddenly increases beyond the preset synchronization threshold at a certain time, it is determined that there may be audio / video splicing or dubbing tampering, and the system refuses to open the account and prompts for manual review.

[0120] In another embodiment, in a remote rehabilitation assessment scenario in the medical and health field, if the distance value of multiple consecutive frames of a patient's pronunciation training video exceeds a preset synchronization threshold after being processed through the same process, the system will prompt that the patient's pronunciation and lip movements are not coordinated, and suggest re-recording or contacting a doctor for further assessment to ensure the accuracy of rehabilitation data.

[0121] This application comprises two stages: a model pre-training stage and a synchronous detection and inference stage. First, a dual-constraint knowledge distillation algorithm is used to train a student model for audio-visual fusion, based on the clean audio standard features extracted from the pre-trained speech teacher model with frozen parameters and the distribution of soft labels. This achieves accurate reconstruction of clean audio features under noisy frequencies and corresponding video frame inputs. Then, in the inference stage, the difference between the features reconstructed by the student model and the baseline features of the teacher model is used to determine lip-phonetic synchronization. Therefore, this scheme eliminates the need for additional training of a synchronization classifier, inherently possessing strong noise resistance. Simultaneously, by capturing the deep, fine-grained correlation between lip shape and articulation through soft label distillation, it significantly improves the detection accuracy of subtle lip-phonetic asynchrony phenomena, greatly reducing the model training's dependence on high-precision labeled data and lowering deployment costs.

[0122] Figure 8 This is a schematic block diagram of a lip-sync detection device 300 based on cross-modal distillation provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described method for lip-phoneme synchronization detection based on cross-modal distillation, the present invention also provides a device 300 for lip-phoneme synchronization detection based on cross-modal distillation. This device 300 includes a unit for performing the above-described method for lip-phoneme synchronization detection based on cross-modal distillation, and can be configured in a computer device. Specifically, please refer to... Figure 8 The lip-phonetic synchronization detection device 300 based on cross-modal distillation includes a preprocessing unit 301, a first generation unit 302, a training unit 303, a separation unit 304, a second generation unit 305, and a judgment unit 306.

[0123] The preprocessing unit 301 is used to acquire training video data and preprocess the training video data to generate training audio data, training video frames and noisy frequency data.

[0124] The first generation unit 302 is used to generate training teacher audio feature data, teacher soft label distribution data, training student audio feature data, and student soft label distribution data based on the preset teacher model and student model, according to the training audio data, the training video frames, and the noisy video data.

[0125] Training unit 303 is used to train the student model based on a preset dual-constraint knowledge distillation algorithm according to the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data;

[0126] The separation unit 304 is used to receive the video data to be detected and to perform separation processing on the video data to be detected to generate audio data to be detected and video frames to be detected.

[0127] The second generation unit 305 is used to generate audio feature data of the teacher to be detected and audio feature data of the student to be detected based on the teacher model and the trained student model, according to the audio data to be detected and the video frame to be detected.

[0128] The judgment unit 306 is used to determine whether the video data to be detected is lip-synced based on the audio feature data of the teacher to be detected and the audio feature data of the student to be detected, according to a preset judgment algorithm.

[0129] In one embodiment, the preprocessing unit 301 includes a first separation unit and a construction unit.

[0130] The first separation unit is used to separate the training video data to generate the training audio data and the training video frames;

[0131] The construction unit is used to superimpose noise of different signal-to-noise ratios onto the audio data to form the noisy frequency data.

[0132] In one embodiment, the first generation unit 302 includes a teacher unit and a student unit.

[0133] The teacher unit is used by the teacher model to perform multi-layer feature extraction on the training audio data to generate the training teacher audio feature data, and then to perform clustering calculation and function transformation on the training teacher audio feature data to generate the teacher soft label distribution data.

[0134] The student unit is used by the student model to calculate the training student audio feature data through an internal fusion encoder based on the noisy frequency data and the training video frames, and then perform clustering calculation and function transformation on the training student audio feature data to generate the student soft label distribution data.

[0135] In one embodiment, the training unit 303 includes a first calculation unit, an update unit, and an iteration unit.

[0136] The first calculation unit is used to calculate the total loss of the student model based on the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data;

[0137] An update unit is used to transmit the total loss back to the student model through a backpropagation algorithm, and the student model continuously updates its network weights using an optimizer based on the total loss.

[0138] An iterative unit is used to iteratively execute the steps of calculating the total loss and updating the network weights until the total loss converges to a preset minimization threshold or reaches a preset number of iterations.

[0139] In one embodiment, the first calculation unit includes a feature loss unit, a soft distribution loss unit, and a summation unit.

[0140] The feature loss unit is used to calculate the feature regression loss of the student model based on the training teacher audio feature data and the training student audio feature data.

[0141] A soft-distribution loss unit is used to calculate the KL divergence loss of the student model based on the teacher soft-label distribution data and the student soft-label distribution data.

[0142] The summation unit is used to perform a weighted summation of the feature regression loss and the KL divergence loss to generate the total loss.

[0143] In one embodiment, the judgment unit 306 includes a distance calculation unit and a judgment unit.

[0144] The distance calculation unit is used to calculate the distance metric between the audio feature data of the teacher to be detected and the audio feature data of the student to be detected.

[0145] The determination unit is used to determine whether the video data to be detected is lip-synced based on the distance metric value according to the preset determination logic; it is used to determine that the video data to be detected is lip-synced if the distance metric value is less than or equal to a preset synchronization threshold; and to determine that the video data to be detected is not lip-synced if the distance metric value is greater than the preset synchronization threshold.

[0146] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned transmodal distillation-based lip-sync detection device and its various units can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0147] The aforementioned lip-sync detection device 300 based on cross-modal distillation can be implemented as a computer program, which can be used in, for example... Figure 9 It runs on the computer device shown.

[0148] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0149] See Figure 9The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0150] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a lip-sync detection method based on cross-modal distillation.

[0151] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0152] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a lip-sync detection method based on cross-modal distillation.

[0153] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0154] The processor 502 is used to run the computer program 5032 stored in the memory to implement the steps of the above-mentioned lip-sync detection method based on cross-modal distillation.

[0155] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0156] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0157] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps of the above-described lip-sync detection method based on cross-modal distillation.

[0158] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0160] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0161] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0163] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0164] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for lip-sync detection based on cross-modal distillation, characterized in that, The method includes: Acquire training video data and preprocess the training video data to generate training audio data, training video frames, and noisy frequency data; Based on the preset teacher and student models, training teacher audio feature data, teacher soft label distribution data, training student audio feature data, and student soft label distribution data are generated according to the training audio data, the training video frames, and the noisy video data. The student model is trained based on the preset dual-constraint knowledge distillation algorithm, which uses the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data. Receive the video data to be detected, and perform separation processing on the video data to be detected to generate audio data and video frames to be detected; Based on the teacher model and the trained student model, audio feature data of the teacher to be detected and audio feature data of the student to be detected are generated according to the audio data to be detected and the video frames to be detected. Based on a preset judgment algorithm, it is determined whether the video data to be detected is lip-synced according to the audio feature data of the teacher to be detected and the audio feature data of the student to be detected. The step of generating training teacher audio feature data, teacher soft label distribution data, training student audio feature data, and student soft label distribution data based on the preset teacher model and student model according to the training audio data, the training video frames, and the noisy frequency data includes: The teacher model performs multi-level feature extraction on the training audio data to generate the training teacher audio feature data, and then performs clustering calculation and function transformation on the training teacher audio feature data to generate the teacher soft label distribution data. The student model calculates the training student audio feature data based on the noisy frequency data and the training video frames through an internal fusion encoder, and then performs clustering calculation and function transformation on the training student audio feature data to generate the student soft label distribution data.

2. The method according to claim 1, characterized in that, The steps of the pre-defined dual-constraint knowledge distillation algorithm for training the student model based on the training teacher audio feature data, the teacher soft-label distribution data, the training student audio feature data, and the student soft-label distribution data include: The total loss of the student model is calculated based on the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data. The total loss is transmitted back to the student model through the backpropagation algorithm, and the student model continuously updates its network weights using the optimizer based on the total loss. The steps of calculating the total loss and updating the network weights are executed iteratively until the total loss converges to a preset minimum threshold or reaches a preset number of iterations.

3. The method according to claim 2, characterized in that, The total loss includes feature regression loss and KL divergence loss. The step of calculating the total loss of the student model based on the training teacher audio feature data, the teacher soft label distribution data, the training student audio feature data, and the student soft label distribution data includes: Calculate the feature regression loss of the student model based on the audio feature data of the training teachers and the audio feature data of the training students; The KL divergence loss of the student model is calculated based on the teacher soft label distribution data and the student soft label distribution data. The feature regression loss and the KL divergence loss are weighted and summed to generate the total loss.

4. The method according to claim 1, characterized in that, The steps of determining whether the video data to be detected is lip-synced based on the audio feature data of the teacher to be detected and the audio feature data of the student to be detected according to the preset judgment algorithm include: Calculate the distance metric between the audio feature data of the teacher to be detected and the audio feature data of the student to be detected; Based on a preset judgment logic, the system determines whether the video data to be detected is lip-synced according to the distance metric.

5. The method according to claim 4, characterized in that, The step of determining whether the video data to be detected is lip-synced based on the distance metric value according to the preset judgment logic includes: If the distance metric is less than or equal to a preset synchronization threshold, then the video data to be detected is determined to be lip-synced. If the distance metric is greater than the preset synchronization threshold, it is determined that the audio and lip movements of the video data to be detected are out of sync.

6. The method according to claim 1, characterized in that, The step of preprocessing the training video data to generate training audio data, training video frames, and noisy video data includes: The training video data is separated to generate the training audio data and the training video frames; The training audio data is superimposed with noise of different signal-to-noise ratios to form the noisy frequency data.

7. A lip-sync detection device based on cross-modal distillation, characterized in that, Includes a unit for performing the method as described in any one of claims 1-6.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the method as described in any one of claims 1-6.

Images