Lip sound synchronous detection and calibration method and device, computer equipment and storage medium

Abstract

The invention belongs to the interdisciplinary field of computer vision, audio signal processing and artificial intelligence, and relates to a lip sound synchronous detection and calibration method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining to-be-detected audio and video data, and carrying out the separation and preprocessing of the audio and video data; performing lip region detection and extraction on the video frame sequence to obtain a lip image sequence; performing multi-view lip shape reconstruction and view normalization on the lip image sequence to generate a front lip map sequence; constructing an audio and video spatio-temporal feature tensor, and generating an audio tensor and a video tensor; performing bimodal heterogeneous three-dimensional convolutional coupling network feature mapping, and calculating a measurement distance between the audio feature vector and the video feature vector; and judging whether the audio and video data has the voice lip movement consistency or not according to the relationship between the measurement distance and a preset judgment threshold. The problems of lip feature deformation and difficult extraction caused by non-front view angles such as side face and head lowering in a digital human teaching video are systematically solved.

Description

Methods, apparatus, computer equipment and storage media for lip-sound synchronization detection and calibration Technical Field

[0001] This invention relates to the interdisciplinary technical field of computer vision, audio signal processing and artificial intelligence, and in particular to a method, apparatus, computer equipment and storage medium for lip-sync detection and calibration. Background Technology

[0002] With the deep integration of digital human technology and online education, digital human course videos have become an important medium for remote teaching, personalized tutoring, and vocational training. Their teaching effectiveness and learner experience directly depend on the synchronization accuracy between the digital human's lip movements and speech content; therefore, lip-sound synchronization quality has become a key indicator for measuring the quality of digital human course videos. Although deep learning methods (such as CNN, 3D-CNN, and GAN) have provided support for digital human video technology, lip-sound asynchrony remains a common problem in practical applications, severely hindering the promotion and application of digital human teaching videos.

[0003] Existing lip-sync technologies have the following main drawbacks: strong viewpoint limitation: most methods assume that the input is a frontal face video, which cannot effectively handle the multi-view situations such as side faces, head down, and turning around that are common in digital human teaching videos, resulting in difficulty in feature extraction; insufficient feature fusion: existing methods mostly use a single modality or simple splicing method to fuse audio and video features, which fails to achieve deep semantic alignment and is susceptible to noise interference. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for lip-sound synchronization detection and calibration, employing the following technical solution, comprising the following steps: S1, acquiring audio and video data to be detected, and separating and preprocessing the audio and video data; S2, performing lip region detection and extraction on the preprocessed video frame sequence to obtain a lip image sequence; S3, performing multi-view lip shape reconstruction and viewpoint normalization on the lip image sequence to generate a frontal lip image sequence; S4, constructing an audio-video spatiotemporal feature tensor based on the frontal lip image sequence to generate an audio tensor and a video tensor; S5, performing feature mapping on a dual-modal heterogeneous three-dimensional convolutional coupled network based on the audio tensor and the video tensor, and calculating the metric distance between the audio feature vector and the video feature vector; S6, determining whether the audio and video data possesses speech lip movement consistency based on the relationship between the metric distance and a preset decision threshold.

[0005] Preferably, step S1, acquiring the audio and video data to be detected and separating and preprocessing the audio and video data, specifically includes: S11, acquiring the audio and video data to be detected and performing format verification on the audio and video data; S12, separating the audio and video streams of the audio and video data after format verification; and S13, performing video frame serialization and audio buffering.

[0006] Preferably, step S2, which involves detecting and extracting the lip region from the preprocessed video frame sequence to obtain a lip image sequence, specifically includes: S21, performing frame-by-frame face detection and key point localization on the preprocessed video frame sequence; S22, performing adaptive cropping of the lip region of interest based on key point localization; and S23, standardizing and outputting the lip image sequence.

[0007] Preferably, step S3, which involves reconstructing the lip shape from multiple perspectives and normalizing the perspective of the lip image sequence to generate a frontal lip image sequence, specifically includes: S31, performing lip pose angle recognition and feature decoupling based on the lip image sequence; S32, performing multi-condition feature fusion to generate a frontal lip image sequence; and S33, performing adversarial training and multi-objective optimization on the frontal lip image sequence.

[0008] Preferably, step S4, which involves constructing an audio-video spatiotemporal feature tensor based on the frontal lip image sequence, and generating an audio tensor and a video tensor, specifically includes: S41, slicing the audio stream into segments of fixed duration, extracting spectral features, and generating an audio three-dimensional feature tensor; S42, truncating and stacking the video stream into frames according to corresponding time windows to generate a video three-dimensional spatiotemporal tensor; and S43, associating the audio three-dimensional feature tensor and the video three-dimensional spatiotemporal tensor to form a paired audio-video tensor pair.

[0009] Preferably, step S5, which involves performing feature mapping of a dual-modal heterogeneous three-dimensional convolutional coupled network based on the audio tensor and the video tensor, and calculating the metric distance between the audio feature vector and the video feature vector, specifically includes: S51, setting up a heterogeneous three-dimensional convolutional sub-network based on the audio tensor and the video tensor, and extracting the audio feature vector and the video feature vector; S52, performing joint training based on the coupling contrastive loss function; and S53, calculating the metric distance between the audio feature vector and the video feature vector.

[0010] Preferably, step S6, determining whether the audio / video data possesses lip-movement consistency based on the relationship between the metric distance and the preset decision threshold, specifically includes: S61, loading the preset decision threshold, comparing the metric distance with the preset decision threshold, determining whether the audio / video data possesses lip-movement consistency, and outputting a binary decision result; S62, performing long video sliding window analysis and different time segment localization; S63, visualizing and calibrating the decision result.

[0011] To address the aforementioned technical problems, this invention also provides a lip-sound synchronization detection and calibration device, employing the following technical solution: a preprocessing module for acquiring audio and video data to be detected and performing separation and preprocessing on the audio and video data; an extraction module for detecting and extracting lip regions from the preprocessed video frame sequence to obtain a lip image sequence; a generation module for performing multi-view lip shape reconstruction and viewpoint normalization on the lip image sequence to generate a frontal lip image sequence; a construction module for constructing an audio-video spatiotemporal feature tensor based on the frontal lip image sequence to generate an audio tensor and a video tensor; a calculation module for performing bimodal heterogeneous three-dimensional convolutional coupled network feature mapping based on the audio tensor and the video tensor to calculate the metric distance between the audio feature vector and the video feature vector; and a judgment module for determining whether the audio and video data possesses speech lip movement consistency based on the relationship between the metric distance and a preset judgment threshold.

[0012] To address the aforementioned technical problems, the present invention also provides a computer device that employs the technical solution described below, comprising a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the aforementioned lip-sound synchronization detection and calibration method.

[0013] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium, which employs the technical solution described below. The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the aforementioned lip-sound synchronization detection and calibration method.

[0014] Compared with the prior art, the present invention has the following advantages: (1) By reconstructing the lip shape from multiple perspectives and normalizing the perspective, the present invention systematically solves the problem of lip feature deformation and extraction difficulties caused by non-frontal perspectives such as side face and head down in digital human teaching videos. It can automatically and faithfully convert the lip image from any angle into a standard frontal view, thereby completely eliminating the detection performance degradation caused by changes in shooting or rendering perspectives, and has real applicability and robustness in actual complex teaching scenarios; (2) By using a dual-modal heterogeneous three-dimensional convolutional coupled network and coupled contrastive loss function for training, it no longer relies on simple feature splicing, but maps audio and video features to the same high-dimensional semantic space, and forces the feature vectors of synchronized lip samples to be close to each other, while the feature vectors of asynchronous samples are far apart. This deep semantic alignment mechanism can effectively model strong temporal correlations between lips and sounds, accurately capture subtle synchronization deviations, and significantly reduce misjudgments caused by single-modal noise (such as background noise and image blur). The detection accuracy meets the stringent requirements of educational scenarios. (3) The output is not only a binary judgment of synchronization / asynchrony, but also provides continuous and quantitative distance measurement, which can accurately locate the specific time point, duration and severity of asynchrony, and generate a detailed diagnostic report. This provides direct and accurate data support for manual calibration or automatic calibration of driving algorithms in the later stage of video production. It can be seamlessly integrated into the entire process of production, optimization and quality control of digital human course videos, realizing a functional leap from detection to detection-location-calibration support, and significantly improving practical value. Attached Figure Description

[0015] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 is a flowchart of an embodiment of the lip-sound synchronization detection and calibration method of the present invention; Figure 2 is a schematic diagram of the multi-view decoupled representation generative adversarial network (MV-DRGAN) structure used in the lip-sound synchronization detection and calibration method of the present invention; Figure 3 is a schematic diagram of the dual-modal heterogeneous three-dimensional convolutional coupled network structure used in the lip-sound synchronization detection and calibration method of the present invention; Figure 4 is a schematic diagram of the audio feature (MFCC) extraction and three-dimensional tensor construction process used in the lip-sound synchronization detection and calibration method of the present invention; Figure 5 is a schematic diagram of an embodiment of the lip-sound synchronization detection and calibration device of the present invention; Figure 6 is a schematic diagram of an embodiment of the computer device of the present invention. Detailed Implementation

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects and not to describe a particular order.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0020] It should be noted that the lip-sound synchronization detection and calibration method provided in the embodiments of the present invention is generally executed by a server / terminal device, and correspondingly, the lip-sound synchronization detection and calibration device is generally set in the server / terminal device.

[0021] It should be understood that the number of terminal devices, networks, and servers is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be used.

[0022] Example 1: Please refer to Figure 1, which shows a flowchart of an embodiment of the lip-sound synchronization detection and calibration method of the present invention. The lip-sound synchronization detection and calibration method includes the following steps: Step S1, acquiring the audio and video data to be detected, and separating and preprocessing the audio and video data.

[0023] In this embodiment, the electronic device (e.g., a server / terminal device) on which the lip-sync detection and calibration method runs can receive lip-sync detection and calibration requests via a wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods.

[0024] In this embodiment, step S1, obtaining the audio and video data to be detected, and separating and preprocessing the audio and video data, may specifically include the following steps: S11, obtaining the audio and video data to be detected, and performing format verification on the audio and video data.

[0025] The software or system calls audio and video processing libraries such as FFmpeg to receive the digital human course video files (such as MP4, AVI, etc.) to be tested uploaded by the user or specified by the system.

[0026] Video file format parsing: Audio and video container parsing libraries (such as FFmpeg's avformat module) are used to read the file header information and obtain the encoding parameters of the audio and video streams. This ensures that the system can understand the data structure.

[0027] Standardization requirements allow for the setting of processing standards. For example, video streams typically require conversion to the RGB color space, a fixed resolution (such as 128×128 or higher to ensure lip detail), and a constant frame rate (such as 30fps). Audio streams require conversion to mono with a fixed sampling rate (such as 16kHz, covering the main frequency range of human voices). If the input data does not meet the standards, transcoding or resampling is performed.

[0028] The purpose of step S11 is to ensure that a unified input format is a prerequisite for the stable operation of all subsequent algorithms. It eliminates analysis errors caused by differences in source files (such as different encoding formats or parameters), and guarantees the consistency, repeatability, and reliability of the processing flow.

[0029] S12 separates the audio and video streams after the audio and video data has been format-verified.

[0030] The audio / video splitter (whose core is the demultiplexing function of FFmpeg or similar tools) operates on verified multimedia files, decapsulates them, separates the intertwined audio and video data packets, and generates two independent data streams: a pure audio data stream and a pure video data stream.

[0031] Demultiplexing is performed by calling functions such as av_read_frame() to read data packets sequentially from the container and store them into different buffers according to their type identifier (audio or video). This process preserves the timestamp information of the original data, which is crucial for subsequent synchronization analysis.

[0032] The separated audio and video streams are temporarily stored in memory or temporary files, serving as direct input for subsequent steps S2 and S4. This separation allows audio and video processing to be performed in parallel or independently, improving system efficiency.

[0033] The purpose of step S12 is to achieve modal decoupling. This creates conditions for subsequent specialized processing based on the different characteristics of audio signals and video images (such as audio spectral analysis and video face detection), and forms the basis for bimodal analysis.

[0034] S13 performs video frame serialization and audio buffering.

[0035] The frame extractor and audio buffer continue processing the separated audio and video streams. For the video stream, image data is decoded frame by frame according to its frame rate (e.g., 30fps), generating a time-ordered RGB image sequence (i.e., a video frame sequence). For the audio stream, PCM (Pulse Code Modulation) data is loaded into consecutive buffers in time order.

[0036] Video decoding and frame extraction use a video decoder (such as an H.264 / AVC or H.265 / HEVC decoder) to decode compressed video data packets into individual bitmap images. These images are stored sequentially in a list or array, forming a sequence of [frame_1, frame_2, …, frame_N], with each frame having a time index.

[0037] The audio data buffer stores the decoded audio PCM samples (usually 16-bit signed integers) in chronological order into a linear array or circular buffer. Simultaneously, the sampling rate and total duration are recorded to enable precise time slicing (such as the 0.3-second slice in step S4).

[0038] The purpose of step S13 is to transform continuous streaming media data into discrete, index-accessible sequences, a fundamental requirement for all frame- or time-window-based digital processing algorithms. It provides a directly usable data structure for frame-by-frame lip detection in step S2 and fixed-duration slice analysis in step S4.

[0039] The purpose of step S1 is to accurately and efficiently separate the independent audio and video streams from the original multimedia file, and perform necessary standardized preprocessing to provide input data with a uniform format and controllable quality for subsequent lip region extraction, feature analysis, and synchronous detection. This ensures the independent operability of the audio and video data and focuses on the core information required for subsequent processing.

[0040] Step S2: Detect and extract the lip region from the preprocessed video frame sequence to obtain the lip image sequence.

[0041] In this embodiment, step S2, which involves detecting and extracting the lip region from the preprocessed video frame sequence to obtain the lip image sequence, may specifically include the following steps: S21, performing frame-by-frame face detection and key point localization on the preprocessed video frame sequence.

[0042] Facial landmark detection models (such as Dlib, MTCNN, MediaPipe, or deep learning-based face alignment networks) process the input video frame sequence frame by frame. For each frame, the model first detects the face bounding box, and then locates dozens or even hundreds of predefined facial landmarks within the bounding box, which must include a specific set of points that identify the lip contour (such as the upper lip, lower lip, corners of the mouth, etc.).

[0043] Cascaded classifiers (such as Viola-Jones) or convolutional neural networks (CNNs) are typically used to quickly determine the location and size of faces in an image, outputting one or more rectangular bounding boxes. In digital human scenarios, a single face is usually assumed.

[0044] On the detected face regions, keypoints are located using shape predictors (such as Dlib's 68-point model) or regression networks (such as CNNs that directly regress keypoint coordinates). These keypoints precisely annotate the contours of the eyebrows, eyes, nose, mouth, and jaw. For the lips, there are typically 20 points (e.g., points 48 to 67 in a 68-point model) to define their shape.

[0045] The purpose of step S21 is to provide a geometric reference for the lip region. The coordinates of the key points are the direct basis for subsequent cropping of the region of interest (ROI) of the lips. Precise positioning is a prerequisite for ensuring that the cropping area completely includes the dynamic lips and minimizes irrelevant areas.

[0046] S22, based on key point localization, performs adaptive cropping of the lip region of interest based on key points.

[0047] Based on the lip keypoint coordinates obtained in step S21, a minimum bounding rectangle or convex polygon region that closely follows the dynamic changes of the lips is calculated. Then, pixels within this region are extracted from the original frame to generate a sub-image containing only the lips and a small portion of their surrounding area.

[0048] ROI Calculation: Find the minimum and maximum x and y coordinates of all lip keypoints. To ensure complete lip capture across different lip shapes, a safety boundary is typically established by extending the minimum bounding rectangle upwards, downwards, left, and right by a certain percentage (e.g., 10%-20%). The formula is expressed as: , , , ,in, It is the set of coordinates of all key points of the lips. and These are boundary values ​​that are dynamically calculated based on the image resolution.

[0049] Image cropping: Use the slice or crop function of an image processing library (such as OpenCV) to extract sub-images from the original frame based on the calculated rectangular coordinates. If the polygonal shape formed by the lip keypoints is complex, a mask can be created first, and then the polygonal region can be extracted.

[0050] Step S22 achieves data dimensionality reduction and feature focusing. It reduces the image size of each frame from potentially millions of pixels (e.g., 1920×1080) to a few thousand pixels (e.g., a 128×128 lip ROI), significantly reducing the computational burden on subsequent neural networks. Simultaneously, it eliminates the direct influence of factors such as head pose and background changes, allowing the model to focus more on the variations in lip shape itself.

[0051] S23, standardize and output the lip image sequence.

[0052] The image normalizer normalizes the size and pixel values ​​of all cropped lip ROI images, and finally outputs a sequence of lip images with uniform size and consistent pixel value range.

[0053] All cropped ROI images are scaled to a fixed size (e.g., 128 pixels × 128 pixels) using an interpolation algorithm (such as bilinear interpolation). This is to meet the fixed size requirement of subsequent neural network input layers.

[0054] The pixel values ​​of an image are normalized from the original integer range of 0-255 to a floating-point range of [-1, 1] or [0, 1]. The normalization formula is: (Get [-1,1]) or (This yields [0,1]). This helps accelerate the training convergence of the neural network and improves its numerical stability.

[0055] All processed single-frame lip images are arranged in their original chronological order to form a three-dimensional array or tensor, typically with the following shape: ,in It is the total number of frames. and It has a fixed height and width. This is the number of color channels (usually 3, representing RGB). This sequence is the input for the subsequent step S3.

[0056] The purpose of step S23 is to provide standardized input for the deep learning model. A uniform size and numerical range are crucial for the efficient and stable operation of algorithms such as batch processing and gradient descent. The output sequence is the core data structure carrying the spatiotemporal information of the lip shape.

[0057] The purpose of step S2 is to accurately locate and crop the region of interest (ROI) in each frame of the preprocessed video frame sequence that is most relevant to lip-sound synchronization. This greatly reduces the amount of data required for subsequent processing, removes interference from irrelevant information such as background, hair, and clothing, and allows focus on the core lip movement features, thereby improving processing efficiency and the accuracy of feature extraction.

[0058] Step S3: Perform multi-view lip shape reconstruction and view normalization on the lip image sequence to generate a frontal lip image sequence.

[0059] In this embodiment, step S3, which involves reconstructing the lip shape from multiple perspectives and normalizing the perspective of the lip image sequence to generate a frontal lip image sequence, may specifically include the following steps: S31, based on the lip image sequence, performing lip pose angle recognition and feature decoupling.

[0060] An independently pre-trained convolutional neural network receives a sequence of raw lip images. For each frame of the sequence... This module performs two tasks: first, classification, predicting the specific deflection angle (e.g., 0°, 15°, 30°, 45°, 60°, 90°, a total of 6 classes); second, feature extraction, extracting feature representations strongly correlated with the pose angle from the output of the middle or high layers of the network. Meanwhile, the generator's encoder (G-encoder) extracts data from the same frame image. In the process, deep essential features related to speaker identity, lip texture and shape, and as independent of posture as possible, are extracted. .

[0061] A modified AlexNet architecture is used, with the input being a lip ROI image (e.g., 128×128×3). After five convolutional layers (with ReLU activation and local response normalization) and three max-pooling layers, the feature map size gradually decreases while the number of channels increases. Finally, the angle classification probability is obtained through a fully connected layer and a 6-neuron softmax output layer. The activation value taken from the last convolutional layer or the first fully connected layer is a high-dimensional vector (e.g., 6×6×256=9216 dimensions) that encodes the pose information of the image.

[0062] It employs the encoder portion of an encoder-decoder architecture (such as CASIA-Net). It is also a multi-layer CNN, but its training objective is to force the network to learn to discard angular information and retain only the essential information of "how this person moves their lips" through adversarial loss and reconstruction loss. Its output... It is also a high-dimensional feature vector (e.g., 6×6×320=11520 dimensions).

[0063] Feature decoupling is crucial for decoupling representation learning. It decomposes image information into what it is (identity / lip shape) and how it's viewed (angle), enabling the generator to base its learning on essential features. Encoding from the target angle Under the guidance of [unclear], the lip shape was re-rendered from a frontal view. This was achieved by minimizing the cross-entropy pose loss function. Perform pre-training: ,in It is a one-hot encoding from a true perspective. This is a prediction probability. After training, the parameters are frozen and not trained jointly with GAN to ensure the stability of angle estimation.

[0064] S32, perform multi-condition feature fusion to generate a frontal lip image sequence.

[0065] The generator's decoder (G-decoder) receives a feature vector that fuses multiple pieces of information. This vector is composed of the following parts: essential lip features. Angular related features One-hot encoding of target pose (Regarding positive reconstruction,) The code corresponding to a 0° angle, such as ), and a random noise vector sampled from a uniform distribution. Based on this fusion feature, the decoder gradually recovers the spatial dimensions of the image through a series of upsampling or deconvolution operations, ultimately generating a reconstructed standard frontal lip image. .

[0066] Feature fusion: , , , Equal vectors are concatenated along the feature dimension. It provides clear target perspective instructions. Randomness is introduced, enabling the generator to produce reasonable lip diagrams with subtle variations, avoiding pattern collapse and improving generation diversity.

[0067] Decoder structure: Consists of multiple deconvolutional layers (or transposed convolutional layers), batch normalization layers, and activation function layers (such as ReLU) stacked together. As the network depth increases, the spatial size of the feature maps ( The number of channels is doubled, the number of channels is reduced, and finally, a deconvolution layer and a Tanh activation function are applied to the input and output. Images of the same size (e.g., 128×128×3) with pixel values ​​in the range [-1, 1]. .

[0068] The significance of generating a sequence of frontal lip images lies in the fact that it is a conditional image generation process. It utilizes... Provides the desired lip shape, using and It provides information on the viewing angle and the desired viewing angle to guide the generation process. Ultimately... Visually it should be in harmony with The same person, the same pronunciation action, but the perspective is a frontal view of the lips.

[0069] S33, adversarial training and multi-objective optimization of the frontal lip image sequence.

[0070] Figure 2 is a schematic diagram of the multi-view decoupled representation generative adversarial network (MV-DRGAN) structure used in the lip-sound synchronization detection and calibration method of the present invention. As shown in Figure 2, the discriminator (D) and the entire generative adversarial network are trained using a comprehensive loss function. The discriminator's task is to distinguish between fake frontal lip images generated by the generator. and real frontal lip images selected from the dataset The generator aims to deceive the discriminator while satisfying multiple auxiliary constraints to generate a high-quality, compliant frontal lip image.

[0071] The discriminator is a CNN classifier that takes an image as input and outputs a scalar representing the probability that the image is a real image. It forces the generator to continuously improve the realism of the generated images.

[0072] Total loss function It is a weighted sum of multiple sub-losses used to comprehensively optimize the quality of the generated image: adversarial loss ( ): Employ loss forms from Wasserstein GAN (WGAN) or Least Squares GAN (LSGAN), such as LSGAN: It drives the generator and discriminator to engage in a game, improving the global realism of the generated images.

[0073] Symmetric loss ( ): Utilizing the physiological symmetry of human lips, calculate and generate images. The difference between it and its horizontally flipped image (such as L1 distance). This constraint generator outputs a symmetrical lip shape that better matches biological structures.

[0074] Reconstruction loss ( ): Constraints are applied in the feature space. The resulting image will be... and real target image Input each of the pre-trained feature extraction networks (such as VGG) and calculate the difference between their outputs at the high-dimensional feature layer (such as L2 distance). ,in This represents the feature extraction network. This ensures that the generated image is semantically consistent with the target.

[0075] Pixel loss ( () Constraints are applied in pixel space. Image generation is directly calculated. Compared with real target image The absolute error between them (L1 distance): This ensures that the generated image is pixel-level aligned with the target image in detail. Total loss: ,in It is a hyperparameter that balances the weights of various components.

[0076] The purpose of step S33 is to ensure that, through this multi-objective optimization strategy, MV-DRGAN can not only generate realistic frontal lip images, but also ensure that the generated lip shapes are structurally reasonable (symmetric), semantically matched (reconstruction loss), and precise in detail (pixel loss). Finally, the trained generator can reconstruct each non-frontal lip image in the input sequence, outputting a viewpoint-normalized frontal lip image sequence composed entirely of standard frontal lip images for the next step.

[0077] The purpose of step S3 is to reconstruct the non-standard lip images extracted in step S2, which may contain various deflection angles (such as side view or head tilt), into a unified standard frontal (0°) view lip image with high fidelity using generative adversarial network (GAN) technology. This solves the fundamental problems of difficult lip feature extraction and large appearance deformation under non-frontal view, providing a consistent and feature-clear input sequence for subsequent synchronous detection, which is key to achieving high-precision cross-view lip sound synchronous detection.

[0078] Step S4: Based on the frontal lip image sequence, construct the audio and video spatiotemporal feature tensor to generate the audio tensor and video tensor.

[0079] In this embodiment, step S4, based on the frontal lip image sequence, constructs an audio-video spatiotemporal feature tensor and generates an audio tensor and a video tensor. Specifically, this may include the following steps: S41, slicing the audio stream according to a fixed duration, extracting spectral features, and generating an audio three-dimensional feature tensor.

[0080] Figure 4 is a schematic diagram of the audio feature extraction (MFCC) and three-dimensional tensor construction process used in the lip-sync detection and calibration method of the present invention. As shown in Figure 4, the audio feature extractor operates on the audio stream to be processed separated from step S1. First, the entire audio stream is sliced ​​into non-overlapping or overlapping segments of fixed short duration (e.g., 0.3 seconds). Then, for each audio segment, its Mel-frequency cepstral coefficients (MFCC) are calculated, and the first-order difference (Delta) and second-order difference (Delta-Delta) of the MFCC are further calculated. Finally, these features are concatenated along the channel dimension.

[0081] Each 0.3-second audio slice (e.g., 4800 samples @ 16kHz) is pre-emphasized (high frequencies are boosted), and then framed in shorter analysis frames (e.g., 25ms, 400 samples), with a frame shift of typically 10ms, to obtain smooth time-varying features.

[0082] MFCC Calculation: Windowing and FFT: Apply a Hanning window to each frame of data and perform a 512-point Fast Fourier Transform (FFT) to obtain the power spectrum.

[0083] Mel filter bank: Passes the power spectrum through a set of 40 triangular Mel-scale filters. Mel frequencies approximate the nonlinear auditory perception characteristics of the human ear.

[0084] Logarithmic transformation and DCT: Take the logarithm of the output energy of each filter, then perform a Discrete Cosine Transform (DCT) on the resulting 40 logarithmic energies, retaining the first 20 coefficients (usually of order 2-13), thus obtaining a 20-dimensional MFCC feature. MFCC describes the short-time power spectrum envelope of sound.

[0085] Difference calculation: The first-order difference (Delta) reflects the rate of change of the MFCC coefficients over time (similar to velocity), and the second-order difference (Delta-Delta) reflects the rate of change of the rate of change (similar to acceleration). Together, they describe the dynamic characteristics of speech.

[0086] Tensor Construction: For a 0.3-second slice, after the above frame division (e.g., 30 frames), each frame yields a 20-dimensional MFCC. Stacking the MFCCs from the 30 frames results in a shape... The matrix. Then calculate its Delta and Delta-Delta respectively to obtain the other two. The matrix is ​​then used. Finally, these three matrices are concatenated along the second dimension (feature dimension) to obtain a three-dimensional tensor with the shape of... .in, It is the number of time frames. It is the Mel band number (MFCC order). It is the number of feature channels (original, first difference, second difference).

[0087] The purpose of step S41 is as follows: MFCC and its differences are classic features in speech recognition and audio analysis. They effectively compress audio information, retain key spectral characteristics related to the pronunciation content, and filter out some personal pronunciation traits and environmental noise. After being constructed into a three-dimensional tensor, it has an image-like structure: the time and frequency dimensions form a plane, and the three channels form the depth, which can be directly processed using 3D-CNN.

[0088] S42 extracts and stacks frames of the video stream according to the corresponding time window to generate a three-dimensional spatiotemporal tensor for the video.

[0089] Based on the timestamp of each audio slice (e.g., starting at t seconds and lasting 0.3 seconds), consecutive video frames within the same time window are precisely extracted from the perspective-normalized frontal lip image sequence output in step S3. These frames are then stacked in the time dimension to form a video clip.

[0090] Time alignment is a critical step. The system maintains a shared timeline for both audio and video. For a given time starting at... Given an audio slice of 0.3 seconds, calculate its corresponding frame index range within the video sequence. For example, for a 30fps video, 0.3 seconds corresponds to 9 frames. The starting frame index is... Nine consecutive frames were captured.

[0091] These 9 images (each frame has a shape of...) Stack them in chronological order, on the first dimension (the new time dimension). The operation can be represented as: The result is a four-dimensional tensor. When feeding the data into a 3D-CNN, the channel dimensions are... Placed last, that is .

[0092] The purpose of step S42 is to organize discrete frame sequences into a continuous spatio-temporal volume. The convolutional kernels of 3D-CNN can slide along the three dimensions of time, height, and width, thereby directly learning the joint temporal and spatial variation patterns of lip movements. This is fundamentally different from the traditional method of extracting 2D features frame by frame and then fusing them temporally, and it can better model the continuity and dynamics of lip movements.

[0093] S43 associates the audio 3D feature tensor and the video 3D spatiotemporal tensor to form a paired audio-video tensor pair.

[0094] These tensor pairs are organized in chronological order and serve as input samples for the bimodal network in step S5.

[0095] Pairing logic: Each 0.3-second time window corresponds precisely to an audio tensor and a video tensor. They cover the exact same physical time period. This alignment is the cornerstone of audio-video synchronization analysis.

[0096] The data structure is organized as a list of samples, where each element is a tuple. ,in The shape is , The shape is . (Audio time frames) and The number of video frames may vary, but this does not matter because subsequent 3D-CNN will process them separately.

[0097] During training or inference, multiple such tensor pairs are combined into a batch. The shape of the audio tensor batch is... The batch shape of the video tensor is .

[0098] The purpose of step S43 is to generate structured, aligned multimodal data samples. This allows subsequent neural networks to easily perform batch processing and learn cross-modal correlation patterns between audio and video through aligned samples.

[0099] Step S4 serves as a bridge connecting the original data and the deep learning model. It transforms the continuous audio signals and video image sequences into a three-dimensional tensor form suitable for processing by 3D convolutional neural networks (3D-CNN), simultaneously preserving temporal dynamic information and spatial / frequency structural information, according to strict time alignment. This provides a unified, structured input representation rich in spatiotemporal context information for subsequent bimodal feature fusion and synchronization judgment.

[0100] Step S5: Based on the audio tensor and video tensor, perform feature mapping of the dual-modal heterogeneous 3D convolutional coupled network, and calculate the metric distance between the audio feature vector and the video feature vector.

[0101] In this embodiment, step S5, which involves performing feature mapping of a dual-modal heterogeneous three-dimensional convolutional coupled network based on audio and video tensors and calculating the metric distance between audio feature vectors and video feature vectors, may specifically include the following steps: S51, setting up a heterogeneous three-dimensional convolutional sub-network based on audio and video tensors to extract audio feature vectors and video feature vectors.

[0102] Figure 3 is a schematic diagram of the bimodal heterogeneous 3D convolutional coupled network structure used in the lip-sync detection and calibration method of the present invention. As shown in Figure 3, the bimodal heterogeneous 3D convolutional coupled network consists of two parallel sub-networks: an audio 3D-CNN sub-network and a video 3D-CNN sub-network. Their heterogeneous structure means that their architecture, such as the number of layers, convolutional kernel size, and pooling method, is independently designed according to the different data characteristics of audio spectrograms and video image sequences to extract the spatiotemporal features of their respective modalities in the optimal way. The ends of the two sub-networks are fully connected layers, which map the features to vectors of the same dimension (e.g., 128 dimensions).

[0103] For the audio subnetwork: Input: Shape is A tensor. It can be viewed as... indivual The "image".

[0104] Architectural characteristics: Convolutional kernels are typically time-varying ( ),frequency( Operate in two dimensions. For example, using A 3D convolutional kernel (time × frequency) is used. Pooling layers may focus more on downsampling in the frequency dimension, as dynamic changes in the time dimension are important for speech. The network structure may be deep to capture complex spectral patterns. Finally, a 128-dimensional audio feature vector is output through global pooling or fully connected layers. .

[0105] For the video subnetwork: Input: Shape is The tensor.

[0106] Architectural features: Convolutional kernels in time ( ),high( ),width( Operating in three dimensions, for example Initial convolutional layers may use larger spatial kernels to capture the lip shape, while later layers use smaller temporal kernels to capture dynamics. Pooling is performed in the spatiotemporal dimension. The network ultimately outputs a 128-dimensional video feature vector. .

[0107] The heterogeneity between audio and video 3D-CNN subnetworks lies in maximizing the feature extraction efficiency of each type of data, rather than forcing them to use the same structure. For example, audio has a stable frequency domain structure, requiring detailed analysis; video has strong spatiotemporal dynamics, requiring joint modeling.

[0108] S52, joint training is performed based on the coupled contrastive loss function.

[0109] During the training phase, the network optimizer uses a special coupled-contrast loss function. This approach optimizes both the audio and video sub-networks simultaneously. Instead of direct classification, the loss function learns a metric space: it adjusts the network parameters to ensure that synchronized (consistent) audio and video samples... The Euclidean distance between the feature vectors of the first pair decreases in space, while the feature vector distance between asynchronous (inconsistent) sample pairs increases, and is at least greater than one interval. .

[0110] The loss function formula is: ,in: A set of audio and video sample pairs in a batch. : No. The true labels of each sample pair. This indicates that the audio and video are synchronized (consistent). This indicates a lack of synchronization (inconsistency). Batch Size. : No. Audio feature vectors in a sample pair and video feature vectors The Euclidean distance between them. . Loss for consistent sample pairs. This means that for a consistent pair, the distance... The larger the distance, the greater the penalty, and the network will be driven to reduce this distance. Loss for inconsistent sample pairs. .in It is a margin parameter greater than 0. This means that for inconsistent pairs, only when their distance... Less than the interval Punishment will only be imposed at that time, and the purpose of the punishment is to push them at least [a certain distance]. Besides. If the distance is already greater than If so, the loss is 0. The weights of the L2 regularization term are used to prevent the network from overfitting. This represents all trainable parameters in the network. : The squared L2 norm of the model parameters (weight decay).

[0111] In each training iteration, forward propagation obtains the feature vectors and distances of a batch of samples, and then calculates... Then, the parameters of the audio and video subnetworks are updated simultaneously through backpropagation and gradient descent algorithms (such as Adam).

[0112] This loss function is a form of contrastive loss. Its core idea is to group similar features together and dissimilar features together. It doesn't care about the absolute positions of feature vectors, only their relative distances in space. In this way, the network learns to encode the abstract concept of lip-sync as proximity in the feature vector space. (Interval) The setting ensures a clear distance boundary between asynchronous samples, improving the robustness of the discrimination.

[0113] S53 calculates the metric distance between the audio feature vector and the video feature vector.

[0114] During the inference (detection) phase, the trained network serves as a fixed feature extractor. For a new audio-video tensor pair to be detected, the network's forward propagation module inputs it into the audio and video subnetworks respectively, obtaining the corresponding 128-dimensional feature vectors. and Subsequently, the distance calculation module calculates the Euclidean distance between the two vectors. .

[0115] Forward propagation: Load the trained network weights. Input audio and video tensors, which undergo convolution, activation, pooling, and fully connected operations at each layer, ultimately yielding two vectors. This process does not involve gradient calculation and is relatively fast.

[0116] For two dimensional vector and The Euclidean distance is defined as: It measures the straight-line distance in vector space.

[0117] Output: Distance It is a scalar value, and it is the core metric for measuring the consistency of audio and video over a given period of time. The smaller the value, the closer the audio and video features are in the metric space obtained during training, meaning the higher the probability of synchronization. The larger the value, the more asynchronous the system.

[0118] The purpose of step S52 is to transform the complex audio-video synchronization problem into a simple scalar distance determination problem. This distance... It contains all the prior knowledge about lip-phonetic associations learned by the network and serves as the direct basis for making the final decision.

[0119] The purpose of step S5 is to design a dedicated dual-branch neural network to extract deep features from the audio and video tensors generated in step S4, respectively, and map the features of the two heterogeneous modalities into a shared high-dimensional semantic space. In this space, the feature vectors of synchronized audio and video pairs should be close to each other, while those of asynchronous pairs should be far apart. This network is trained end-to-end using a coupled contrastive loss, enabling it to accurately measure the similarity of audio and video features.

[0120] Step S6: Determine whether the audio and video data have consistent speech lip movements based on the relationship between the metric distance and the preset decision threshold.

[0121] In this embodiment, step S6, determining whether the audio and video data has speech lip movement consistency based on the relationship between the measurement distance and the preset decision threshold, may specifically include the following steps: S61, loading the preset decision threshold, comparing the measurement distance with the preset decision threshold, determining whether the audio and video data has speech lip movement consistency, and outputting a binary decision result.

[0122] The decision maker reads a decision threshold determined through optimization on the model validation set. Then, it calculates the distance from step S5. The result is compared with the threshold, and a binary decision is output.

[0123] Decision threshold It's not set arbitrarily. After model training is complete, on an independent validation set, the optimal threshold is determined through a search, using metrics such as F1 score, accuracy, or the area under the receiver operating characteristic (ROC) curve (AUC) as optimization targets. This threshold represents the optimal dividing point for distinguishing between synchronous and asynchronous data under the training data distribution.

[0124] Judgment rule: If If so, it is determined that the audio and video data within the current 0.3-second time window of analysis have lip-sync (consistency); if If so, it is determined that there is no lip-sound synchronization (inconsistency).

[0125] For the current window, output a binary label (1 or 0, True or False) and the original distance value. Distance value This can itself be regarded as a quantitative indicator of the degree of asynchrony. The larger the size, the more severe the asynchrony may be.

[0126] Step S61 serves to transform the continuous distance values ​​into a defined synchronization state. This is a crucial step in converting the model's capabilities into practically usable detection results.

[0127] S62 performs long video sliding window analysis and asynchronous segment localization.

[0128] For a complete digital human course video (e.g., several minutes long), the sliding window analysis engine automatically performs the following operations: starting from the video's start time, it slides with a window length defined in step S4 (e.g., 0.3 seconds) and a certain step size (the step size can be smaller than the window length, such as 0.1 seconds, to achieve overlap analysis and improve positioning accuracy). For each window position, steps S1 to S61 are repeated to obtain a continuous sequence of decision results and a corresponding distance value sequence.

[0129] Sequence generation: Assuming the total video duration The window size is 0.3 seconds, and the step size is 0.1 seconds. Approximately [number] seconds will be generated. There are 100 analysis points. Each point corresponds to a timestamp (window center time) and a decision result (synchronized / desynchronized).

[0130] Asynchronous Segment Location: Analyze the sequence of decision results. Window containing consecutive asynchronous decisions constitutes an asynchronous segment. The start time, end time, and duration of each asynchronous segment can be recorded. Simultaneously, the distance between all windows within that segment can be calculated. The average or maximum value is used as an indicator of the severity of the asynchrony in that segment.

[0131] Generate a detailed test report, including the overall synchronization rate, a list of asynchronous segments, and distance-time curves. The overall synchronization rate is... The list of out-of-sync segments indicates the location and severity of each segment. The distance-time curve visualizes the changes in synchronization quality throughout the video.

[0132] The purpose of step S62 is to achieve fine-grained, quantitative diagnosis of lip-sync issues in long videos. It not only tells teachers whether there is a problem with the video, but also precisely identifies the point in time when the problem occurred, how long it lasted, and how severe it was. This provides extremely accurate targeting information for subsequent manual review or automatic calibration.

[0133] S63 provides a visualization and calibration interface for the judgment results.

[0134] The detection report generated in step S62 is presented to the user (such as a course production teacher or post-production engineer) in the form of a graphical user interface (GUI) or structured data (such as a JSON file). Simultaneously, this module also provides a programming interface (API) for interacting with video editing software or digital human-driven engines.

[0135] The visualization includes waveform graphs, markers, and data tables. Waveform Graphs: A color-coded synchronization status bar (green for synchronization, red for desynchronization) is overlaid above the audio / video waveform graphs, visually indicating problem areas. Markers: Detected desynchronization segments are directly marked on the video player's timeline; clicking them allows navigation. Data Tables: Detailed information on all desynchronized segments is listed in tabular form.

[0136] The API interface provides standard function calls, such as `detect_sync(video_path)` which returns detection results in JSON format. More advanced interfaces can receive detected misalignment information and automatically invoke digital human lip-sync algorithms to regenerate or adjust the timeline of lip-sync sequences within specific time periods, achieving preliminary automatic calibration. Alternatively, misalignment information (such as video leading audio by 200ms) can be output to non-linear editing software (such as Adobe Premiere) to assist in rapid manual calibration.

[0137] Step S63 aims to seamlessly integrate the technical testing results into the actual workflow. Visualization lowers the barrier to entry, enabling teachers without a technical background to quickly understand and locate problems. The API interface makes it possible to automate or semi-automate the closed loop of course video quality optimization, truly realizing an integrated solution of testing, location, and calibration.

[0138] The beneficial effects of implementing this embodiment are: (1) By multi-view lip reconstruction and view normalization, the problem of lip feature deformation and extraction difficulties caused by non-frontal viewpoints such as side face and head down in digital human teaching videos is systematically solved. It can automatically and faithfully convert lip images from any angle into standard frontal views, thereby completely eliminating the detection performance degradation caused by changes in shooting or rendering viewpoints, and has real applicability and robustness in actual complex teaching scenarios; (2) By using a dual-modal heterogeneous three-dimensional convolutional coupled network and coupled contrastive loss function for training, it no longer relies on simple feature splicing, but maps audio and video features to the same high-dimensional semantic space, and forces the feature vectors of synchronized lip samples to be close to each other, while the feature vectors of asynchronous samples are far apart. This deep semantic alignment mechanism can effectively model strong temporal correlations between lips and sounds, accurately capture subtle synchronization deviations, and significantly reduce misjudgments caused by single-modal noise (such as background noise and image blur). The detection accuracy meets the stringent requirements of educational scenarios. (3) The output is not only a binary judgment of synchronization / asynchrony, but also provides continuous and quantitative distance measurement, which can accurately locate the specific time point, duration and severity of asynchrony, and generate a detailed diagnostic report. This provides direct and accurate data support for manual calibration or automatic calibration of driving algorithms in the later stage of video production. It can be seamlessly integrated into the entire process of production, optimization and quality control of digital human course videos, realizing a functional leap from detection to detection-location-calibration support, and significantly improving practical value.

[0139] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0141] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0142] Referring further to FIG5, as an implementation of the method shown in FIG1, the present invention provides an embodiment of a lip-sound synchronization detection and calibration device, which corresponds to the method embodiment shown in FIG1, and can be specifically applied to various electronic devices.

[0143] As shown in Figure 5, the lip-sound synchronization detection and calibration device 70 in this embodiment includes: a preprocessing module 71, an extraction module 72, a generation module 73, a construction module 74, a calculation module 75, and a judgment module 76. Specifically: the preprocessing module 71 acquires the audio and video data to be detected and performs separation and preprocessing on the audio and video data; the extraction module 72 performs lip region detection and extraction on the preprocessed video frame sequence to obtain a lip image sequence; the generation module 73 performs multi-view lip shape reconstruction and viewpoint normalization on the lip image sequence to generate a frontal lip image sequence; the construction module 74 constructs an audio-video spatiotemporal feature tensor based on the frontal lip image sequence, generating an audio tensor and a video tensor; the calculation module 75 performs feature mapping on a dual-modal heterogeneous three-dimensional convolutional coupled network based on the audio tensor and the video tensor, calculating the metric distance between the audio feature vector and the video feature vector; and the judgment module 76 determines whether the audio and video data possesses speech lip movement consistency based on the relationship between the metric distance and a preset decision threshold.

[0144] The beneficial effects of implementing this embodiment are as follows: it systematically solves the problem of lip feature deformation and extraction difficulties caused by non-frontal perspectives such as side profiles and head tilts in digital human teaching videos; it can accurately capture subtle synchronization deviations, significantly reducing misjudgments caused by single-modal noise (such as background noise and image blur), and the detection accuracy meets the stringent requirements of educational scenarios; it can be seamlessly integrated into the entire process of digital human course video production, optimization, and quality control, realizing a functional leap from detection to detection-positioning-calibration support, and significantly improving practical value.

[0145] Example 3: To solve the above-mentioned technical problems, this embodiment of the invention also provides a computer device. Please refer to Figure 6 for details; Figure 6 is a basic structural block diagram of the computer device in this embodiment.

[0146] The aforementioned computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only the computer device 8 with components 81, 82, and 83 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0147] The aforementioned computer devices can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. These devices can facilitate human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0148] The aforementioned memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the aforementioned memory 81 may be an internal storage unit of the aforementioned computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the aforementioned memory 81 may also be an external storage device of the aforementioned computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the aforementioned memory 81 may also include both the internal storage unit and its external storage device of the aforementioned computer device 8. In this embodiment, the aforementioned memory 81 is typically used to store the operating system and various application software installed on the aforementioned computer device 8, such as computer-readable instructions for lip-sync detection and calibration methods. In addition, the aforementioned memory 81 can also be used to temporarily store various types of data that have been output or will be output.

[0149] In some embodiments, the processor 82 described above may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to execute computer-readable instructions stored in the memory 81 or to process data, for example, to execute computer-readable instructions for the lip-sync detection and calibration method described above.

[0150] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0151] The beneficial effects of implementing this embodiment are as follows: it systematically solves the problem of lip feature deformation and extraction difficulties caused by non-frontal perspectives such as side profiles and head tilts in digital human teaching videos; it can accurately capture subtle synchronization deviations, significantly reducing misjudgments caused by single-modal noise (such as background noise and image blur), and the detection accuracy meets the stringent requirements of educational scenarios; it can be seamlessly integrated into the entire process of digital human course video production, optimization, and quality control, realizing a functional leap from detection to detection-positioning-calibration support, and significantly improving practical value.

[0152] Example 4: The present invention also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the lip-sound synchronization detection and calibration method described above.

[0153] The beneficial effects of implementing this embodiment are as follows: it systematically solves the problem of lip feature deformation and extraction difficulties caused by non-frontal perspectives such as side profiles and head tilts in digital human teaching videos; it can accurately capture subtle synchronization deviations, significantly reducing misjudgments caused by single-modal noise (such as background noise and image blur), and the detection accuracy meets the stringent requirements of educational scenarios; it can be seamlessly integrated into the entire process of digital human course video production, optimization, and quality control, realizing a functional leap from detection to detection-positioning-calibration support, and significantly improving practical value.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0155] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A method for lip-sound synchronization detection and calibration, characterized in that, The process includes the following steps: S1, acquiring the audio and video data to be detected, and separating and preprocessing the audio and video data; S2, performing lip region detection and extraction on the preprocessed video frame sequence to obtain a lip image sequence; S3. Perform multi-view lip shape reconstruction and viewpoint normalization on the lip image sequence to generate a frontal lip image sequence; S4. Based on the frontal lip image sequence, construct an audio-video spatiotemporal feature tensor to generate an audio tensor and a video tensor; S5. Based on the audio tensor and the video tensor, perform feature mapping on a dual-modal heterogeneous three-dimensional convolutional coupled network to calculate the metric distance between the audio feature vector and the video feature vector; S6. Determine whether the audio-video data has speech lip movement consistency based on the relationship between the metric distance and a preset decision threshold.

2. The lip-sound synchronization detection and calibration method according to claim 1, characterized in that, The steps of S1, acquiring the audio and video data to be detected and separating and preprocessing the audio and video data, specifically include: S11, acquiring the audio and video data to be detected and performing format verification on the audio and video data; S12, separating the audio and video streams of the audio and video data after format verification; and S13, performing video frame serialization and audio buffering.

3. The lip-sound synchronization detection and calibration method according to claim 1, characterized in that, The step S2, which involves detecting and extracting the lip region from the preprocessed video frame sequence to obtain the lip image sequence, specifically includes: S21, performing frame-by-frame face detection and key point localization on the preprocessed video frame sequence; S22, performing adaptive cropping of the lip region of interest based on key point localization; and S23, standardizing and outputting the lip image sequence.

4. The lip-sound synchronization detection and calibration method according to claim 1, characterized in that, The step S3, which involves reconstructing the lip shape from multiple perspectives and normalizing the perspectives of the lip image sequence to generate a frontal lip image sequence, specifically includes: S31, performing lip pose angle recognition and feature decoupling based on the lip image sequence; S32, performing multi-condition feature fusion to generate a frontal lip image sequence; and S33, performing adversarial training and multi-objective optimization on the frontal lip image sequence.

5. The lip-sound synchronization detection and calibration method according to claim 1, characterized in that, The step S4, which involves constructing an audio-video spatiotemporal feature tensor based on the frontal lip image sequence, and generating an audio tensor and a video tensor, specifically includes: S41, slicing the audio stream into segments of fixed duration, extracting spectral features, and generating an audio three-dimensional feature tensor; S42, truncating and stacking the video stream into frames according to corresponding time windows to generate a video three-dimensional spatiotemporal tensor; and S43, associating the audio three-dimensional feature tensor and the video three-dimensional spatiotemporal tensor to form a paired audio-video tensor pair.

6. The lip-sound synchronization detection and calibration method according to claim 1, characterized in that, The step S5, which involves performing feature mapping of a dual-modal heterogeneous three-dimensional convolutional coupled network based on the audio tensor and the video tensor, and calculating the metric distance between the audio feature vector and the video feature vector, specifically includes: S51, setting up a heterogeneous three-dimensional convolutional sub-network based on the audio tensor and the video tensor, and extracting the audio feature vector and the video feature vector; S52, performing joint training based on the coupling contrastive loss function; and S53, calculating the metric distance between the audio feature vector and the video feature vector.

7. The method for lip-sound synchronization detection and calibration according to any one of claims 1 to 6, characterized in that, The step S6, determining whether the audio / video data possesses lip-movement consistency based on the relationship between the metric distance and the preset decision threshold, specifically includes: S61, loading the preset decision threshold, comparing the metric distance with the preset decision threshold, determining whether the audio / video data possesses lip-movement consistency, and outputting a binary decision result; S62, performing long video sliding window analysis and different time segment localization; S63, visualizing and calibrating the decision result.

8. A lip-sound synchronization detection and calibration device, characterized in that, include: The preprocessing module is used to acquire the audio and video data to be detected, and to separate and preprocess the audio and video data; The extraction module is used to detect and extract the lip region from the preprocessed video frame sequence to obtain a lip image sequence. The generation module is used to reconstruct the lip shape and normalize the viewpoint of the lip image sequence from multiple perspectives to generate a frontal lip image sequence; the construction module is used to construct an audio-visual spatiotemporal feature tensor based on the frontal lip image sequence to generate an audio tensor and a video tensor. The calculation module is used to perform feature mapping of a dual-modal heterogeneous three-dimensional convolutional coupled network based on the audio tensor and the video tensor, and to calculate the metric distance between the audio feature vector and the video feature vector. The determination module is used to determine whether the audio and video data has lip movement consistency based on the relationship between the measured distance and the preset decision threshold.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the lip-voice synchronization detection and calibration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the lip-sound synchronization detection and calibration method as described in any one of claims 1 to 7.

Images