Method for detecting pickup based on synthetic lips and related equipment
By combining audio and video modal data and using CNN and lip generation models to generate synthetic lip feature sequences, the problem of low accuracy in traditional proxy answering detection methods is solved, achieving higher proxy answering detection accuracy and environmental adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-04-28
AI Technical Summary
Traditional methods for detecting proxy answers have low accuracy and effectiveness. Single features are easily affected by factors such as environmental noise and changes in lighting, making it difficult to comprehensively and accurately judge proxy answering behavior.
A method for detecting substitute answers based on synthetic lip sounds is adopted. By receiving raw audio and video data, lip feature sequences are extracted using a CNN face detector. Synthetic lip feature sequences are generated by combining a pre-trained lip generation model. Differential features are calculated and substitute answers are identified through a multi-scale temporal convolutional network and a classifier. Data from audio and video modalities are fused.
It improves the accuracy and robustness of proxy detection, enables stable operation in complex environments, fully utilizes the complementarity of multimodal information, and enhances the stability of the detection system.
Smart Images
Figure CN121935728A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and related equipment for detecting substitute responses based on synthetic lip sounds. Background Technology
[0002] In many scenarios involving identity verification or interaction, proxy answering is a serious problem. For example, in online exams, candidates may have someone else answer the questions for them to obtain unfair scores; in remote interviews, job applicants may have someone else take the interview on their behalf to deceive the recruiter.
[0003] Traditional methods for detecting proxy answers mainly rely on single audio or video features, such as identifying the speaker solely through speech recognition or identifying the person in the scene solely through facial recognition.
[0004] However, the applicant found that these single-feature methods have significant limitations. Specifically, audio features are easily affected by environmental noise, speech imitation, and other factors, leading to a decrease in recognition accuracy. For example, in noisy environments, speech recognition systems may fail to accurately extract the speaker's features, thus misjudging proxy behavior. While video features can provide more information to some extent, they are also easily affected by factors such as changes in lighting and occlusion. Moreover, relying solely on a single feature is insufficient to comprehensively and accurately determine whether proxy behavior exists, as the proxy may meticulously prepare to imitate the audio or video features of the person being represented.
[0005] This shows that traditional methods for detecting proxy responses have problems with low accuracy and effectiveness. Summary of the Invention
[0006] The purpose of this application is to propose a method and related equipment for detecting substitution based on synthetic lip sounds, so as to solve the problems of low detection accuracy and effectiveness of traditional substitution detection methods.
[0007] To address the aforementioned technical problems, this application provides a method for detecting substitute responses based on synthesized lip sounds, employing the following technical solution: Receive proxy response detection requests carrying raw audio and video data; The original audio and video data are segmented to obtain original video data and original audio data; The original video data is processed by a CNN face detector to extract lip sequences, resulting in lip feature sequences. The original audio data is processed by a pre-trained lip generation model to generate a synthetic lip sequence, resulting in a synthetic lip feature sequence. The difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space is calculated to obtain the difference feature sequence; The differential feature sequence is segmented using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input. The proxy response detection result is obtained by performing proxy response identification processing on the probability distribution sequence of the actions according to the classifier.
[0008] Furthermore, the step of generating a synthetic lip sequence from the original audio data using a pre-trained lip generation model to obtain a synthetic lip feature sequence specifically includes the following steps: The original audio data is converted into spectral feature data using the Fourier transform method; The key speech feature sequence is obtained by performing speech feature extraction processing on the spectral feature data according to the deep learning model. The key speech feature sequence is converted into a visual dynamic effect using a generative adversarial network to obtain the synthetic lip feature sequence that matches the original audio data.
[0009] Furthermore, after the step of extracting speech features from the spectral feature data using a deep learning model to obtain a key speech feature sequence, the following step is also included: The temporal correspondence between the key speech feature sequence and the preset visual template is calculated using a time-series analysis algorithm to obtain the time correspondence deviation. If the time deviation exceeds a preset deviation threshold, then the key speech feature sequence is adjusted to obtain an optimized key speech feature sequence.
[0010] Furthermore, after the step of converting the key speech feature sequence into a visual dynamic effect based on the generative adversarial network to obtain the synthetic lip feature sequence matching the original audio data, the following step is also included: Dynamic adaptation parameters are extracted from the synthesized lip feature sequence; The synthesized lip feature sequence is smoothed according to the interpolation algorithm and the dynamic adaptation parameters to obtain the optimized synthesized lip feature sequence.
[0011] Furthermore, after the step of performing action segmentation processing on the differential feature sequence using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input, the following step is also included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted according to time series smoothing filtering to obtain a smooth action probability distribution sequence.
[0012] Furthermore, after the step of receiving the proxy detection request carrying the original audio and video data, the following steps are also included: The original audio data is subjected to noise reduction processing; and / or The original audio data is then processed by frame segmentation.
[0013] To address the aforementioned technical problems, this application also provides a device for detecting substitute responses based on synthesized lip sounds, employing the following technical solution: The request receiving module is used to receive proxy detection requests carrying raw audio and video data; The data segmentation module is used to segment the original audio and video data to obtain original video data and original audio data. The lip sequence extraction module is used to extract lip sequences from the original video data based on a CNN face detector to obtain lip feature sequences. The synthetic lip sequence generation module is used to process the original audio data into a synthetic lip sequence based on a pre-trained lip generation model, so as to obtain a synthetic lip feature sequence. The differential feature calculation module is used to calculate the difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space to obtain the differential feature sequence; The action segmentation module is used to perform action segmentation processing on the differential feature sequence according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module is used to perform proxy answering processing on the probability distribution sequence of the actions according to the classifier to obtain proxy answering detection results.
[0014] Furthermore, the synthetic lip sequence generation module includes: The audio data conversion submodule is used to convert the original audio data into spectral feature data according to the Fourier transform method; The speech feature extraction submodule is used to perform speech feature extraction processing on the spectral feature data according to the deep learning model to obtain key speech feature sequences. The speech feature conversion submodule is used to convert the key speech feature sequence into a visual dynamic effect based on the generative adversarial network, so as to obtain the synthetic lip feature sequence that matches the original audio data.
[0015] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the synthetic lip-based substitution detection method as described above.
[0016] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the synthetic lip-based substitution detection method as described above.
[0017] This application provides a method for detecting proxy answers based on synthesized lip sounds, comprising: receiving a proxy answer detection request carrying original audio and video data; segmenting the original audio and video data to obtain original video data and original audio data; extracting lip sequences from the original video data using a CNN face detector to obtain lip feature sequences; generating synthetic lip sequences from the original audio data using a pre-trained lip generation model to obtain synthetic lip feature sequences; calculating the difference between the lip feature sequences and the synthetic lip feature sequences in a low-dimensional embedding space to obtain a difference feature sequence; performing action segmentation on the difference feature sequence using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input; and performing proxy answer recognition on the action probability distribution sequence using a classifier to obtain a proxy answer detection result. Compared with existing technologies, this application integrates data from both audio and video modalities, making full use of the complementarity between multimodal information. This enables a more comprehensive and accurate capture of the characteristics of proxy answering behavior, significantly improving the accuracy of proxy answering detection compared to single-modal detection methods. Furthermore, due to the use of multimodal fusion, when the data of one modality is interfered with (such as audio being affected by noise or a face being obscured in the video), the data of another modality can still provide effective information, thereby enhancing the robustness of the entire detection system and enabling it to work stably in various complex environments. Attached Figure Description
[0018] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart illustrating the implementation of the substitution detection method based on synthetic lip sounds provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the substitution detection device based on synthetic lip sounds provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0023] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.
[0024] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0025] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.
[0026] Server 103 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal device 101.
[0027] It should be noted that the method for detecting substitute responses based on synthesized lip sounds provided in this application is generally executed by a server / terminal device, and correspondingly, the device for detecting substitute responses based on synthesized lip sounds is generally located in the server / terminal device.
[0028] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0029] Continue to refer to Figure 2 The diagram shows a flowchart of an embodiment of the substitution detection method based on synthetic lip sounds according to this application. The above-described substitution detection method based on synthetic lip sounds includes steps S201, S202, S203, S204, S205, S206, and S207.
[0030] In step S201, a proxy detection request carrying the original audio and video data is received.
[0031] In the embodiments of this application, a user inputs a request through their terminal device, which is received by the system. The terminal device may be a mobile terminal such as a mobile phone, smartphone, laptop, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), navigation device, etc., or a fixed terminal such as a digital TV, desktop computer, etc. It should be understood that the examples of terminal devices given herein are for convenience of understanding only and are not intended to limit this application.
[0032] In this embodiment of the application, the aforementioned original audio and video data is the specific content that users hope the system will use to effectively identify fraudulent behavior such as proxy answering in online review scenarios. The original audio and video data contains audio and video information that may involve proxy answering and serves as the basic data for subsequent detection.
[0033] In some optional implementations of the embodiments of this application, after the step of receiving the proxy detection request carrying the original audio and video data, the following steps are further included: Noise reduction processing is performed on the raw audio data; and / or The raw audio data is processed by frame segmentation.
[0034] In step S202, the original audio and video data are segmented to obtain the original video data and the original audio data.
[0035] In this embodiment, a face detection algorithm (such as the deep learning-based face detection model MTCNN) is used to locate the face region in the video frame. Then, a face keypoint detection algorithm (such as the keypoint detection method in the Dlib library) is used to determine the position and range of the mouth region. The mouth region is then cropped out from the video frame to form a mouth region video. Simultaneously, the audio signal is separated from the original audio and video data to obtain audio waveform data, preparing for subsequent feature extraction.
[0036] In step S203, the lip sequence is extracted from the original video data using a CNN face detector to obtain the lip feature sequence.
[0037] In this embodiment, the application locates the lip region of a target person in the input video and extracts a continuous frame sequence as real lip movement features. Specifically, a lightweight CNN face detector (such as MobileNet-SSD) is used to locate the target person's face in the video, adapting to different lighting conditions in remote scenes. The lip region (96×96 pixel RGB image) is extracted using a 68-point facial landmarks algorithm and converted to grayscale to reduce computation. 25 consecutive frames of lip images are sampled from the video stream to constitute a real lip sequence. It supports dynamic adjustment of the sampling frame rate to balance real-time performance and accuracy.
[0038] In step S204, the original audio data is processed to generate a synthetic lip sequence based on the pre-trained lip generation model to obtain a synthetic lip feature sequence.
[0039] In this embodiment, a pre-trained lip generation model is used to generate a synthetic lip sequence matching the speech content from audio, converting the audio modality into a visual lip modality to achieve a unified cross-modal representation; a pre-trained lightweight Wav2lip model (trained based on the LRS3 speech-lip movement dataset and noise scene enhancement) is invoked to convert the audio into a synthetic lip sequence time-aligned with the real lip sequence.
[0040] In step S205, the difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space is calculated to obtain the difference feature sequence.
[0041] In this embodiment, a shared-weight deep network is used to encode features of real and synthetic lip sequences, calculating the differences between them in a low-dimensional embedding space to quantify lip movement-speech synchronization. Specifically, a "3D deep separable convolution + ResNet" is used as the feature extractor. The 3D convolutional layer (3×3×3 kernels) captures the spatiotemporal information of lip movement, and ResNet extracts high-order semantic features. The absolute difference of features is calculated frame by frame to generate a difference feature sequence R.
[0042] In step S206, the differential feature sequence is processed by action segmentation according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input.
[0043] In this embodiment, the multi-scale temporal convolutional network has multiple convolutional kernels of different scales, enabling it to capture feature changes at different time scales. By performing multi-scale convolution operations on the fused feature vector, the network can learn action patterns in different time periods of audio and video data, and divide each time step into different action categories, outputting a corresponding action probability distribution sequence. This sequence reflects the probability of different actions occurring at each time step, providing a foundation for subsequent proxy recognition.
[0044] In step S207, the proxy answering process is performed on the action probability distribution sequence according to the classifier to obtain the proxy answering detection result.
[0045] In the embodiments of this application, the classifier can be various common machine learning classification models, such as support vector machines (SVM), decision trees, neural networks, etc. By analyzing and judging the action probability distribution sequence, the classifier can determine whether there is proxy answering behavior in the current audio and video data, and output the corresponding detection result, such as "it is proxy answering" or "it is not proxy answering".
[0046] In practical applications, taking the remote bank account opening scenario as an example, when a customer submits an account opening application through a mobile app, the system simultaneously collects raw audio and video data (such as 1080P video + 48kHz audio), including close-ups of the customer's face and voice responses to verification questions such as "Please confirm your ID number." After collecting the raw audio and video data, the system separates the audio and video streams into raw video data and raw audio data. Then, a CNN face detector (such as the MTCNN model) is used to locate the lip region, and an LSTM network is used to track the lip movement trajectory to generate a raw lip feature sequence. In addition, the system synthesizes a lip feature sequence from the raw audio data using a pre-trained lip generation model to obtain a synthesized lip feature sequence. A multi-scale temporal convolutional network is used to segment the raw and synthesized lip feature sequences, outputting an action probability distribution sequence. Finally, the system uses a classifier to classify and recognize the action probability distribution sequence to obtain the proxy answer detection result for the remote bank account opening scenario, specifically a decision of "high risk of proxy answering, manual review required" or "verification passed."
[0047] This application provides a method for detecting proxy answers based on synthesized lip sounds, comprising: receiving a proxy answer detection request carrying original audio and video data; segmenting the original audio and video data to obtain original video data and original audio data; extracting lip sequences from the original video data using a CNN face detector to obtain lip feature sequences; generating synthetic lip sequences from the original audio data using a pre-trained lip generation model to obtain synthetic lip feature sequences; calculating the difference between the lip feature sequences and the synthetic lip feature sequences in a low-dimensional embedding space to obtain a difference feature sequence; performing action segmentation on the difference feature sequence using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input; and performing proxy answer recognition on the action probability distribution sequence using a classifier to obtain a proxy answer detection result. Compared with existing technologies, this application integrates data from both audio and video modalities, making full use of the complementarity between multimodal information. This enables a more comprehensive and accurate capture of the characteristics of proxy answering behavior, significantly improving the accuracy of proxy answering detection compared to single-modal detection methods. Furthermore, due to the use of multimodal fusion, when the data of one modality is interfered with (such as audio being affected by noise or a face being obscured in the video), the data of another modality can still provide effective information, thereby enhancing the robustness of the entire detection system and enabling it to work stably in various complex environments.
[0048] In some optional implementations of the embodiments of this application, the step of generating a synthetic lip sequence from the original audio data based on a pre-trained lip generation model to obtain a synthetic lip feature sequence specifically includes the following steps: The original audio data is converted into spectral feature data using the Fourier transform method; The key speech feature sequence is obtained by extracting speech features from the spectral feature data using a deep learning model. By using a generative adversarial network, key speech feature sequences are transformed into visual dynamic effects, resulting in a synthetic lip feature sequence that matches the original audio data.
[0049] In the embodiments of this application, Fourier transform refers to performing a short-time Fourier transform (STFT) on an audio signal to generate a time-frequency matrix X(t,f), where t is a time frame and f is a frequency component.
[0050] In this embodiment of the application, performing speech feature extraction processing on the spectral feature data refers to extracting Mel frequency cepstral coefficients (MFCC) or log-Mel spectrogram to obtain spectral feature data F∈RT×D, where T is the number of time frames and D is the feature dimension.
[0051] In this embodiment of the application, the process of extracting key speech feature sequences can be as follows: 1) Deep learning model encoding: A hybrid structure of convolutional neural network (CNN) and bidirectional long short-term memory network (Bi-LSTM) is adopted to model the spatiotemporal features of F and output the hidden feature sequence H∈RT×C, where C is the dimension of the hidden layer; 2) Attention mechanism optimization: Introduce a self-attention mechanism to focus on key speech segments and generate key speech feature sequences K∈RT′×C′ (T′≤T, C′≤C).
[0052] In this embodiment of the application, the architecture of the Generative Adversarial Network (GAN) is as follows: Generator (G): Uses U-Net structure to map K to lip keypoint sequence L∈RT′×2N (N is the number of keypoints); Discriminator (D): A 3D convolutional network is used to distinguish between real lip sequences (Lreal) and generated sequences (Lfake).
[0053] Compared with existing technologies, this application improves the accuracy and naturalness of the generated results through spectral feature optimization and adversarial training.
[0054] In some optional implementations of the embodiments of this application, after the step of extracting speech features from the spectral feature data according to the deep learning model to obtain the key speech feature sequence, the following steps are further included: The temporal correspondence between key speech feature sequences and preset visual templates is calculated using a time-series analysis algorithm to obtain the time correspondence deviation. If the time deviation exceeds the preset deviation threshold, the parameters of the key speech feature sequence are adjusted to obtain the optimized key speech feature sequence.
[0055] In this embodiment of the application, the method for calculating the temporal correspondence between key speech feature sequences and preset visual templates can be as follows: 1) Temporal analysis algorithm: Using an improved dynamic time warping (DTW) or a deep learning-based temporal alignment model, calculate the temporal correspondence matrix M∈RT×S between K and V, where M(i,j) represents the matching probability between ki and vj.
[0056] 2) Calculation of time-corresponding deviation: ① Determine the optimal matching path P={(i1,j1),(i2,j2),…,(iL,jL)}; ② Calculate the time deviation sequence Δt={il jlα}l=1L, where α is the time scaling factor of the visual template (such as the frame rate conversion coefficient). The statistical deviations, including the mean and variance, yield the overall time-corresponding deviation δ = mean(∣Δt∣) + β. var(Δt) (β is the weighting coefficient).
[0057] Compared with existing technologies, this application achieves high-precision audio and video synchronization by calculating the time correspondence deviation between speech features and visual templates and dynamically adjusting speech feature parameters.
[0058] In some optional implementations of the embodiments of this application, after the step of converting the key speech feature sequence into a visual dynamic effect according to the generative adversarial network to obtain a synthetic lip feature sequence that matches the original audio data, the following step is further included: Extracting dynamic adaptation parameters from synthetic lip feature sequences; The synthesized lip feature sequence is smoothed using an interpolation algorithm and dynamic adaptation parameters to obtain an optimized synthesized lip feature sequence.
[0059] In this embodiment of the application, the extraction of dynamic adaptation parameters can be implemented as follows: 1) Speech dynamic feature analysis: ① Calculate the speech energy gradient Ei=Ei+1 Ei (Ei is the energy of the i-th frame) and pitch change rate Fi = Fi + 1 Fi; ② Mark areas with high change rates (such as plosives and vowel transitions) as dynamic sensitive areas.
[0060] 2) Lip motion correlation modeling: ① Establish the mapping relationship between speech features and lip deformation f: A→ΔL through linear regression or neural network, where ΔL is the amount of change in lip features; ② Extract the dynamic adaptation parameters P={p1,p2,…,pN}; in =α | Ei∣+β | Fi∣+γ ∥f(ai)∥2 (α, β, γ are weighting coefficients).
[0061] In this embodiment of the application, the smoothing process based on dynamic parameters can be: 1) Dynamic weight allocation: ① Calculate the smoothing weight wi=σ(Pi) for each frame based on P, where σ( () is the Sigmoid function, which maps the parameters to the interval [0,1]. ②High Values (dynamic sensitive areas) correspond to low weights (preserving details), low The value corresponds to a high weight (enhanced smoothing).
[0062] 2) Weighted interpolation smoothing: Cubic spline interpolation or locally weighted regression (LWR) is used, combined with weights. Smoothing L is achieved using the following formula:
[0063] Compared with existing technologies, this application improves the naturalness and robustness of lip animation by extracting dynamic parameters that are strongly correlated with speech signals and combining them with interpolation algorithms to achieve adaptive smoothing.
[0064] In some optional implementations of the embodiments of this application, after the step of performing action segmentation processing on the differential feature sequence according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same length as the input time step, the following step is further included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted by time series smoothing filtering to obtain a smooth action probability distribution sequence.
[0065] In the embodiments of this application, due to the influence of various factors, such as noise interference during data acquisition and the possibility that the multi-scale temporal convolutional network may not fully fit all complex situations during training, the obtained action probability distribution sequence may exhibit abnormal peaks. Abnormal peaks refer to situations in the action probability distribution sequence where the probability values at certain time steps are significantly higher than the probability values at surrounding normal time steps, and these high probability values do not conform to the actual action change patterns. Such abnormal peaks can interfere with the subsequent accurate judgment and recognition of actions; for example, they may lead to the incorrect segmentation of a continuous action into multiple segments, or the misclassification of different actions as the same action, thereby reducing the accuracy and reliability of action segmentation and recognition.
[0066] In this embodiment, to address the aforementioned issues, a judgment and adjustment step is added after obtaining the action probability distribution sequence. Specifically, the action probability distribution sequence is first detected to determine if any abnormal peaks exist. If abnormal peaks are detected, a time-series smoothing filter is used to adjust the action probability distribution sequence.
[0067] In this embodiment, time series smoothing filtering primarily reduces random fluctuations and noise in the data by applying some form of weighted averaging or local fitting to each data point in the time series data, thereby making the data smoother and more stable. This smoothing filtering method can be moving average filtering, exponential smoothing filtering, etc.
[0068] Compared with existing technologies, this application can improve the accuracy of subsequent judgment and recognition of actions.
[0069] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0070] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0072] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0073] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a substitution detection device based on synthetic lip sounds. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0074] like Figure 3 As shown, the synthetic lip-based substitution detection device 200 of this application includes: The request receiving module 210 is used to receive a proxy detection request carrying the original audio and video data; The data segmentation module 220 is used to segment the original audio and video data to obtain the original video data and the original audio data. The lip sequence extraction module 230 is used to extract lip sequences from the original video data based on the CNN face detector to obtain lip feature sequences. The synthetic lip sequence generation module 240 is used to process the original audio data into a synthetic lip sequence based on a pre-trained lip generation model to obtain a synthetic lip feature sequence. The differential feature calculation module 250 is used to calculate the difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space to obtain the differential feature sequence. The action segmentation module 260 is used to perform action segmentation processing on the differential feature sequence based on the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module 270 is used to perform proxy answering processing on the action probability distribution sequence according to the classifier to obtain proxy answering detection results.
[0075] In this embodiment of the application, a proxy response detection device 200 based on synthesized lip sounds is provided, comprising: a request receiving module 210 for receiving a proxy response detection request carrying original audio and video data; a data segmentation module 220 for segmenting the original audio and video data to obtain original video data and original audio data; a lip sequence extraction module 230 for extracting lip sequences from the original video data using a CNN face detector to obtain lip feature sequences; a synthesized lip sequence generation module 240 for generating synthesized lip sequences from the original audio data using a pre-trained lip generation model to obtain synthesized lip feature sequences; a difference feature calculation module 250 for calculating the difference between the lip feature sequences and the synthesized lip feature sequences in a low-dimensional embedding space to obtain difference feature sequences; an action segmentation module 260 for performing action segmentation on the difference feature sequences using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same length as the input time step; and a proxy response recognition module 270 for performing proxy response recognition on the action probability distribution sequence using a classifier to obtain proxy response detection results. Compared with existing technologies, this application integrates data from both audio and video modalities, making full use of the complementarity between multimodal information. This enables a more comprehensive and accurate capture of the characteristics of proxy answering behavior, significantly improving the accuracy of proxy answering detection compared to single-modal detection methods. Furthermore, due to the use of multimodal fusion, when the data of one modality is interfered with (such as audio being affected by noise or a face being obscured in the video), the data of another modality can still provide effective information, thereby enhancing the robustness of the entire detection system and enabling it to work stably in various complex environments.
[0076] In some optional implementations of the embodiments of this application, the above-mentioned synthetic lip sequence generation module includes: The audio data conversion submodule is used to convert the raw audio data into spectral feature data according to the Fourier transform method; The speech feature extraction submodule is used to extract speech features from spectral feature data based on a deep learning model to obtain key speech feature sequences. The speech feature conversion submodule is used to convert key speech feature sequences into visual dynamic effects based on the generative adversarial network, resulting in a synthetic lip feature sequence that matches the original audio data.
[0077] In some optional implementations of the embodiments of this application, the above-mentioned synthetic lip sequence generation module further includes: The time correspondence deviation submodule is used to calculate the time correspondence between key speech feature sequences and preset visual templates based on time series analysis algorithms, and to obtain the time correspondence deviation. The feature sequence adjustment submodule is used to perform parameter adjustment operations on the key speech feature sequence if the time correspondence deviation exceeds a preset deviation threshold, so as to obtain an optimized key speech feature sequence.
[0078] In some optional implementations of the embodiments of this application, the above-mentioned synthetic lip sequence generation module further includes: The dynamic adaptation parameter extraction submodule is used to extract dynamic adaptation parameters from the synthesized lip feature sequence; The smoothing submodule is used to smooth the synthesized lip feature sequence according to the interpolation algorithm and dynamic adaptation parameters to obtain the optimized synthesized lip feature sequence.
[0079] In some optional implementations of the embodiments of this application, the above-mentioned synthetic lip-based substitution detection device 200 further includes: The probability distribution sequence adjustment module is used to adjust the action probability distribution sequence according to time series smoothing filtering if there are abnormal peaks in the action probability distribution sequence, so as to obtain a smooth action probability distribution sequence.
[0080] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of this application.
[0081] Computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0082] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.
[0083] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device 300. In the embodiments of this application, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions based on a synthetic lip-sync detection method. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.
[0084] In some embodiments, processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 320 is typically used to control the overall operation of computer device 300. In embodiments of this application, processor 320 is used to execute computer-readable instructions stored in memory 310 or process data, such as executing computer-readable instructions for a method of detecting substitution based on synthesized lip sounds.
[0085] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 300 and other electronic devices.
[0086] The computer device provided in this application integrates data from both audio and video modalities, making full use of the complementarity between multimodal information. It can capture the characteristics of proxy answering behavior more comprehensively and accurately, greatly improving the accuracy of proxy answering detection compared to single-modal detection methods. In addition, due to the use of multimodal fusion, when the data of one modality is disturbed (such as the audio being affected by noise or the face in the video being obscured), the data of another modality can still provide effective information, thereby enhancing the robustness of the entire detection system and enabling it to work stably in various complex environments.
[0087] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described synthetic lip-based substitution detection method.
[0088] The computer-readable storage medium provided in this application integrates audio and video data, making full use of the complementarity between multimodal information. It can capture the characteristics of proxy answering behavior more comprehensively and accurately, greatly improving the accuracy of proxy answering detection compared with single-modal detection methods. In addition, due to the use of multimodal fusion, when the data of one modality is disturbed (such as the audio being affected by noise or the face in the video being obscured), the data of another modality can still provide effective information, thereby enhancing the robustness of the entire detection system and enabling it to work stably in various complex environments.
[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.
[0090] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A method for detecting substitution based on synthetic lip sounds, characterized in that, Includes the following steps: Receive proxy response detection requests carrying raw audio and video data; The original audio and video data are segmented to obtain original video data and original audio data; The original video data is processed by a CNN face detector to extract lip sequences, resulting in lip feature sequences. The original audio data is processed by a pre-trained lip generation model to generate a synthetic lip sequence, resulting in a synthetic lip feature sequence. The difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space is calculated to obtain the difference feature sequence; The differential feature sequence is segmented using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input. The proxy response detection result is obtained by performing proxy response identification processing on the probability distribution sequence of the actions according to the classifier.
2. The method for detecting substitution based on synthetic lip sounds according to claim 1, characterized in that, The step of generating a synthetic lip sequence from the original audio data using a pre-trained lip generation model to obtain a synthetic lip feature sequence specifically includes the following steps: The original audio data is converted into spectral feature data using the Fourier transform method; The key speech feature sequence is obtained by performing speech feature extraction processing on the spectral feature data according to the deep learning model. The key speech feature sequence is converted into a visual dynamic effect using a generative adversarial network to obtain the synthetic lip feature sequence that matches the original audio data.
3. The method for detecting substitution based on synthetic lip sounds according to claim 2, characterized in that, After the step of extracting speech features from the spectral feature data using a deep learning model to obtain a key speech feature sequence, the following steps are also included: The temporal correspondence between the key speech feature sequence and the preset visual template is calculated using a time-series analysis algorithm to obtain the time correspondence deviation. If the time deviation exceeds a preset deviation threshold, then the key speech feature sequence is adjusted to obtain an optimized key speech feature sequence.
4. The method for detecting substitution based on synthetic lip sounds according to claim 2, characterized in that, After the step of converting the key speech feature sequence into a visual dynamic effect based on the generative adversarial network to obtain the synthetic lip feature sequence that matches the original audio data, the following step is further included: Dynamic adaptation parameters are extracted from the synthesized lip feature sequence; The synthesized lip feature sequence is smoothed according to the interpolation algorithm and the dynamic adaptation parameters to obtain the optimized synthesized lip feature sequence.
5. The method for detecting substitution based on synthetic lip sounds according to claim 1, characterized in that, After performing action segmentation processing on the differential feature sequence using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input, the following steps are also included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted according to time series smoothing filtering to obtain a smooth action probability distribution sequence.
6. The method for detecting substitution based on synthetic lip sounds according to claim 1, characterized in that, Following the step of receiving the proxy detection request carrying the original audio and video data, the following steps are also included: The original audio data is subjected to noise reduction processing; and / or The original audio data is then processed by frame segmentation.
7. A device for detecting substitution based on synthesized lip sounds, characterized in that, include: The request receiving module is used to receive proxy detection requests carrying raw audio and video data; The data segmentation module is used to segment the original audio and video data to obtain original video data and original audio data. The lip sequence extraction module is used to extract lip sequences from the original video data based on a CNN face detector to obtain lip feature sequences. The synthetic lip sequence generation module is used to process the original audio data into a synthetic lip sequence based on a pre-trained lip generation model, so as to obtain a synthetic lip feature sequence. The differential feature calculation module is used to calculate the difference between the lip feature sequence and the synthesized lip feature sequence in the low-dimensional embedding space to obtain the differential feature sequence; The action segmentation module is used to perform action segmentation processing on the differential feature sequence according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module is used to perform proxy answering processing on the probability distribution sequence of the actions according to the classifier to obtain proxy answering detection results.
8. The substitution detection device based on synthetic lip sounds according to claim 7, characterized in that, The synthetic lip sequence generation module includes: The audio data conversion submodule is used to convert the original audio data into spectral feature data according to the Fourier transform method; The speech feature extraction submodule is used to perform speech feature extraction processing on the spectral feature data according to the deep learning model to obtain key speech feature sequences. The speech feature conversion submodule is used to convert the key speech feature sequence into a visual dynamic effect based on the generative adversarial network, so as to obtain the synthetic lip feature sequence that matches the original audio data.
9. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the substitution detection method based on synthetic lip sounds as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the substitution detection method based on synthetic lip sounds as described in any one of claims 1 to 6.